Newest papers

ALIAS: Evaluating Performance and Memorization in LLM-Written TLA+ Specifications

Arslan Bisharat, Mujtaba Nazari, Khushboo Bhadauria, Brian Ortiz, Eric Spencer, Beatriz Santos, TaiNing Wang, George K. Thiruvathukal, Konstantin Läufer, Mohammed Abuhamad · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29767 · cited by 0 works (OpenAlex) · open-access PDF

French

La mémorisation augmente les scores de référence des grands modèles linguistiques (LLM), car un modèle peut reproduire les réponses qu'il a vues dans la formation. Les chercheurs le testent avec un renommage d'identifiant cohérent (CIR), qui donne à chaque identifiant dans une tâche un nouveau nom, et traite un score inférieur sur la tâche renommée comme preuve de rappel. Nous testons cette méthode sur le langage Temporal Logic of Actions Plus (TLA+), dont le vérificateur de modèle agit comme un oracle en confirmant si une tâche renommée conserve les comportements exacts de l'original. Au-delà des 111 tâches TLA+-Bench, le renommage abaisse la véracité de Claude Opus 4.8

Machine translation (Meta M2M100); the English original follows.

Abstract

Memorization inflates the benchmark scores of large language models (LLMs), since a model can reproduce answers it saw in training. Researchers test for it with consistent identifier renaming (CIR), which gives every identifier in a task a new name, and treat a lower score on the renamed task as evidence of recall. We test this method on the Temporal Logic of Actions Plus (TLA+) language, whose model checker acts as an oracle by confirming whether a renamed task preserves the exact behaviors of the original. Across 111 TLA+-Bench tasks, renaming lowers the correctness of Claude Opus 4.8 from 28% to 18% and of GPT-5 from 9% to 7%. Most of this decrease comes from how the test grades, which requires exact names that most descriptions never state. Grading by behavior shrinks the decrease to about 3 percentage points for Opus 4.8 and to 0 for GPT-5. In two model families fine-tuned on disjoint halves of the tasks, renaming removes about half of the training gain in likelihood and costs 4.9 percentage points (Qwen) and 1.3 percentage points (Gemma) on unseen tasks. A lower score after renaming therefore mixes memorization, name-based grading, and the cost of new names.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Memorization, Correctness, Computer science, Oracle, Task (project management), Disjoint sets

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange