Arslan Bisharat, Mujtaba Nazari, Khushboo Bhadauria, Brian Ortiz, Eric Spencer, Beatriz Santos, TaiNing Wang, George K. Thiruvathukal · 2026 · arXiv
DOI: 10.48550/arXiv.2610.29767 · open-access PDF
记忆化增加了大语言模型(LLMs)的基准分数,因为模型可以复制在训练中看到的答案。研究人员用一致的标识符重命名(CIR)来测试它,这使得任务中的每一个标识符都得到一个新的名字,并将重命名任务的较低分数作为回忆的证据。我们在TLA+(Temporary Logic of Actions Plus)语言上测试这种方法,其模型检查器通过确认重命名任务是否保留原始的准确行为。
Machine translation (Meta M2M100); the English original follows.
Memorization inflates the benchmark scores of large language models (LLMs), since a model can reproduce answers it saw in training. Researchers test for it with consistent identifier renaming (CIR), which gives every identifier in a task a new name, and treat a lower score on the renamed task as evidence of recall. We test this method on the Temporal Logic of Actions Plus (TLA+) language, whose model checker acts as an oracle by confirming whether a renamed task preserves the exact behaviors of the original. Across 111 TLA+-Bench tasks, renaming lowers the correctness of Claude Opus 4.8 from 2
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.29767, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange