Newest papers

What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation

Idunnuoluwa A. Adeniji, Shilpika, Michael E. Papka · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30491 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

这些标签是由解析器,行为探测器和利用规则产生的测量结果,并且它们可能无法完全捕捉到痕迹显示的内容。我们在GSM8K上构建的工具使用设置中研究了这一点,在那里代理人呼叫一个计算机,其输出有时会被破坏,这些标签为工具返回的信息提供了清晰的实地真理,模型后来使用了什么,以及检测器报告了什么。我们在六种生产模型中完全捕捉了三种痕迹级检测器和30个问题。在GSM8K上构建的工具使用设置中,我们研究了最终正确性,可见性重新验证,以及错误传播可能有系统的证据。关键的错误模式是对工具返回的无声实地真理:我们的计算机从27号

Machine translation (Meta M2M100); the English original follows.

Abstract

LLM agents are often evaluated by reducing execution traces to labels such as final-answer correctness, tool verification, and fault propagation. These labels are measurements produced by parsers, behavioral detectors, and harness rules, and they may not fully capture what the trace shows. We study this in a tool-use setting built on GSM8K, where agents call a calculator whose outputs are occasionally corrupted, giving clear ground truth for what the tool returned, what the model later used, and what the detectors reported. We audit three trace-level detectors across six production models and 30 problems per model. Final correctness, visible re-verification, and fault propagation can disagree in systematic ways. A key failure mode is silent tool-output override: the calculator returns a corrupted value, but the model later uses a different, often correct, value without calling the tool again, so the bad value stops propagating while a verification detector records no re-check. Human review of 90 traces agrees perfectly with the automated labels for final correctness (67/67), fault impact (19/19), and fault relevance (18/18), but only 67/81 for verification; all 14 disagreements fall into four recurring patterns. Execution limits also change the reported result: in an ablation across two models, one ranges from 21/30 to 27/30 correct as step and token budgets change, while the other stays between 27/30 and 29/30. These findings motivate measurement transparency: an automated label should come with the trace evidence behind it, whether the question the detector asks applied to that trace, whether the run was cut off by an execution limit, and any human corrections. Our trace viewer makes this information inspectable alongside the reported results.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: TRACE (psycholinguistics), Correctness, Calculator, Computer science, Detector, Security token

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange