Newest papers

EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence

Qijin Zhang, Hanzhen Wang, Yuehan Zhao, Haoyu Dai, Yao Shu, Zhongxiang Dai, Shuang Qiu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31650 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

使用工具的语言模型代理人依赖于环境事实来决定是否允许采取行动。当工具响应忽略了安全相关事实时,代理人可以执行禁止的行动,即使用户请求是良性的,每个返回的陈述都是真实的。我们引入了\bench{},一个可执行的基准来评估安全执行、安全边界和证据恢复在LLM代理人缺失证据下时,它包含300个匹配的安全/安全任务对在六个领域。通过系统地抑制关键事实,基准测量了关键证据缺失下不安全的执行,并确定了观察到的最低失败核心性,包括在完整证据和没有观察到的失败的情况下,经过全面测试。一个独立审计工具测试了代理人是否在完成允许的任务时寻找缺失事实,然后

Machine translation (Meta M2M100); the English original follows.

Abstract

Tool-using language model agents rely on environmental facts to decide whether an action is permitted. When a tool response omits a safety-relevant fact, an agent may carry out a prohibited action even though the user request is benign and every returned statement is true. We introduce \bench{}, an executable benchmark for evaluating unsafe execution, safety boundaries, and evidence recovery in LLM agents under missing evidence. It contains 300 matched unsafe/safe task pairs across six domains. By systematically suppressing critical facts, the benchmark measures unsafe execution under critical evidence omission and determines the minimum observed failure cardinality, including failures with full evidence and cases with no observed failure after exhaustive testing. An independent audit tool tests whether agents seek missing facts and then withhold prohibited actions while completing permissible tasks. The benchmark reveals substantial variation across the ten models in omission robustness, evidence acquisition, and appropriate action. Resistance to missing evidence does not necessarily translate into useful recovery. Recovery access reduces unsafe execution for all ten models, but agents differ in whether they actively obtain the missing facts and use them to act appropriately in both worlds.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Executable, Task (project management), Computer science, Action (physics), Audit

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange