Newest papers

EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence

Qijin Zhang, Hanzhen Wang, Yuehan Zhao, Haoyu Dai, Yao Shu, Zhongxiang Dai, Shuang Qiu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31650 · cited by 0 works (OpenAlex) · open-access PDF

Arabic

ويستند العميل إلى حقائق بيئية لتحديد ما إذا كانت العملية موثوقة. عندما ينسى استجابة الأداة حقائق ذات صلة بالسلامة، فإن العميل يمكن أن يمارس العملية الممنوحة حتى لو كان طلب المستخدم صافيًا ومتوافقًا مع كل بيانات المراجعة صحيحة. نحن نقدم نموذجًا متكاملًا قابلًا للتنفيذ لتقييم تنفيذ غير آمن، حدود الأمان، واستعادة الأدلة في العميلات LLM تحت الأدلة المفقودة. يحتوي ذلك على 300 زوجة عمل غير آمنة / آمنة متوافقة في ست مجالات. عن طريق تقسيم الحقائق الأساسية بشكل مستمر، يحدد نموذجًا متكاملًا تنفيذ غير آمنًا تحت الأدلة

Machine translation (Meta M2M100); the English original follows.

Abstract

Tool-using language model agents rely on environmental facts to decide whether an action is permitted. When a tool response omits a safety-relevant fact, an agent may carry out a prohibited action even though the user request is benign and every returned statement is true. We introduce \bench{}, an executable benchmark for evaluating unsafe execution, safety boundaries, and evidence recovery in LLM agents under missing evidence. It contains 300 matched unsafe/safe task pairs across six domains. By systematically suppressing critical facts, the benchmark measures unsafe execution under critical evidence omission and determines the minimum observed failure cardinality, including failures with full evidence and cases with no observed failure after exhaustive testing. An independent audit tool tests whether agents seek missing facts and then withhold prohibited actions while completing permissible tasks. The benchmark reveals substantial variation across the ten models in omission robustness, evidence acquisition, and appropriate action. Resistance to missing evidence does not necessarily translate into useful recovery. Recovery access reduces unsafe execution for all ten models, but agents differ in whether they actively obtain the missing facts and use them to act appropriately in both worlds.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Executable, Task (project management), Computer science, Action (physics), Audit

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange