Newest papers

Behavior Beyond Text: Evaluating Refusal Transfer Failure in LLM Agents

Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang, Adam Al Gharib · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28982 · cited by 0 works (OpenAlex) · open-access PDF

Korean

큰 언어 에이전트는 도구 호출 에이전트가 아닌 도구 호출 에이전트로 점점 더 직접적으로 배포되지만, 그들의 보안 평가는 여전히 텍스트에서 유해한 행동 요청을 거부 할 수있는 능력에 주로 초점을 맞추고 있습니다. 우리는 텍스트에서 거부 행동이 도구 사용에 전송되는지 여부를 조사합니다. 우리는 4 개의 도메인에 걸쳐 2,304 개의 벤스 폰트의 벤치 벤치 폰트의 벤치 벤치 벤치 폰트를 구축하고, 텍스트에서 동일한 유해한 또는 유해한 요청을 자주 나타내는 트리플렛으로 구성되어 있습니다. 표준 도구가 활성화 된 도구가 활성화 된 벤치 폰트

Machine translation (Meta M2M100); the English original follows.

Abstract

While large language models are increasingly deployed as tool-calling agents, their safety evaluations still focus primarily on their ability to refuse harmful requests in text. We investigate whether the refusal behavior in text transfers to tool use. We construct a benchmark of 2,304 prompts across four domains, consisting of matched triplets that present the same harmful or benign request in text-only, standard tool-enabled, and adversarially framed tool-enabled settings. We then assess tool-call safety using 20 predefined forbidden actions. Testing five models of varying sizes and families, we find that requests refused in text frequently result in unsafe tool calls. Under standard tool-enabled prompting, this divergence occurs for 10-52 percent of harmful requests refused in chat. To mechanistically investigate this gap, we identify a refusal-associated direction in the residual stream of each of the five models. We find that in tool-enabled contexts, the activations along this direction decrease in four of five models. Furthermore, lower activation predicts unsafe tool calls across all five models (AUC 0.76-0.86). However, restoring the signal to the respective text-only value through activation patching only prevents a subset of unsafe calls, primarily by suppressing tool use rather than by producing safer calls. Steering along the refusal direction nearly eliminates unsafe calls only at strengths that also suppress benign tool use or degrade the output quality. Our findings indicate that the identified refusal direction contributes to whether a model acts, but only partially explains the refusal divergence. Therefore, evaluating the safety of tool-calling agents requires directly examining their actions along with their textual responses.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: SAFER, Computer science, Construct (python library), Focus (optics), Benchmark (surveying), Risk analysis (engineering)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange