Newest papers

Behavior Beyond Text: Evaluating Refusal Transfer Failure in LLM Agents

Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang, Adam Al Gharib · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28982 · cited by 0 works (OpenAlex) · open-access PDF

French

Alors que les grands agents de langue trouvent leurs modèles de sécurité plutôt directement déployés en tant qu'agents d'appel d'outils, leurs évaluations de sécurité se concentrent toujours principalement sur leur capacité à refuser les actions nuisibles dans le texte. Nous enquêtons sur la question de savoir si le comportement de refus dans le texte transfère à l'utilisation de l'outil. Nous construisons un benchmark de 2,304 prompts sur quatre domaines, composés de triplets correspondants qui présentent la même demande nuisible ou bénigne dans les appels d'outils d'orientation non sécurisée. Sous la suppression de la demande d'outil d'examen standard et les paramètres d'outil d'orientation adversaire. Nous évaluons ensuite la sécurité des appels d'

Machine translation (Meta M2M100); the English original follows.

Abstract

While large language models are increasingly deployed as tool-calling agents, their safety evaluations still focus primarily on their ability to refuse harmful requests in text. We investigate whether the refusal behavior in text transfers to tool use. We construct a benchmark of 2,304 prompts across four domains, consisting of matched triplets that present the same harmful or benign request in text-only, standard tool-enabled, and adversarially framed tool-enabled settings. We then assess tool-call safety using 20 predefined forbidden actions. Testing five models of varying sizes and families, we find that requests refused in text frequently result in unsafe tool calls. Under standard tool-enabled prompting, this divergence occurs for 10-52 percent of harmful requests refused in chat. To mechanistically investigate this gap, we identify a refusal-associated direction in the residual stream of each of the five models. We find that in tool-enabled contexts, the activations along this direction decrease in four of five models. Furthermore, lower activation predicts unsafe tool calls across all five models (AUC 0.76-0.86). However, restoring the signal to the respective text-only value through activation patching only prevents a subset of unsafe calls, primarily by suppressing tool use rather than by producing safer calls. Steering along the refusal direction nearly eliminates unsafe calls only at strengths that also suppress benign tool use or degrade the output quality. Our findings indicate that the identified refusal direction contributes to whether a model acts, but only partially explains the refusal divergence. Therefore, evaluating the safety of tool-calling agents requires directly examining their actions along with their textual responses.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: SAFER, Computer science, Construct (python library), Focus (optics), Benchmark (surveying), Risk analysis (engineering)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange