Newest papers

AgentPhishing: A Benchmark of LLM Email Agents Under Adaptive Attack

Justin Tan, Miao Li, Ding Zhao · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31404 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

Agentes de avaliação de segurança de computador estão cada vez mais lidando com informações privadas e agindo em nome dos usuários em aplicações de desktop e web, incluindo e-mail. O phishing, portanto, apresenta riscos de privacidade e segurança que dependem não apenas de se os agentes reconhecem mensagens maliciosas, mas também de como eles respondem. Os conjuntos de dados de phishing existentes foram desenvolvidos principalmente para a detecção convencional, deixando os agentes de e-mail do LLM suscetíveis de cumprir com solicitações maliciosas persuasivas insuficientemente caracterizadas. Para otimizar as habilidades de geração de e-mail reutilizáveis contra agentes individuais sob restrições de segurança fixas e uma restrição de realismo, introduzimos o AgentPhishing, um benchmark para avaliar se os ata

Machine translation (Meta M2M100); the English original follows.

Abstract

Computer-use agents increasingly handle private information and act on users' behalf across desktop and web applications, including email. Phishing therefore poses privacy and security risks that depend not only on whether agents recognize malicious messages, but also on how they respond. Existing phishing datasets were developed primarily for conventional detection, leaving agent susceptibility to persuasive malicious requests insufficiently characterized. To expose vulnerabilities that pre-existing datasets may miss, we introduce AgentPhishing, a benchmark for evaluating whether persuasive phishing attacks can evade detection and induce LLM email agents to comply with malicious requests. We develop PhishOpt, which optimizes reusable email-generation skills against individual agents under fixed benchmark constraints and a realism constraint. We analyze the generated emails using Cialdini's persuasion principles to identify strategies associated with agent failures and use these findings to construct a universal generation skill. Our evaluation shows that agents can handle pre-existing phishing emails safely yet remain susceptible to attacks adapted to agent behavior, while in an AgentPhishing evaluation of nine agents, four exhibit phishing-detection rates below $10\%$ and malicious-request compliance rates above $45\%$. These results highlight the limits of inferring agent safety from conventional phishing evaluations and the need to assess how agents respond to persuasive malicious requests.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Phishing, Computer science, Persuasion, Benchmark (surveying), Computer security, Construct (python library)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange