Newest papers

Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro, Eric Aubinais, Chih-Hong Cheng · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26595 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

Os métodos de jailbreak são comumente comparados usando a Taxa de Sucesso de Ataque (ASR), mas essas comparações dependem de como as respostas são geradas e julgadas. Neste artigo, examinamos oito ataques contra três modelos de linguagem aberta usando três procedimentos de avaliação. Os ataques de linguagem de baixo recurso relatam uma média de ASR de 96.0% sob a correspondência de palavras-chave, mas de 2,7% sob um avaliador Qwen3 seguindo as instruções de pontuação do StrongREJECT. Mudar o número e a disposição de tentativas reverte o ranking de DeepInception e reescrever. Em um experimento separado, amostrar 100 respostas para cada solicitação inalterada na temperatura padrão de Ollama de 0,8 resulta em 100% de ASR sob a correspondência de palavras-chave, mas

Machine translation (Meta M2M100); the English original follows.

Abstract

Jailbreak methods are commonly compared using Attack Success Rate (ASR), but these comparisons depend on how responses are generated and judged. In this paper, we examine eight attacks against three open-weight language models using three evaluation procedures. Low-resource language attacks achieve a mean ASR of 96.0% under keyword matching but 2.7% under a Qwen3 evaluator following StrongREJECT scoring instructions. Changing the number and arrangement of attempts reverses the ranking of DeepInception and rewriting. In a separate experiment, sampling 100 responses for each unchanged request at Ollama's default temperature of 0.8 yields 100% ASR under keyword matching but 50% under the Qwen3 evaluator. Beyond evaluator sensitivity, these findings raise a safety concern: repeated sampling can expose residual unsafe behavior despite an initial refusal, suggesting probabilistic suppression rather than robust exclusion. We recommend explicit success definitions, matched response budgets, validated evaluators, and reporting cumulative failure under repeated interaction rather than relying on single-generation tests.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmarking, Matching (statistics), Computer science, Empirical research, Probabilistic logic, Ranking (information retrieval)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange