Newest papers

Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro, Eric Aubinais, Chih-Hong Cheng · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26595 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Los métodos de jailbreak se comparan comúnmente utilizando la tasa de éxito de ataque (ASR), pero estas comparaciones dependen de cómo se generan y juzgan las respuestas. En este artículo, examinamos ocho ataques contra tres modelos de lenguaje abierto utilizando tres procedimientos de evaluación. Los ataques de lenguaje de bajo recurso reportan un ASR promedio de 96,0% bajo la combinación de palabras clave, pero un 2,7% bajo un evaluador Qwen3 siguiendo las instrucciones de puntuación de StrongREJECT. Cambiar el número y el arranque de los intentos revuelve el ranking de DeepInception y reescritura. En un experimento separado, el muestreo de 100 respuestas para cada solicitud inalterada a la temperatura predeterminada de Ollama de 0,8 da un resultado de 100% de ASR bajo la

Machine translation (Meta M2M100); the English original follows.

Abstract

Jailbreak methods are commonly compared using Attack Success Rate (ASR), but these comparisons depend on how responses are generated and judged. In this paper, we examine eight attacks against three open-weight language models using three evaluation procedures. Low-resource language attacks achieve a mean ASR of 96.0% under keyword matching but 2.7% under a Qwen3 evaluator following StrongREJECT scoring instructions. Changing the number and arrangement of attempts reverses the ranking of DeepInception and rewriting. In a separate experiment, sampling 100 responses for each unchanged request at Ollama's default temperature of 0.8 yields 100% ASR under keyword matching but 50% under the Qwen3 evaluator. Beyond evaluator sensitivity, these findings raise a safety concern: repeated sampling can expose residual unsafe behavior despite an initial refusal, suggesting probabilistic suppression rather than robust exclusion. We recommend explicit success definitions, matched response budgets, validated evaluators, and reporting cumulative failure under repeated interaction rather than relying on single-generation tests.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmarking, Matching (statistics), Computer science, Empirical research, Probabilistic logic, Ranking (information retrieval)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange