Newest papers

Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro, Eric Aubinais, Chih-Hong Cheng · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26595 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

在这篇论文中,我们使用三种评估程序对三种开放式语言模型进行八次攻击。使用攻击成功率(ASR)的低资源语言攻击报告的平均ASR为96.0%在关键字匹配下,但在Qwen3评估器下为2.7%取决于如何产生和评估响应。在这篇论文中,我们研究了对三种开放式语言模型的八次攻击。使用三种评估程序的低资源语言攻击报告的平均ASR为96.0%在关键字匹配下,但根据StrongREJECT评估指示的Qwen3评估器为2.7%。改变尝试的数量和安排会扭转DeepInception和重写的排名。在一个单独的实验中,对每一个未改变的请求的100个响应的采样

Machine translation (Meta M2M100); the English original follows.

Abstract

Jailbreak methods are commonly compared using Attack Success Rate (ASR), but these comparisons depend on how responses are generated and judged. In this paper, we examine eight attacks against three open-weight language models using three evaluation procedures. Low-resource language attacks achieve a mean ASR of 96.0% under keyword matching but 2.7% under a Qwen3 evaluator following StrongREJECT scoring instructions. Changing the number and arrangement of attempts reverses the ranking of DeepInception and rewriting. In a separate experiment, sampling 100 responses for each unchanged request at Ollama's default temperature of 0.8 yields 100% ASR under keyword matching but 50% under the Qwen3 evaluator. Beyond evaluator sensitivity, these findings raise a safety concern: repeated sampling can expose residual unsafe behavior despite an initial refusal, suggesting probabilistic suppression rather than robust exclusion. We recommend explicit success definitions, matched response budgets, validated evaluators, and reporting cumulative failure under repeated interaction rather than relying on single-generation tests.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmarking, Matching (statistics), Computer science, Empirical research, Probabilistic logic, Ranking (information retrieval)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange