Newest papers

Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro, Eric Aubinais, Chih-Hong Cheng · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26595 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Metode Jailbreak umumnya dibandingkan menggunakan Rata Kesuksesan Serangan (ASR), tetapi perbandingan ini tergantung pada bagaimana tanggapan dihasilkan dan dinilai. Dalam makalah ini, kami memeriksa delapan serangan terhadap tiga model bahasa terbuka menggunakan tiga prosedur evaluasi. Serangan bahasa sumber rendah melaporkan rata-rata ASR 96.0% di bawah kata kunci yang cocok tetapi 2.7% di bawah evaluator Qwen3 mengikuti instruksi penilaian StrongREJECT. Mengubah jumlah dan pengaturan upaya membalikkan peringkat DeepInception dan rewrite. Dalam percobaan terpisah, sampling 100 tanggapan untuk setiap permintaan yang tidak berubah pada suhu default Ollama 0,8 menghasilkan 100% ASR di bawah kata kunci yang cocok tetapi 50% di bawah evaluator Qwen3. Di luar sensitivitas evaluator, temuan ini menimbulkan kekhawatiran keamanan: pengambilan sampel berulang dapat

Machine translation (Meta M2M100); the English original follows.

Abstract

Jailbreak methods are commonly compared using Attack Success Rate (ASR), but these comparisons depend on how responses are generated and judged. In this paper, we examine eight attacks against three open-weight language models using three evaluation procedures. Low-resource language attacks achieve a mean ASR of 96.0% under keyword matching but 2.7% under a Qwen3 evaluator following StrongREJECT scoring instructions. Changing the number and arrangement of attempts reverses the ranking of DeepInception and rewriting. In a separate experiment, sampling 100 responses for each unchanged request at Ollama's default temperature of 0.8 yields 100% ASR under keyword matching but 50% under the Qwen3 evaluator. Beyond evaluator sensitivity, these findings raise a safety concern: repeated sampling can expose residual unsafe behavior despite an initial refusal, suggesting probabilistic suppression rather than robust exclusion. We recommend explicit success definitions, matched response budgets, validated evaluators, and reporting cumulative failure under repeated interaction rather than relying on single-generation tests.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmarking, Matching (statistics), Computer science, Empirical research, Probabilistic logic, Ranking (information retrieval)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange