Newest papers

Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro, Eric Aubinais, Chih-Hong Cheng · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26595 · cited by 0 works (OpenAlex) · open-access PDF

Korean

Jailbreak 방법은 일반적으로 공격 성공률 (ASR)을 사용하여 비교되지만 이러한 비교는 응답이 어떻게 생성되고 판단되는지에 달려 있습니다. 이 논문에서 우리는 세 가지 평가 절차를 사용하여 세 개의 개방형 언어 모델에 대한 8 개의 공격을 검토합니다. 낮은 자원 언어 공격은 키워드 일치하에 평균 ASR이 96.0 %이지만 Qwen3 평가자에 따라 2.7%입니다. StrongREJECT 점수 지침에 따라 응답을 생성하고 판단합니다. 시도 수와 배열을 변경하면 DeepInception 및 재설정의 순위가 반전됩니다. 별도의 실험에서 Ollama의 기본 온도 0.8의 변하지 않은 각 요청에 대한 100 개의 응답을 샘플링하면 키워드 일치하에 100%의 AS

Machine translation (Meta M2M100); the English original follows.

Abstract

Jailbreak methods are commonly compared using Attack Success Rate (ASR), but these comparisons depend on how responses are generated and judged. In this paper, we examine eight attacks against three open-weight language models using three evaluation procedures. Low-resource language attacks achieve a mean ASR of 96.0% under keyword matching but 2.7% under a Qwen3 evaluator following StrongREJECT scoring instructions. Changing the number and arrangement of attempts reverses the ranking of DeepInception and rewriting. In a separate experiment, sampling 100 responses for each unchanged request at Ollama's default temperature of 0.8 yields 100% ASR under keyword matching but 50% under the Qwen3 evaluator. Beyond evaluator sensitivity, these findings raise a safety concern: repeated sampling can expose residual unsafe behavior despite an initial refusal, suggesting probabilistic suppression rather than robust exclusion. We recommend explicit success definitions, matched response budgets, validated evaluators, and reporting cumulative failure under repeated interaction rather than relying on single-generation tests.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmarking, Matching (statistics), Computer science, Empirical research, Probabilistic logic, Ranking (information retrieval)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange