Newest papers

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31172 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

LLM-as-a-Judge评估越来越多地应用于多语言任务,但建议仅基于英语基准的推理策略,如思维链和角色调节。无论他们的有效性是否在法官模型和语言之间转移是未知的:推理敏感性研究在很大程度上是以英语为中心,而现有的多语言法官研究解决了推理模板。在这项工作中,我们通过衡量9个推理策略的基准,4个开放的法官,27种语言,以及13个数据集中的4个任务,在黄金参考输出中获得了超过1M的点数,对人类判断进行了验证。我们发现推理策略是不可转移的:法官模型的选择主导了推理策略的选择,对推理策略的反向影响,应该在多种语言中

Machine translation (Meta M2M100); the English original follows.

Abstract

LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Robustness (evolution), Pointwise, Computer science, Artificial intelligence, Cognitive psychology, Machine learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange