Newest papers

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31172 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

A avaliação do LLM-as-a-Judge é cada vez mais aplicada a tarefas multilíngues, mas estratégias prompt como Chain-of-Thought e Role Conditioning são recomendadas com base exclusivamente em benchmarks ingleses. Se a sua eficácia transfere através de modelos de juiz e idiomas permanece desconhecida: estudos prompt-sensitivity são em grande parte centrados em Inglês, enquanto estudos de juiz multilíngue existentes fixam o modelo prompt. Neste trabalho, abordamos essa lacuna medindo essa interação através de um benchmark de 9 estratégias prompting, 4 juízes de peso aberto, 27 idiomas e 4 tarefas em 13 conjuntos de dados, dando mais de 1M pontuações pontuais em saídas de referência de ouro, validadas contra julg

Machine translation (Meta M2M100); the English original follows.

Abstract

LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Robustness (evolution), Pointwise, Computer science, Artificial intelligence, Cognitive psychology, Machine learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange