Newest papers

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31172 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

La evaluación del LLM-as-a-Judge se aplica cada vez más a tareas multilingües, sin embargo, se recomiendan estrategias impulsivas como Chain-of-Thought y Role Conditioning basadas únicamente en los parámetros de referencia en inglés. Se desconoce si la eficacia de sus transferencias a través de modelos de jueces y idiomas: los estudios de sensibilidad impulsiva son en gran medida centrados en inglés, mientras que los estudios de jueces multilingües existentes fijan el modelo impulsivo. En este trabajo, abordamos esta brecha mediante la medición de esta interacción a través de un parámetro de 9 estrategias impulsivas, 4 jueces de peso abierto, 27 idiomas y 4 tareas a través de 13 conjuntos de datos, produciendo más de 1M de puntuación en la salida de referencia de

Machine translation (Meta M2M100); the English original follows.

Abstract

LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Robustness (evolution), Pointwise, Computer science, Artificial intelligence, Cognitive psychology, Machine learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange