Newest papers

Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models

Haonan Huang, Yuwei Lu, Chengkun Cai · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28066 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Las evaluaciones multilingües a menudo ponen de manifiesto la apertura de un modelo de sí o no, un token que también lleva convenciones de respuesta de modelo y lenguaje. Preguntamos cuando diferentes maneras de provocar y leer una respuesta recuperan el mismo juicio, y cuando miden un hábito de respuesta. Recopilamos 1,1 millones de respuestas de ocho modelos en seis idiomas en hasta 105 vignettes morales, variando la forma de la pregunta, la restricción de la respuesta, y la lectura de la respuesta, token o frase, en ambas formulaciones (la batería FAR-6). Los perfiles de juicio de nivel de historia convergen entre las lecturas probadas y las formas de pregunta, excepto el sí/no token después de una lectura negativa de interrogación bajo un mapeo predefinido: en las mismas respu

Machine translation (Meta M2M100); the English original follows.

Abstract

Multilingual evaluations often score a model's opening yes or no, a token that also carries model and language answering conventions. We ask when different ways of eliciting and reading an answer recover the same judgment, and when they measure an answer habit. We collect 1.1 million replies from eight models in six languages on up to 105 moral vignettes, varying the question's form, the answer constraint, and the reading of the reply, token or sentence, in both wordings (the FAR-6 battery). Story-level judgment profiles converge across the tested readouts and question forms, except for the yes/no token after a negative interrogative read under a preset mapping: on the same replies, token and sentence profiles agree at a median r = 0.67 across cells, while sentence profiles agree with free-sentence profiles at 0.94, a known linguistic variability. With each language's story consensus removed, a model's profile is nearest to its own in another language for 39 of 48 model-language cells under proposition labels and 15 under free sentences, against 5 expected by chance. Particle, position, and named-term preferences follow their own patterns. A model can keep its judgment and change its yes: scoring that token requires establishing which proposition it affirms.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Security token, Proposition, Sentence, Reading (process), Computer science, Natural language processing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange