Newest papers

Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models

Haonan Huang, Yuwei Lu, Chengkun Cai · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28066 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

As avaliações estabelecidas multilíngues frequentemente pontuam a abertura de um modelo sim ou não, um token que também carrega convenções de resposta do modelo e da língua. Perguntamos quando diferentes maneiras de induzir e ler uma resposta recuperam o mesmo julgamento, e quando medem um hábito de resposta. Coletamos 1,1 milhão de respostas de oito modelos em seis idiomas em até 105 vignettes morais, variando a forma da pergunta, a restrição da resposta, e a leitura da resposta, token ou frase, em ambas as formulações (a bateria FAR-6). Os perfis de julgamento de nível de história convergem entre as leituras testadas e formulários de perguntas, exceto para o sim/não token após uma leitura negativa de interrogação sob um mapeamento pré-

Machine translation (Meta M2M100); the English original follows.

Abstract

Multilingual evaluations often score a model's opening yes or no, a token that also carries model and language answering conventions. We ask when different ways of eliciting and reading an answer recover the same judgment, and when they measure an answer habit. We collect 1.1 million replies from eight models in six languages on up to 105 moral vignettes, varying the question's form, the answer constraint, and the reading of the reply, token or sentence, in both wordings (the FAR-6 battery). Story-level judgment profiles converge across the tested readouts and question forms, except for the yes/no token after a negative interrogative read under a preset mapping: on the same replies, token and sentence profiles agree at a median r = 0.67 across cells, while sentence profiles agree with free-sentence profiles at 0.94, a known linguistic variability. With each language's story consensus removed, a model's profile is nearest to its own in another language for 39 of 48 model-language cells under proposition labels and 15 under free sentences, against 5 expected by chance. Particle, position, and named-term preferences follow their own patterns. A model can keep its judgment and change its yes: scoring that token requires establishing which proposition it affirms.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Security token, Proposition, Sentence, Reading (process), Computer science, Natural language processing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange