Newest papers

Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models

Haonan Huang, Yuwei Lu, Chengkun Cai · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28066 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

多语言评估经常得分一个模型的开头是是或不是,一个符号也带有模型和语言响应的约束。我们问,当不同的方式引发和读取一个答案恢复相同的判断,当他们测量一个答案的习惯时,我们收集了从八种模型在六种语言中的110万个答案,在多达105个道德符号上,变化问题的形式,答案的约束,和读取的答案,符号或句子,在两个词汇(FAR-6电池)时。故事级的判断配置在测试的读取和问卷形式之间汇聚,除了是 / 没有符号,在预设地图下负面的问卷读后:在相同的答案,符号和句子配置一致,在单元格中的平均值为0.

Machine translation (Meta M2M100); the English original follows.

Abstract

Multilingual evaluations often score a model's opening yes or no, a token that also carries model and language answering conventions. We ask when different ways of eliciting and reading an answer recover the same judgment, and when they measure an answer habit. We collect 1.1 million replies from eight models in six languages on up to 105 moral vignettes, varying the question's form, the answer constraint, and the reading of the reply, token or sentence, in both wordings (the FAR-6 battery). Story-level judgment profiles converge across the tested readouts and question forms, except for the yes/no token after a negative interrogative read under a preset mapping: on the same replies, token and sentence profiles agree at a median r = 0.67 across cells, while sentence profiles agree with free-sentence profiles at 0.94, a known linguistic variability. With each language's story consensus removed, a model's profile is nearest to its own in another language for 39 of 48 model-language cells under proposition labels and 15 under free sentences, against 5 expected by chance. Particle, position, and named-term preferences follow their own patterns. A model can keep its judgment and change its yes: scoring that token requires establishing which proposition it affirms.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Security token, Proposition, Sentence, Reading (process), Computer science, Natural language processing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange