Newest papers

Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models

Haonan Huang, Yuwei Lu, Chengkun Cai · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28066 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

複数の言語の評価は、しばしばモデルのオープンYesまたはNoを記録し、またモデルと言語の回答制約を運ぶトークンです。我々は、答えの異なる方法が同じ判断を回復し、それらが答えの習慣を測定するとき、同じ判断を回復するときに尋ねます。我々は、質問の形態、答えの制約、および答えの読み方(FAR-6バッテリー)の両方の文言で同じ答え、トークンまたは文言の読み方で、6言語の8つのモデルから110万件の答えを収集します。我々は、ノー/ノートークンを除き、6つのテストされた読解および質問フォームを超えて、ノー/ノートークン

Machine translation (Meta M2M100); the English original follows.

Abstract

Multilingual evaluations often score a model's opening yes or no, a token that also carries model and language answering conventions. We ask when different ways of eliciting and reading an answer recover the same judgment, and when they measure an answer habit. We collect 1.1 million replies from eight models in six languages on up to 105 moral vignettes, varying the question's form, the answer constraint, and the reading of the reply, token or sentence, in both wordings (the FAR-6 battery). Story-level judgment profiles converge across the tested readouts and question forms, except for the yes/no token after a negative interrogative read under a preset mapping: on the same replies, token and sentence profiles agree at a median r = 0.67 across cells, while sentence profiles agree with free-sentence profiles at 0.94, a known linguistic variability. With each language's story consensus removed, a model's profile is nearest to its own in another language for 39 of 48 model-language cells under proposition labels and 15 under free sentences, against 5 expected by chance. Particle, position, and named-term preferences follow their own patterns. A model can keep its judgment and change its yes: scoring that token requires establishing which proposition it affirms.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Security token, Proposition, Sentence, Reading (process), Computer science, Natural language processing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange