Newest papers

Same Entity, Different Script: Diagnosing Cross-Script Entity Matching Failures in Language Models

Athira Gopal, Patil Prathamesh Pravin, Ankush Kadu, Arpit Jain, Ashwanth Krishnan · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31407 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

我们使用601个经过验证的Devanagari和拉丁形式的认证身份对象研究这种效果,保持问题和判断基准固定。Qwen2.5-7B的同一人的准确性在拉丁语中为Devanagari的35.1%下降,而不同人的准确性仍然高于98%;在五个模型中进行评估显示出不同的响应偏见,而减少的代币化分裂并不能可靠地缩小差距。在374个经过验证的马来西亚对子上复制设计,从相同实体构建的Devanagari的35.1%在马来西亚语中复制效果:同一人的准确性下降到5.3%;这五个模型的评估显示出完全匹配的Devanagari率,而此类似的QQ

Machine translation (Meta M2M100); the English original follows.

Abstract

Multilingual language models can give different identity judgments when the same people's names are written in different scripts. We study this effect using 601 audited identity pairs with attested Devanagari and Latin forms, holding the question and relation fixed. Qwen2.5-7B's same-person accuracy falls from 35.1% in Latin to 8.5% in Devanagari under Marathi prompts, while different-person accuracy remains above 98%. Evaluation across five models reveals distinct response biases, and reduced tokenization fragmentation does not reliably close the gap. Replicating the design on 374 audited Malayalam pairs built from the same entities reproduces the effect in a second script and language family: same-person accuracy falls to 5.3%, about half the matched Devanagari rate, while the margin still ranks matching above non-matching names at 0.925 AUROC. Latin-trained probes recover label information from Devanagari states, exceeding a string-feature baseline under similarity-matched negatives. Scalar calibration recovers 87% of the raw model-probe balanced-accuracy gap, identifying decision criterion placement as a major contributor. Activation patching localises causal sensitivity to the second entity in Qwen's middle layers; one shared sensitivity direction transfers about half the full-state margin correction. A fact-independent intervention at this site raises held-out Marathi same-person accuracy from 6.5% to 33.5%, with 98.5% different-person accuracy. Together, these results distinguish script-dependent discrimination from criterion placement and identify a causal site for improving identity judgments.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Devanagari, Computer science, Marathi, Artificial intelligence, Natural language processing, Telugu

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange