Newest papers

Scaling Multilingual Pretraining of Pixel Language Models

Ilker Kesen, Jenny Kunz, Kasra Malihi, Desmond Elliott · 2026 · arXiv

DOI: 10.48550/arXiv.2610.29681 · open-access PDF

Chinese

像素语言模型处理文本作为渲染图像而不是有限的词汇,这为它们提供了强大的跨语言传输能力。然而,关于这些能力如何受到更大规模的预训练的影响并不多。在这篇论文中,我们在三个变量轴上训练了一系列MPIXEL模型:更多的语言,更多的脚本和更大的模型容量,每种语言的固定计算。在涵盖83种语言的五个任务的评估中,我们发现扩展是本地的:每个轴提高了添加的语言和脚本,而增加的容量缓解了M的诅咒

Machine translation (Meta M2M100); the English original follows.

Abstract

Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities. However, not much is known about how these abilities are influenced by larger-scale pretraining. In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute. In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of m

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange