Newest papers

Scaling Multilingual Pretraining of Pixel Language Models

İlker Kesen, Jenny Kunz, Kasra Malihi, Desmond Elliott · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29681 · cited by 0 works (OpenAlex) · open-access PDF

French

Les modèles de langues pixel traitent le texte comme des images rendues au lieu d’un vocabulaire fini, ce qui leur confère de fortes capacités de transfert entre langues. Cependant, on ne sait pas grand-chose sur la façon dont ces capacités sont influencées par la pré-entraînement à plus grande échelle. Dans ce document, nous formons une famille de modèles MPIXEL sur trois axes de variation: plus de langues, plus de scripts et plus de capacité de modèle sous un ordinateur par langue fixe. Dans les évaluations de cinq tâches couvrant 83 langues, nous constatons que l’échelle agit localement: chaque axe améliore les langues et les scripts ajoutés, tandis que la capacité accrue atténue la malédiction de la mult

Machine translation (Meta M2M100); the English original follows.

Abstract

Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities. However, not much is known about how these abilities are influenced by larger-scale pretraining. In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute. In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of multilinguality from pretraining on more languages. Transfer to unseen scripts improves with multilingual pretraining, but does not further improve with scaling. On typologically distant low-resource languages, our models outperform subword and character-level baselines across many writing systems. We also show that larger pixel models reserve more layers for visual processing, and that cross-lingual alignment weakens as more languages are added. Overall, our findings characterize how multilingual scaling improves pixel language models.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Scripting language, Scaling, Pixel, Artificial intelligence, Process (computing)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange