Newest papers

Scaling Multilingual Pretraining of Pixel Language Models

İlker Kesen, Jenny Kunz, Kasra Malihi, Desmond Elliott · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29681 · cited by 0 works (OpenAlex) · open-access PDF

Korean

픽셀 언어 모델은 최종 사전 대신 렌더링 된 이미지로 텍스트를 처리하여 강력한 교차 언어 전송 능력을 제공합니다. 그러나 이러한 능력은 더 큰 규모의 사전 훈련에 의해 어떻게 영향을받는지에 대해 많이 알려져 있지 않습니다. 이 논문에서, 우리는 3 개의 변형의 축을 따라 MPIXEL 모델의 가족을 훈련합니다 : 더 많은 언어, 더 많은 스크립트, 고정 언어 컴퓨팅에서 더 큰 모델 용량을 제공합니다. 83 개 언어를 다루는 다섯 가지 작업에 대한 평가에서, 우리는 스케일링이 지역적으로 작용한다는 것을 발견합니다 : 각 축은 추가 된 언어와 스크립트를 개선하며, 증가 된 용량은 더 많은 언어에 대한 사전 훈련으로부터 다국어

Machine translation (Meta M2M100); the English original follows.

Abstract

Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities. However, not much is known about how these abilities are influenced by larger-scale pretraining. In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute. In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of multilinguality from pretraining on more languages. Transfer to unseen scripts improves with multilingual pretraining, but does not further improve with scaling. On typologically distant low-resource languages, our models outperform subword and character-level baselines across many writing systems. We also show that larger pixel models reserve more layers for visual processing, and that cross-lingual alignment weakens as more languages are added. Overall, our findings characterize how multilingual scaling improves pixel language models.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Scripting language, Scaling, Pixel, Artificial intelligence, Process (computing)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange