Newest papers

Looped Uniform Diffusion Language Models

Saiteja Utpala, Chieh-Hsin Lai, Yuki Mitsufuji, Praneeth Vepakomma · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30066 · cited by 0 works (OpenAlex) · open-access PDF

Korean

이 유일한 루프 확산 언어 모델 (UDLMs)은 평범한 사전 토큰의 일련을 반복적으로 해독함으로써 텍스트를 생성하여 모든 것이 손상되고 나중에 수정될 수 있습니다. 훈련 중 UDLMs는 자신이 신뢰할 수없는 맥락에서 denoise해야하며, 마스크된 확산 언어 모델 (MDLMs)은 항상 깨끗한 맥락에서 조건을 생성합니다.이 도전적인 훈련 목표는 UDLMs를 MDLMs보다 훈련에 더 FLOP 집중하게 만듭니다. 그러나 훈련 목표와 샘플링 성능이 더 강하게 유지됩니다. 최근에, 선택적 루프는 각각의 단계 내에서 몇 개의 초기 변형 언어 레이어를 반복적으로 적용하며, 연구

Machine translation (Meta M2M100); the English original follows.

Abstract

Uniform diffusion language models (UDLMs) generate text by iteratively denoising sequences of ordinary vocabulary tokens, any of which may be corrupted and subsequently revised. During training, UDLMs must denoise from a context that may itself be unreliable, whereas masked diffusion language models (MDLMs) always condition on a clean context. This challenging training objective makes UDLMs more FLOP-intensive to train than MDLMs, but it yields stronger downstream reasoning performance. Recently, selective looping, which recursively applies a few early Transformer layers within each denoising step, has been shown to improve training efficiency in MDLMs under an iso-FLOP budget. We investigate looping in UDLMs while keeping the training objective and sampler unchanged. We find that looping significantly improves training efficiency: looped UDLMs reach the validation NLL of a non-looped UDLM trained on about 0.5 trillion tokens with up to 2.8x fewer training FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu. We also find that, at a matched NFE budget, looped UDLMs reach up to 9.8% lower generation perplexity than the non-looped UDLM, with most gains achieved at moderate loop counts. Finally, we study different loop-count distributions and find that sampling smaller loop counts more frequently during training improves inference efficiency: on downstream tasks, this strategy uses 12% fewer loops than uniform loop-count sampling without degrading accuracy.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Perplexity, Language model, Computer science, Inference, Vocabulary, Context (archaeology)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange