Newest papers

Looped Uniform Diffusion Language Models

Saiteja Utpala, Chieh-Hsin Lai, Yuki Mitsufuji, Praneeth Vepakomma · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30066 · cited by 0 works (OpenAlex) · open-access PDF

German

Diese Uniform Loop Diffusion Language Models (UDLMs) erzeugen Text, indem sie iterativ Sequenzen von gewöhnlichen Wortschatz-Token denoxieren, von denen jeder korrumperter und anschließend überarbeitet werden kann. Während des Trainings müssen UDLMs aus einem Kontext denoxieren, der selbst unzuverlässig sein kann, während maskierte Diffusion-Sprachmodelle (MDLMs) immer in einem sauberen Kontext konditionieren. Dieses anspruchsvolle Trainingsziel macht UDLMs FLOP-intensiver zu trainieren als MDLMs, aber es liefert eine stärkere nachgelagerte Argumentation. In letzter Zeit verbessert selektives Looping, das wiederholt einige frühe Transformer-Schichtschichten innerhalb des denoxing-Schrittes

Machine translation (Meta M2M100); the English original follows.

Abstract

Uniform diffusion language models (UDLMs) generate text by iteratively denoising sequences of ordinary vocabulary tokens, any of which may be corrupted and subsequently revised. During training, UDLMs must denoise from a context that may itself be unreliable, whereas masked diffusion language models (MDLMs) always condition on a clean context. This challenging training objective makes UDLMs more FLOP-intensive to train than MDLMs, but it yields stronger downstream reasoning performance. Recently, selective looping, which recursively applies a few early Transformer layers within each denoising step, has been shown to improve training efficiency in MDLMs under an iso-FLOP budget. We investigate looping in UDLMs while keeping the training objective and sampler unchanged. We find that looping significantly improves training efficiency: looped UDLMs reach the validation NLL of a non-looped UDLM trained on about 0.5 trillion tokens with up to 2.8x fewer training FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu. We also find that, at a matched NFE budget, looped UDLMs reach up to 9.8% lower generation perplexity than the non-looped UDLM, with most gains achieved at moderate loop counts. Finally, we study different loop-count distributions and find that sampling smaller loop counts more frequently during training improves inference efficiency: on downstream tasks, this strategy uses 12% fewer loops than uniform loop-count sampling without degrading accuracy.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Perplexity, Language model, Computer science, Inference, Vocabulary, Context (archaeology)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange