Newest papers

Loopholing All the Way: Language Modeling with pure Self-Conditioning

Adrien Deschenaux, Nadezhda Ilieva, Justin Deschenaux, Manuel Madeira, Dorina Thanou, Aurélien Lucchi, Antonio Orvieto, Çağlar Gülçehre, Pascal Frossard B · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30068 · cited by 0 works (OpenAlex) · open-access PDF

German

Recent Diffusion Language Models (DLMs) verwenden Self-Conditioning (SC) oder Loopholing, um ihre Leistung zu verbessern. Während SC traditionell als eine Hilfsheuristik angesehen wird, erhebt sein empirischer Erfolg die Frage: Kann ein kontinuierliches generatives Modell rein auf Self-Conditioning basieren? Wir antworten ja mit LAW (“Loopholing All the Way“), das das Erwartete Embedding unter der Denoiser-Vorhersage über Schritte propagiert, das Decodieren der Token einmal, am Ende. Auf Sudoku, LAW übertrifft frühere Discrete und Continuous Diffusion Models, einschließlich Loopholing Discrete Diffusion. Auf Maze, LAW ist am genauesten von 2 bis 64 NFE. Nach dem Training auf TinyGSM und Sampling bei

Machine translation (Meta M2M100); the English original follows.

Abstract

Recent Diffusion Language Models (DLMs) use Self-Conditioning (SC), or Loopholing, to improve their performance. While SC is traditionally seen as an auxiliary heuristic, its empirical success raises the question: can a continuous generative model be based purely on Self-Conditioning? We answer affirmatively with LAW (``Loopholing All the Way''), which propagates the Expected Embedding under the denoiser's prediction across steps, decoding the tokens once, at the end. On Sudoku, LAW outperforms previous Discrete and Continuous Diffusion Models, including Loopholing Discrete Diffusion. On Maze, LAW is the most accurate from 2 to 64 NFE. After training on TinyGSM and sampling at temperature 1 with 1024 NFE, LAW solves 42.2% of GSM8K problems against 34.5% for uniform Loopholing Discrete Diffusion. Additionally, fine-tuning a pre-trained Uniform Diffusion Model (Duo) with LAW increases its accuracy from 17.2% to 48.1%. Finally, on OpenWebText, LAW pushes the Pareto frontier and approaches Autoregressive Models (ARMs) with 64 NFE. Find our code at https://github.com/ilievanadezhda/law.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Language model, Computer science, Generative grammar, Autoregressive model, Embedding, Decoding methods

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange