Newest papers

Loopholing All the Way: Language Modeling with pure Self-Conditioning

Adrien Deschenaux, Nadezhda Ilieva, Justin Deschenaux, Manuel Madeira, Dorina Thanou, Aurélien Lucchi, Antonio Orvieto, Çağlar Gülçehre, Pascal Frossard B · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30068 · cited by 0 works (OpenAlex) · open-access PDF

French

Les modèles récents de langage de diffusion (DLM) utilisent le self-conditioning (SC), ou loopholing, pour améliorer leurs performances. Alors que SC est traditionnellement considéré comme un heuristique auxiliaire, son succès empirique soulève la question: un modèle de génération continue peut-il être basé uniquement sur le self-conditioning? Nous répondons affirmativement avec LAW (« Loopholing All the Way »), qui propage l'Insertion attendue sous la prédiction du denoiser à travers les étapes, décodant les jetons une fois, à la fin. Sur Sudoku, LAW surpasse les précédents Modèles de diffusion discrète et continue, y compris Loopholing Discrete Diffusion. Sur Maze, LAW est le plus précis de 2 à

Machine translation (Meta M2M100); the English original follows.

Abstract

Recent Diffusion Language Models (DLMs) use Self-Conditioning (SC), or Loopholing, to improve their performance. While SC is traditionally seen as an auxiliary heuristic, its empirical success raises the question: can a continuous generative model be based purely on Self-Conditioning? We answer affirmatively with LAW (``Loopholing All the Way''), which propagates the Expected Embedding under the denoiser's prediction across steps, decoding the tokens once, at the end. On Sudoku, LAW outperforms previous Discrete and Continuous Diffusion Models, including Loopholing Discrete Diffusion. On Maze, LAW is the most accurate from 2 to 64 NFE. After training on TinyGSM and sampling at temperature 1 with 1024 NFE, LAW solves 42.2% of GSM8K problems against 34.5% for uniform Loopholing Discrete Diffusion. Additionally, fine-tuning a pre-trained Uniform Diffusion Model (Duo) with LAW increases its accuracy from 17.2% to 48.1%. Finally, on OpenWebText, LAW pushes the Pareto frontier and approaches Autoregressive Models (ARMs) with 64 NFE. Find our code at https://github.com/ilievanadezhda/law.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Language model, Computer science, Generative grammar, Autoregressive model, Embedding, Decoding methods

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange