Newest papers

BERT Is Already a Denoiser: Cheaply Unlocking Iterative Generation in Masked Language Models

Julio C. Amador Diaz Lopez · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26601 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

マスクされた言語マスクの操作者モデル(MLMs)は、BERTのように、通常、デノイズプレトレーニングにもかかわらず、デノイズプレトレーニングのために使用されます。我々は、この能力が、デノイズトレーニングなしに条件の生成をサポートするかどうかをテストします。私たちのデコードは、マスクされたトークンを繰り返し予測し、コシンのスケジュールを使用して高い信頼性を予測します。それは、ディフォーシングタイムステップや逆のディフォーシングターゲットを必要としません。Frozen BERTはすでに、61.0%、37,5%、および同じマスクされたトークンの24.4%を再構築し、マスクされたトークンをそれぞれ15%、45%、60%、および

Machine translation (Meta M2M100); the English original follows.

Abstract

Masked language models (MLMs) such as BERT are typically used for representation learning despite their denoising pretraining. We test whether this capability supports conditional generation without diffusion training. Our decoder repeatedly predicts masked tokens and commits high-confidence predictions using a cosine schedule. It requires no diffusion timestep or reverse-diffusion objective. Frozen BERT already exhibits substantial denoising capability, reconstructing 61.0%, 37.5%, and 24.4% of masked tokens at 15%, 45%, and 60% corruption, respectively, although generation deteriorates sharply as visible context disappears. This capability alone does not yield task-directed generation: frozen BERT reaches only 11.0% on ARC-Easy and 4.2% on MMLU. Ordinary MLM adaptation, however, rapidly converts the same architecture into a usable generator. Full adaptation reaches 28.2% and 24.6%, respectively, while parameter-efficient adaptation substantially reduces unmatched outputs. On open-ended conditional generation, full MLM adaptation reaches BLEU/ROUGE-L of 0.163/0.583 on QQP and 0.107/0.327 on Quasar-T, placing it in the same broad performance range as published BERT models trained through diffusion without explicit diffusion training. Finally, applying the same generative interface to stronger masked representations substantially improves performance: RoBERTa-large reaches 44.2% on ARC-Easy and 33.8% on MMLU. These results separate three components of masked iterative generation: pretraining supplies the representation and denoising operator, task adaptation determines what the model generates, and an iterative commitment rule supplies the decoding process. Explicit diffusion training is therefore not necessary to obtain effective iterative conditional generation from pretrained MLMs in the settings studied here.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Context (archaeology), Language model, USable, Adaptation (eye), Decoding methods

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange