Newest papers

BERT Is Already a Denoiser: Cheaply Unlocking Iterative Generation in Masked Language Models

Julio C. Amador Diaz Lopez · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26601 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Dalam hal ini, kami akan menguji apakah kemampuan ini mendukung produksi kondisional tanpa pelatihan difusi. Decoder kami berulang kali memprediksi token yang disamarkan dan melakukan prediksi kepercayaan tinggi dengan menggunakan jadwal kosin. Ini biasanya tidak memerlukan tahap waktu difusi atau tujuan difusi terbalik. Frozen BERT sudah menunjukkan kemampuan denosis substansial, mengkonstruksi 61.0%, 37.5%, dan 24.4% dari token yang disamarkan yang sama pada 15%, 45%, dan 60% korupsi, masing-masing, meskipun generasi memburuk tajam pada pasokan saat konteks terlihat menghilang. Kemampuan ini saja tidak menghasilkan generasi yang ditargetkan tugas: BERT yang disamarkan hanya mencapai 11.0% pada ARC-Easy dan 4.2% pada MMLU yang terpisah. Adaptasi diffusi MLM biasa, bagaimanapun, dengan cepat mengubah proses adaptasi yang diperlukan ke adaptor yang

Machine translation (Meta M2M100); the English original follows.

Abstract

Masked language models (MLMs) such as BERT are typically used for representation learning despite their denoising pretraining. We test whether this capability supports conditional generation without diffusion training. Our decoder repeatedly predicts masked tokens and commits high-confidence predictions using a cosine schedule. It requires no diffusion timestep or reverse-diffusion objective. Frozen BERT already exhibits substantial denoising capability, reconstructing 61.0%, 37.5%, and 24.4% of masked tokens at 15%, 45%, and 60% corruption, respectively, although generation deteriorates sharply as visible context disappears. This capability alone does not yield task-directed generation: frozen BERT reaches only 11.0% on ARC-Easy and 4.2% on MMLU. Ordinary MLM adaptation, however, rapidly converts the same architecture into a usable generator. Full adaptation reaches 28.2% and 24.6%, respectively, while parameter-efficient adaptation substantially reduces unmatched outputs. On open-ended conditional generation, full MLM adaptation reaches BLEU/ROUGE-L of 0.163/0.583 on QQP and 0.107/0.327 on Quasar-T, placing it in the same broad performance range as published BERT models trained through diffusion without explicit diffusion training. Finally, applying the same generative interface to stronger masked representations substantially improves performance: RoBERTa-large reaches 44.2% on ARC-Easy and 33.8% on MMLU. These results separate three components of masked iterative generation: pretraining supplies the representation and denoising operator, task adaptation determines what the model generates, and an iterative commitment rule supplies the decoding process. Explicit diffusion training is therefore not necessary to obtain effective iterative conditional generation from pretrained MLMs in the settings studied here.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Context (archaeology), Language model, USable, Adaptation (eye), Decoding methods

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange