Newest papers

Mitigating Model Collapse through Dreaming Learning

Matteo Benati, Alessandro Londei, Denise Lanzieri, Raffaello Mastromarino, Vittorio Loreto · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30009 · cited by 0 works (OpenAlex) · open-access PDF

Korean

모형 붕괴로 알려진 현상이다. 우리는 모형 붕괴로 알려진 현상이다. 우리는 통제 된 Markov 환경에서 스펙트럼으로이 퇴행을 연구합니다. 학습 된 전환 운영자의 eigenvalue 스펙트럼은 합리적인 다양성의 손실과 성능의 감소로 인해 과정의 직접적인 시각을 제공합니다. subdominant eigenvalues는 단위 서클에 접근하고 스펙트럼 격차가 좁아지기 때문에 체인 믹스의 속도를 설정합니다. $T_s < 1$의 상태 공간의 얼마나 많은 부분을 탐구합니다. 우리는 $T_s < 1$의 온도에서 샘플링이 운영자에게 반복적으로 가열하는 것처럼 작용한다는 것을 발견합니다. subdominant eigenvalues는 단위 서클에

Machine translation (Meta M2M100); the English original follows.

Abstract

Training generative models on synthetic data causes loss of diversity and a decrease in performance, a phenomenon known as model collapse. We study this degeneration spectrally in a controlled Markov setting. The eigenvalue spectrum of the learned transition operator offers a direct view of the process, since the subdominant eigenvalues set how fast the chain mixes and therefore how much of the state space it continues to explore. We find that sampling at temperature $T_s < 1$ acts on the operator like repeated sharpening, pushing it toward a deterministic system: the subdominant eigenvalues approach the unit circle and the spectral gap narrows. When $T_s < 1$, even data accumulation does not appear to arrest this drift. We then analyze Dreaming Learning, a regulariser that, in parallel with regular training, adds a loss term on the model's own samples drawn at a raised temperature $T_d > 1$. Because high-temperature samples flatten the predicted distribution, training on them keeps the entries of the learned operator from becoming vanishingly small, which in turn keeps the eigenvalues bounded away from the unit circle. Measurements of the spectra over 5000 generations are consistent with this picture, and the mitigation effect persists in more complex settings such as LSTM language models and a 124M-parameter GPT-2.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Eigenvalues and eigenvectors, Operator (biology), Bounded function, Set (abstract data type), Markov chain, Spectrum (functional analysis)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange