Newest papers

Predictive Latents for Autoregressive Language Models: Possibility, Gauge, and Obstruction

Yuzhou Hong · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28540 · cited by 0 works (OpenAlex) · open-access PDF

Korean

다음 토큰 크로스 엔트로피에 의해 훈련된 자극적 토큰 언어 모델은 다음 토큰 크로스 엔트로피에 의해 훈련되지만, 몇 가지 최근 시스템은 토큰을 발행하지 않고도 숨겨진 상태를 반복합니다 : 지속적인 생각, 휴식 단계, 그리고 반복적인 깊이. 이 노트는 한 잠겨있는 과정에서 세 가지 위험을 분리합니다. 토큰 위험 $R_{\mathrm{tok}}$가 다음 토큰을 기록한다면, 강제 잠재 위험 $R_1$는 진정한 숨겨진 상태를 볼 수있는 단일 단계지도를 기록합니다. 오픈 루프 위험 $R_H$는 자체 출력에 반복된 후 동일한지도를 기록합니다. 바이즈의 다음 토큰은 $k$ 차원

Machine translation (Meta M2M100); the English original follows.

Abstract

Autoregressive language models are trained by next-token cross-entropy, yet several recent systems also iterate a hidden state without emitting a token: continuous thoughts, pause steps, and recurrent depth. This note separates three risks on one latent process. The token risk $R_{\mathrm{tok}}$ scores a readout. The teacher-forced latent risk $R_1$ scores a one-step map that sees the true hidden state. The open-loop risk $R_H$ scores the same map after it has been iterated on its own outputs. If the Bayes next-token posterior is a function of a $k$-dimensional statistic with full-rank covariance, a width-$k$ latent, a linear least-squares transition, and a refit unembedding match the Bayes token predictor. The statistic is identified only up to an invertible gauge: whitening the residual stream does not change the token law once the unembedding is refit. The matching obstructions are sharp. A constant latent drives $R_1$ to zero while token cross-entropy falls back to the marginal entropy. A perfect token predictor can still hide a null-space walk whose open-loop error grows linearly in the horizon. A small next-latent loss does not certificate next-token prediction, and a small next-token loss does not certificate free latent rollout, unless a rank or injectivity condition is stated with the number.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Security token, Mathematics, Autoregressive model, Statistic, Statistics, Bayes' theorem

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange