Huiting Liu, Jinsong Wu · 2026 · arXiv
DOI: 10.48550/arXiv.2610.31522 · open-access PDF
Kebocoran token masa depan yang tersembunyi dapat menciptakan keuntungan arsitektur palsu: model autoregresif mungkin tampak jauh lebih baik di bawah evaluasi yang dipaksa oleh guru sementara generasinya yang berjalan bebas memburuk. Kami menyelidiki kegagalan ini dalam dekoder routing residual adaptif yang ringkasan routing tingkat urutan mengekspos posisi prefix ke token masa depan meskipun masker perhatian penyebab yang benar.
Machine translation (Meta M2M100); the English original follows.
Hidden future-token leakage can create false architectural gains: an autoregressive model may appear substantially better under teacher-forced evaluation while its free-running generation deteriorates. We investigate this failure in an adaptive residual-routing decoder whose sequence-level routing summary exposes prefix positions to future tokens despite a correct causal attention mask. An anomalous 24-layer checkpoint reduces validation loss from the baseline's 3.5958 to 1.4259, yet it fails a prefix-invariance audit and has substantially higher reference-model perplexity during generation. L
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.31522, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange