Newest papers

Beyond Causal Masks: Auditing and Localizing Hidden Future-Token Leakage in Adaptive Sequence Models

Huiting Liu, Jinsong Wu · 2026 · arXiv

DOI: 10.48550/arXiv.2610.31522 · open-access PDF

Japanese

隠された未来トークン漏洩は、偽の建築的利益を生み出す可能性があります: 自動回帰型モデルは、教師による強制評価の下で大幅に改善される可能性がありますが、その自由運行型生成が悪化します。 私たちは、順序レベルのルーティング概要が、正しい因果的注意マスクにもかかわらず、前線位置を将来のトークンに暴露するアダプティブな残留ルーティングデコードでこの欠陥を調査します。

Machine translation (Meta M2M100); the English original follows.

Abstract

Hidden future-token leakage can create false architectural gains: an autoregressive model may appear substantially better under teacher-forced evaluation while its free-running generation deteriorates. We investigate this failure in an adaptive residual-routing decoder whose sequence-level routing summary exposes prefix positions to future tokens despite a correct causal attention mask. An anomalous 24-layer checkpoint reduces validation loss from the baseline's 3.5958 to 1.4259, yet it fails a prefix-invariance audit and has substantially higher reference-model perplexity during generation. L

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange