Newest papers

Beyond Causal Masks: Auditing and Localizing Hidden Future-Token Leakage in Adaptive Sequence Models

Huiting Liu, Jinsong Wu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31522 · cited by 0 works (OpenAlex) · open-access PDF

Hindi

गुप्त भविष्य टोकन लीकरेज गलत आर्किटेक्चरल लाभ पैदा कर सकता है: एक आत्म-अनुकूलित मॉडल शिक्षकों द्वारा मजबूत मूल्यांकन के तहत काफी बेहतर दिखाई दे सकता है जबकि इसकी मुफ्त चलने वाली साफ-फिक्स पीढ़ी खराब हो जाती है। हम एक अनुकूलित शेष-रेटिंग डिकोडर में इस विफलता की जांच करते हैं, जिसके अनुक्रम स्तर के राउडिंग संक्षेप में भविष्य टोकनों को सही कारण ध्यान के मास्क के बावजूद पूर्वावलोकन पदों को प्रकट करता है। एक असामान्य 24 परतों का निरीक्षण बिंदु मूल लाइन के 3.5958 से 1.4259 तक वैलिंग नुकसान को कम करता है, फिर भी यह एक अनुकूलित पूर्वावलोकन-इ

Machine translation (Meta M2M100); the English original follows.

Abstract

Hidden future-token leakage can create false architectural gains: an autoregressive model may appear substantially better under teacher-forced evaluation while its free-running generation deteriorates. We investigate this failure in an adaptive residual-routing decoder whose sequence-level routing summary exposes prefix positions to future tokens despite a correct causal attention mask. An anomalous 24-layer checkpoint reduces validation loss from the baseline's 3.5958 to 1.4259, yet it fails a prefix-invariance audit and has substantially higher reference-model perplexity during generation. Layer-wise diagnostics and implementation inspection trace the violation to a full-sequence summary broadcast through the dynamic routing-query path. Two independently trained 12-layer causal repairs eliminate detectable leakage and return validation loss and reference perplexity to the control scale, removing the large apparent advantage. As diagnostic tools, we use the established prefix-invariance contract, maximum prefix-output discrepancy (LeakScore), complementary behavioral metrics, and layer-wise profiles. Supporting experiments cover six synthetic implementation pairs spanning five module families, clean pretrained controls, controlled fault injection, and cache consistency across numerical precisions. These results illustrate how hidden future-token access can produce misleading claims of architectural progress and why lower teacher-forced loss should be interpreted together with whole-model causal checks and free-running evaluation.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Perplexity, Computer science, Prefix, Leakage (economics), Consistency (knowledge bases), Data mining

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange