Newest papers

Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

Aman Behera, Namit Solanki, Mehul Anand · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31135 · cited by 0 works (OpenAlex) · open-access PDF

Hindi

लंबे संदर्भ के मॉडल अक्सर उत्तर देने वाले सबूत को कम प्रभावी ढंग से उपयोग करते हैं जब यह एक तत्काल सीमा से दूर दिखाई देता है। हम ट्रांसफॉर्मर, राज्य-स्पेस और हाइब्रिड मंबा आर्किटेक्चर में इस स्थिति पूर्वाग्रह की तुलना करते हैं, एक ठोस सेट के माध्यम से एक सोने के दस्तावेज़ को स्थानांतरित करके नौ विचलकों का उपयोग करते हैं। मूल्यांकित ट्रांसफॉर्मर्स एक शुरुआती लाभ दिखाते हैं कि मूल्यांकित राज्य-स्पेस मॉडल की कमी है, जबकि दोनों परिवार अंत के पास सबूत को पसंद करते हैं। एक एक राज्य-स्पेस आर्किटेक्चर के भीतर एक समायोज्य शुद्ध-विरोधी मंबा तुलना बिंदुओं को

Machine translation (Meta M2M100); the English original follows.

Abstract

Long-context models often use answer-bearing evidence less effectively when it appears away from a prompt boundary. We compare this position bias across Transformer, state-space, and hybrid Mamba architectures by moving one gold document through a fixed set of nine distractors. The evaluated Transformers show a beginning advantage that the evaluated state-space models lack, while both families favor evidence near the end. A matched pure-versus-hybrid Mamba comparison points in the same direction but remains statistically uncertain. The family difference appears among models that can answer the task. Transformer primacy reappears on synthetic retrieval, where state-space ceiling prevents a family comparison. A corpus swap within one state-space architecture changes overall accuracy without a detectable change in curve shape. Prompt changes can create or nearly remove a beginning advantage. Position remains decodable across architectures even when response curves differ, and attention concentration near initial tokens covaries with beginning sensitivity in the tested Transformer family. These exploratory results show that effective context use depends on the evaluated architecture family, capability, task, and prompt, without isolating the sequence mixer as the sole cause.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Swap (finance), Question answering, Transformer, Computer science, Artificial intelligence, Context (archaeology)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange