Newest papers

Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

Aman Behera, Namit Solanki, Mehul Anand · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31135 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

长期背景模型通常使用响应的证据更少有效,当它出现在快速边界时。我们将这种位置偏差与Transformer、状态空间和混合Mamba架构进行比较,通过通过一个固定的九个分散器的集合移动一个金文件。评估的Transformers显示出一个初始优势,而评估的状态空间模型缺乏,而两个家庭都喜欢接近终点的证据。在一个国家空间架构中,匹配的纯混合Mamba比较点在同一方向,但仍然不确定。家庭差异在能够响应任务的模型中出现。Transformer primacy在合成检索中重现,其中国家空间天花板阻止了家庭对比。在一个国家空间架构中进行体交换改变了总体准确性,但没有可检测的曲线形状

Machine translation (Meta M2M100); the English original follows.

Abstract

Long-context models often use answer-bearing evidence less effectively when it appears away from a prompt boundary. We compare this position bias across Transformer, state-space, and hybrid Mamba architectures by moving one gold document through a fixed set of nine distractors. The evaluated Transformers show a beginning advantage that the evaluated state-space models lack, while both families favor evidence near the end. A matched pure-versus-hybrid Mamba comparison points in the same direction but remains statistically uncertain. The family difference appears among models that can answer the task. Transformer primacy reappears on synthetic retrieval, where state-space ceiling prevents a family comparison. A corpus swap within one state-space architecture changes overall accuracy without a detectable change in curve shape. Prompt changes can create or nearly remove a beginning advantage. Position remains decodable across architectures even when response curves differ, and attention concentration near initial tokens covaries with beginning sensitivity in the tested Transformer family. These exploratory results show that effective context use depends on the evaluated architecture family, capability, task, and prompt, without isolating the sequence mixer as the sole cause.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Swap (finance), Question answering, Transformer, Computer science, Artificial intelligence, Context (archaeology)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange