Newest papers

Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

Aman Behera, Namit Solanki, Mehul Anand · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31135 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Model konteks panjang sering menggunakan bukti responsif kurang efektif ketika muncul jauh dari batas cepat. Kami membandingkan bias posisi ini di antara Transformer, ruang negara, dan arsitektur Mamba hibrida dengan memindahkan satu dokumen emas melalui set tetap dari sembilan distractor. Transformers yang dinilai menunjukkan keuntungan awal yang kurang dari model ruang negara yang dinilai, sementara kedua keluarga lebih memilih bukti dekat akhir. Sebuah pertukaran korpus dalam arsitektur ruang negara-hybrid Mamba yang disesuaikan dengan titik perbandingan murni ke arah yang sama tetapi tetap tidak pasti secara statistik. Perbedaan keluarga muncul di antara model yang dapat menjawab tugas. Transformer primacy muncul kembali pada retrieval sintetis, di mana langit-langit ruang negara yang dinilai mencegah perbandingan keluarga. Sebuah pertukaran korpus tugas dalam satu arsitektur ruang negara mengubah akurasi keseluruhan tanpa perubahan yang dapat dideteksi dalam bentuk

Machine translation (Meta M2M100); the English original follows.

Abstract

Long-context models often use answer-bearing evidence less effectively when it appears away from a prompt boundary. We compare this position bias across Transformer, state-space, and hybrid Mamba architectures by moving one gold document through a fixed set of nine distractors. The evaluated Transformers show a beginning advantage that the evaluated state-space models lack, while both families favor evidence near the end. A matched pure-versus-hybrid Mamba comparison points in the same direction but remains statistically uncertain. The family difference appears among models that can answer the task. Transformer primacy reappears on synthetic retrieval, where state-space ceiling prevents a family comparison. A corpus swap within one state-space architecture changes overall accuracy without a detectable change in curve shape. Prompt changes can create or nearly remove a beginning advantage. Position remains decodable across architectures even when response curves differ, and attention concentration near initial tokens covaries with beginning sensitivity in the tested Transformer family. These exploratory results show that effective context use depends on the evaluated architecture family, capability, task, and prompt, without isolating the sequence mixer as the sole cause.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Swap (finance), Question answering, Transformer, Computer science, Artificial intelligence, Context (archaeology)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange