Newest papers

KEEP-Bench: Evaluating Evidence Preservation in Context Compression

Ziqing Man, Di Wu, Jia-Chen Gu, Junyi Zhang, Kai‐Wei Chang, Nanyun Peng · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26827 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

现有评估依靠QA指标,但读者仍然可以从不忠诚的压缩中正确地回答,所以这些指标低估了这种损失。为了弥合这种差距,我们引入了Keep-Bench,一个通过信息瓶颈直接评估压缩忠诚度的诊断基准,在三个具有挑战性的设置中有4,102个案例,需要保持可信的参考分辨率,明确分辨率和少数视角的证据。实验表明,即使有定制的提示,最强的模型仍然不可靠:尽管背景窗口超过100K代币,并且输入模型的平均容量仅为3.1K代币,但最佳的开放式模型(31B)仍然保留了仅在66%的案例中所需的证据,而主要的专有模型仅在

Machine translation (Meta M2M100); the English original follows.

Abstract

Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck. Existing evaluations rely on QA metrics, but a reader can still answer correctly from an unfaithful compression, so these metrics underestimate such loss. To bridge this gap, we introduce KEEP-Bench, a diagnostic benchmark that directly assesses compression faithfulness with 4,102 cases across three challenging settings that require keeping evidence for plausible reference resolutions, explicit qualifiers, and minority perspectives. Experiments show that even with tailored prompting, the strongest models remain unreliable: despite context windows exceeding 100K tokens and inputs averaging only 3.1K tokens, the best open-weight model (31B) preserves all required evidence in only 66.8% of cases, and the best proprietary model in only 61.6%. Performance also varies sharply across evidence structures, so conclusions from limited structures may be biased. We further introduce a metric, Evidence Preservation Return (EPR), to relate evidence recall to inference FLOPs, and find that it peaks at moderate scales (4B-8B), where recall remains low. Additional output budget mainly benefits already strong models and widens capability gaps, yet the stronger preservation of these models comes at a steep computational cost. These findings highlight the importance for future work to investigate compressors with strong information preservation under a small computation budget, rather than relying on a large output budget or model scaling.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Context (archaeology), Inference, Benchmark (surveying), Compression (physics), Data compression

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange