Newest papers

KEEP-Bench: Evaluating Evidence Preservation in Context Compression

Ziqing Man, Di Wu, Jia-Chen Gu, Junyi Zhang, Kai‐Wei Chang, Nanyun Peng · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26827 · cited by 0 works (OpenAlex) · open-access PDF

Korean

컨텍스트 압축은 오래된 컨텍스트 효율성을 향상시킬 수 있지만 정보 bottleneck를 통해 충실성 문제를 도입할 수 있습니다. 기존 평가는 QA 매개 변수에 의존하지만 독자는 여전히 부정적인 압축에서 올바른 대답을 할 수 있으므로 이러한 매개 변수는 그러한 손실을 과소평가합니다. 이 격차를 조정하기 위해 우리는 KEEP-Bench를 도입합니다. 정보 포켓에 대한 진단 기준은 4102건의 압축 충실성을 직접적으로 평가하며 3개의 도전적인 설정을 통해 압축 충실성을 평가합니다. 여전히 읽는 사람은 확실한 참조 해석, 명시적 인 자격증 및 소수의 관점에 대한 증거를 유지해야합니다. 실험은 조정된 인프팅으로도 가장 강력한 모델

Machine translation (Meta M2M100); the English original follows.

Abstract

Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck. Existing evaluations rely on QA metrics, but a reader can still answer correctly from an unfaithful compression, so these metrics underestimate such loss. To bridge this gap, we introduce KEEP-Bench, a diagnostic benchmark that directly assesses compression faithfulness with 4,102 cases across three challenging settings that require keeping evidence for plausible reference resolutions, explicit qualifiers, and minority perspectives. Experiments show that even with tailored prompting, the strongest models remain unreliable: despite context windows exceeding 100K tokens and inputs averaging only 3.1K tokens, the best open-weight model (31B) preserves all required evidence in only 66.8% of cases, and the best proprietary model in only 61.6%. Performance also varies sharply across evidence structures, so conclusions from limited structures may be biased. We further introduce a metric, Evidence Preservation Return (EPR), to relate evidence recall to inference FLOPs, and find that it peaks at moderate scales (4B-8B), where recall remains low. Additional output budget mainly benefits already strong models and widens capability gaps, yet the stronger preservation of these models comes at a steep computational cost. These findings highlight the importance for future work to investigate compressors with strong information preservation under a small computation budget, rather than relying on a large output budget or model scaling.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Context (archaeology), Inference, Benchmark (surveying), Compression (physics), Data compression

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange