Newest papers

Your Benchmark Is an Empirical Measure Over Difficulty

Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29927 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

ベンチマークは単にテスト質問のコレクションではありませんが、自然に困難に対する実験的測定を誘発します。この作品では、ベンチマーク評価のこの測定ベースの視点を開発します。この視点により、我々は2つの中心的な失敗モードを識別し、公式化することができます:測定不一致、ベンチマークの難易度の分布が意図された難易度の分布から異なる場合、およびカバーギャップ、困難軸に沿って発見された領域は、ベンチマークが意義のあるローカリズム化トレーニングの進歩を検出するのを妨げます。我々は、これらの2つの失敗モードをキャプチャし、厳格な理論的保証を提供する2つの診断メ

Machine translation (Meta M2M100); the English original follows.

Abstract

A benchmark is not merely a collection of test questions; it naturally induces an empirical measure over difficulty. In this work, we develop this measure-based view of benchmark evaluation. This perspective allows us to identify and formalize two central failure modes: measure mismatch, where the benchmark difficulty distribution differs from the intended difficulty distribution, and coverage gaps, where uncovered regions along the difficulty axis prevent benchmarks from detecting meaningful localized training progress. We propose two diagnostic metrics that capture these two failure modes and provide rigorous theoretical guarantees. Empirically, we analyze 25 popular benchmarks spanning 7 domains using carefully estimated item difficulties. Our results show that many benchmarks implicitly overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution; others leave substantial portions of the difficulty axis uncovered, limiting their ability to monitor the model improvement process. Finally, we turn diagnosis into action by introducing BenchFill, a simple difficulty-targeted item rewriting algorithm that effectively reduces local coverage gaps.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Measure (data warehouse), Computer science, Perspective (graphical), Limiting, Machine learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange