Newest papers

Your Benchmark Is an Empirical Measure Over Difficulty

Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29927 · cited by 0 works (OpenAlex) · open-access PDF

Korean

벤치마크는 단순히 테스트 질문의 집합이 아닙니다; 그것은 자연스럽게 어려움에 대한 실험적 측정을 유도합니다. 이 작품에서, 우리는 벤치마크 평가의 측정 기반 견해를 개발합니다. 이 관점에서 우리는 두 가지 중앙 실패 모드를 식별하고 공식화 할 수 있습니다 : 측정 불일치, 벤치마크 어려움 분포가 의도 된 어려움 분포와 다르며, 커버 격차, 어려움 축을 따라 발견 된 지역은 벤치마크가 의미있는 지역화 된 훈련 진행을 감지하지 못하게합니다. 우리는이 두 가지 실패 모드를 캡처하고 엄격한 이론적 보증을 제공하는 두 가지 진단 메트릭을 제안합니다. 경험적으로, 우리는 신중하게 추정된 항목 어려

Machine translation (Meta M2M100); the English original follows.

Abstract

A benchmark is not merely a collection of test questions; it naturally induces an empirical measure over difficulty. In this work, we develop this measure-based view of benchmark evaluation. This perspective allows us to identify and formalize two central failure modes: measure mismatch, where the benchmark difficulty distribution differs from the intended difficulty distribution, and coverage gaps, where uncovered regions along the difficulty axis prevent benchmarks from detecting meaningful localized training progress. We propose two diagnostic metrics that capture these two failure modes and provide rigorous theoretical guarantees. Empirically, we analyze 25 popular benchmarks spanning 7 domains using carefully estimated item difficulties. Our results show that many benchmarks implicitly overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution; others leave substantial portions of the difficulty axis uncovered, limiting their ability to monitor the model improvement process. Finally, we turn diagnosis into action by introducing BenchFill, a simple difficulty-targeted item rewriting algorithm that effectively reduces local coverage gaps.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Measure (data warehouse), Computer science, Perspective (graphical), Limiting, Machine learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange