Newest papers

HoloCosm: Beyond Single-Metric Benchmark Compression Across Multiple Metrics

Yupei Lin, Zhongzhan Huang, Junxin Li, Guoming Ling, Shanshan Zhong, Hefeng Wu, Yukai Shi, Liang Lin · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26714 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

基准密度压缩通过选择一小组样本来降低评估成本。现有方法通常使用单个指标来指导选择,虽然基准密度通常在多个维度上评估模型。我们发现模型排名往往在指标之间相对较差,这表明不同的指标在模型性能的70%以上捕捉不同的方面。 针对一个指标选择的样本在使用压缩评估时不会可靠地保留其他人的分数和排名。 在80%压缩时,每一个单一指标基线都会产生比平等尺寸的随机采样更高的重构错误。 为了应对这一挑战,我们介绍了HoloCosm,它共同减少了对源模型和指标的正常分数重建错误,同时限制了排名扭曲。 我们评估HoloCosm在15

Machine translation (Meta M2M100); the English original follows.

Abstract

Benchmark compression reduces evaluation cost by selecting a small set of samples. Existing methods typically guide selection using a single metric, although benchmarks often assess models along multiple dimensions. We find that model rankings are often weakly correlated across metrics, indicating that different metrics capture distinct aspects of model performance. Samples selected for one metric do not reliably preserve scores and rankings on others when used for compressed evaluation. At 80\% compression, each of the five single-metric baselines yields higher reconstruction error than equal-size random sampling in over 70\% of cross-metric comparisons. To address this challenge, we present HoloCosm, which jointly minimizes normalized score-reconstruction error across source models and metrics while constraining ranking distortion. We evaluate HoloCosm across multiple compression rates on 15 language, multimodal, VLA, and agent benchmark configurations with 85 component metrics. With 80\% compression, HoloCosm outperforms Random, the strongest baseline overall: average NMAE drops by 35.3\% to 0.236, worst-component NMAE drops by 39.5\% to 0.330, and average Spearman correlation rises from 0.884 to 0.915. In addition, HoloCosm remains competitive with existing baselines in single-metric evaluation. We will release our code and experimental results.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Metric (unit), Computer science, Compression (physics), Set (abstract data type), Data mining

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange