Newest papers

Intra-Rubric Reward Divergence: Mitigating Rubric Hacking through Latent Dimension Recovery and Balancing

Jianqiang Xia, Han Xu, Dianxi Shi · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26685 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

通过将分类分解为子维度,我们揭示了内部分类奖励差异:随着总分数的增加,某些子维度分数系统地减少,这表明模型牺牲特定质量方面来优化总体目标。从子维度的角度来看,我们观察到两种现象:(1)维度失衡,导致政策有利于重度覆盖的维度;(2)维度缺失,导致奖励和真实质量之间的系统差距。为了减轻这些问题,我们提出了公平竞争的维度聚合(DAFR),它有几代人分别在每个维度内竞争,以减轻红利不平衡,以及残留的语义维度聚合(RRSRS),这表明DC采矿数字化信号的多维度响应暗示地在医学领域中产生了更高水平的

Machine translation (Meta M2M100); the English original follows.

Abstract

Rubric hacking limits both performance and training robustness in rubric-based reinforcement learning. By decomposing rubrics into sub-dimensions, we reveal the Intra-Rubric Reward Divergence: as the total rubric score increases, certain sub-dimension scores systematically decrease, suggesting the model sacrifices specific quality aspects to optimize the aggregated objective. From the sub-dimension perspective, we observe two phenomena: (1) dimension imbalance, causing the policy to favor heavily covered dimensions; (2) dimension absence, leading to a systematic gap between rewards and true quality. To mitigate these issues, we propose Fair-Race Dimension Aggregation (FRDA), which has generations compete within each dimension separately to mitigate rubric imbalance, and Residual Semantic Dimension Completion (RSDC), which mines evaluation dimensions implicitly encoded in a holistic preference signal to complete rubric coverage. Experiments in chat, science, and medical domains indicate that FRDA mitigates rubric hacking with more stable improvements, while RSDC outperforms both single-reward training and naive multi-reward aggregation, yielding generally higher-quality responses.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Rubric, Dimension (graph theory), Robustness (evolution), Computer science, Hacker, Artificial intelligence

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange