Newest papers

Intra-Rubric Reward Divergence: Mitigating Rubric Hacking through Latent Dimension Recovery and Balancing

Jianqiang Xia, Han Xu, Dianxi Shi · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26685 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Dengan memecah rubrik menjadi sub-dimensi, kami mengungkapkan Intra-Rubric Reward Divergence: ketika total skor rubrik meningkat, skor sub-dimensi tertentu secara sistematis menurun, menunjukkan bahwa model mengorbankan aspek kualitas tertentu untuk mengoptimalkan tujuan agregasi. Dari sudut pandang sub-dimensi, kami mengamati dua fenomena: (1) ketidakseimbangan dimensi, menyebabkan kebijakan untuk mendukung dimensi yang sangat tercakup; (2) ketidakhadiran dimensi, yang mengarah pada kesenjangan sistematis antara penghargaan dan kualitas sejati. Untuk meringankan masalah ini, kami mengusulkan Fair-Race Dimension Aggregation (DA), yang memiliki generasi bersaing dalam setiap dimensi secara terpisah untuk meringankan ketidakseimbangan rubric, dan Residual Semantic Dimension Completion (RSRS), yang mengindikasikan

Machine translation (Meta M2M100); the English original follows.

Abstract

Rubric hacking limits both performance and training robustness in rubric-based reinforcement learning. By decomposing rubrics into sub-dimensions, we reveal the Intra-Rubric Reward Divergence: as the total rubric score increases, certain sub-dimension scores systematically decrease, suggesting the model sacrifices specific quality aspects to optimize the aggregated objective. From the sub-dimension perspective, we observe two phenomena: (1) dimension imbalance, causing the policy to favor heavily covered dimensions; (2) dimension absence, leading to a systematic gap between rewards and true quality. To mitigate these issues, we propose Fair-Race Dimension Aggregation (FRDA), which has generations compete within each dimension separately to mitigate rubric imbalance, and Residual Semantic Dimension Completion (RSDC), which mines evaluation dimensions implicitly encoded in a holistic preference signal to complete rubric coverage. Experiments in chat, science, and medical domains indicate that FRDA mitigates rubric hacking with more stable improvements, while RSDC outperforms both single-reward training and naive multi-reward aggregation, yielding generally higher-quality responses.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Rubric, Dimension (graph theory), Robustness (evolution), Computer science, Hacker, Artificial intelligence

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange