Newest papers

Intra-Rubric Reward Divergence: Mitigating Rubric Hacking through Latent Dimension Recovery and Balancing

Jianqiang Xia, Han Xu, Dianxi Shi · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26685 · cited by 0 works (OpenAlex) · open-access PDF

German

Rubric-Hacking beschränkt sowohl die Leistung als auch die Trainingsstärke im Rubric-basierten Verstärkungslernen. Durch die Zersetzung der Rubriken in Subdimensionen enthüllen wir die Intra-Rubric-Belohnung Divergenz: Wenn die Gesamtrubrik-Score zunimmt, sinken bestimmte Subdimension-Score systematisch, was darauf hindeutet, dass das Modell spezifische Qualitätsaspekte opfert, um das aggregierte Ziel zu optimieren. Aus der Subdimensionsperspektive beobachten wir zwei Phänomene: (1) Dimensionales Ungleichgewicht, das die Politik dazu veranlasst, stark abgedeckte Dimensionen zu begünstigen; (2) Dimensionale Abwesenheit, was zu einer systematischen Lücke zwischen

Machine translation (Meta M2M100); the English original follows.

Abstract

Rubric hacking limits both performance and training robustness in rubric-based reinforcement learning. By decomposing rubrics into sub-dimensions, we reveal the Intra-Rubric Reward Divergence: as the total rubric score increases, certain sub-dimension scores systematically decrease, suggesting the model sacrifices specific quality aspects to optimize the aggregated objective. From the sub-dimension perspective, we observe two phenomena: (1) dimension imbalance, causing the policy to favor heavily covered dimensions; (2) dimension absence, leading to a systematic gap between rewards and true quality. To mitigate these issues, we propose Fair-Race Dimension Aggregation (FRDA), which has generations compete within each dimension separately to mitigate rubric imbalance, and Residual Semantic Dimension Completion (RSDC), which mines evaluation dimensions implicitly encoded in a holistic preference signal to complete rubric coverage. Experiments in chat, science, and medical domains indicate that FRDA mitigates rubric hacking with more stable improvements, while RSDC outperforms both single-reward training and naive multi-reward aggregation, yielding generally higher-quality responses.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Rubric, Dimension (graph theory), Robustness (evolution), Computer science, Hacker, Artificial intelligence

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange