Newest papers

Intra-Rubric Reward Divergence: Mitigating Rubric Hacking through Latent Dimension Recovery and Balancing

Jianqiang Xia, Han Xu, Dianxi Shi · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26685 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

El hackeo de rubricas limita tanto el rendimiento como la robustez del entrenamiento en el aprendizaje de reforzamiento basado en rubricas. Al descomponer las rubricas en sub-dimensiones, revelamos la Divergencia de la Recompensa Intra-Rubric: a medida que aumenta la puntuación total de la rubrica, ciertas puntuaciones de sub-dimensión disminuyen sistemáticamente, lo que sugiere que el modelo sacrifica aspectos de calidad específicos para optimizar el objetivo agregado. Desde la perspectiva sub-dimensional, observamos dos fenómenos: (1) el desequilibrio de la dimensión, lo que hace que la política favorezca las dimensiones muy cubiertas; (2) la ausencia de dimensión, lo que conduce a una brecha sistemática entre las recompensas y la calidad verdadera

Machine translation (Meta M2M100); the English original follows.

Abstract

Rubric hacking limits both performance and training robustness in rubric-based reinforcement learning. By decomposing rubrics into sub-dimensions, we reveal the Intra-Rubric Reward Divergence: as the total rubric score increases, certain sub-dimension scores systematically decrease, suggesting the model sacrifices specific quality aspects to optimize the aggregated objective. From the sub-dimension perspective, we observe two phenomena: (1) dimension imbalance, causing the policy to favor heavily covered dimensions; (2) dimension absence, leading to a systematic gap between rewards and true quality. To mitigate these issues, we propose Fair-Race Dimension Aggregation (FRDA), which has generations compete within each dimension separately to mitigate rubric imbalance, and Residual Semantic Dimension Completion (RSDC), which mines evaluation dimensions implicitly encoded in a holistic preference signal to complete rubric coverage. Experiments in chat, science, and medical domains indicate that FRDA mitigates rubric hacking with more stable improvements, while RSDC outperforms both single-reward training and naive multi-reward aggregation, yielding generally higher-quality responses.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Rubric, Dimension (graph theory), Robustness (evolution), Computer science, Hacker, Artificial intelligence

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange