Newest papers

Mitigating Reward Hacking in Preference Reward Models with Evolving Penalties

Afonso Baldo, António Farinhas, Marcos Treviso, Ricardo Rei, Nuno M. Guerreiro · 2026 · arXiv

DOI: 10.48550/arXiv.2610.26496 · open-access PDF

Chinese

语言模型的增强学习已经从偏好奖励模型转移到可验证的奖励模型,这些模型能够支持更长时间的优化。偏好模型容易受到奖励黑客攻击:该策略提高了他们的分数,而不会提高响应质量,甚至在降低它时。然而,对于开放式任务,质量既不能被验证,也无法完全写入标题,它们仍然有价值。我们在政策培训期间引入了Dynalize,该模型缓解了在线奖励黑客攻击。

Machine translation (Meta M2M100); the English original follows.

Abstract

Reinforcement learning for language models has moved from preference reward models to verifiable rewards, which sustain much longer optimization. Preference models are prone to reward hacking: the policy raises their score without improving response quality, or even while degrading it. Yet for open-ended tasks, where quality can neither be verified nor fully written down as rubrics, they remain valuable. We introduce Dynalize, which mitigates reward hacking online, during policy training. Periodically and asynchronously, a language model analyzes training rollouts and writes penalties for unde

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange