Newest papers

Mitigating Reward Hacking in Preference Reward Models with Evolving Penalties

Afonso Baldo, António Farinhas, Marcos Treviso, Ricardo Rei, Nuno M. Guerreiro · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26496 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

El aprendizaje de comportamiento reforzado para modelos de lenguaje se ha trasladado de modelos de recompensa de preferencia a recompensas verificables, que soportan una optimización mucho más larga. Los modelos de preferencia son propensos a recompensar el hacking: la política eleva su puntuación sin mejorar la calidad de respuesta, o incluso mientras la degrada. Sin embargo, para tareas abiertas, donde la calidad no puede ser verificada ni completamente escrita como rubricas, siguen siendo valiosas. Introducimos Dynalize, que mitiga el hacking de recompensas en línea, durante el entrenamiento de políticas. Periódicamente y de forma asíncrona, un modelo de lenguaje analiza los rollos de formación y escribe penas por comportamientos indeseables que el modelo de recompensa no parece recompensar; un juez las aplica a cada rollout.

Machine translation (Meta M2M100); the English original follows.

Abstract

Reinforcement learning for language models has moved from preference reward models to verifiable rewards, which sustain much longer optimization. Preference models are prone to reward hacking: the policy raises their score without improving response quality, or even while degrading it. Yet for open-ended tasks, where quality can neither be verified nor fully written down as rubrics, they remain valuable. We introduce Dynalize, which mitigates reward hacking online, during policy training. Periodically and asynchronously, a language model analyzes training rollouts and writes penalties for undesirable behaviors that the reward model appears to reward; a judge applies them to every rollout. In clinical counseling and creative writing, training on the reward model alone raises its score continuously but ultimately leads to a decline in external benchmarks. Dynalize sustains longer training and results in improved downstream performance. For example, we find that our method suppresses patterns incentivized by preference models that, without penalties, spread to most responses, and even to the model's reasoning, which the reward model does not explicitly score. Furthermore, we show that its gains hold in multi-turn training, and even after we retrain the reward model on the discovered behaviors.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Preference, Hacker, Reinforcement learning, Quality (philosophy), Computer science, Cognitive psychology

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange