Newest papers

Reward Misspecification Degrades Chain-of-Thought Monitorability

Tianyi Yan, John Hewitt · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28303 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

El monitoreo del código específico (CoT) de los modelos de lenguaje (LMs) es una estrategia clave para detectar y prevenir los malos comportamientos para la seguridad de la IA. Como tal, es crucial identificar las prácticas de aprendizaje reforzado (RL) que conducen a una cadena de pensamiento no controlable. Una de estas prácticas puede ser inevitable: la malespecificación de la recompensa. A medida que los entornos de RL se vuelven más complejos, es imposible tener una verificación perfecta, y los modelos son recompensados por comportamientos diferentes de la verdadera intención de la tarea. Mostramos que la malespecificación de la recompensa por sí sola ya degrada la monitorización. Creamos un testbed con pares de recompensas verdaderas alineadas con la intención de la tarea y malespecificadas que recom

Machine translation (Meta M2M100); the English original follows.

Abstract

Monitoring the chain-of-thought (CoT) of language models (LMs) is a key strategy for detecting and preventing misbehaviors for AI safety. As such, it is critical to identify reinforcement learning (RL) practices that lead to unmonitorable chain-of-thought. One such practice may be unavoidable: reward misspecification. As RL environments become more complex, it is infeasible to have perfect verification, and models are rewarded for behaviors that are different from the true task intent. We show that reward misspecification alone already degrades monitorability. We create a testbed with pairs of true reward aligned with task intent and misspecified rewards that favor a different behavior, so that reward hacking can be induced by training and labeled verifiably. Across three models and three tasks, CoT monitorability remains low as models' hack rates increase: the CoTs keep reasoning about the task intent while the outputs violate the intent and get rewarded. To mitigate unmonitorability without prior knowledge of the true reward or the hack, we collect rollouts with diverse rewards across training steps and use a LM to infer a reward description. We then inject the description into the input, which teaches the model to verbalize the reward. This reward description training improves monitorability by an average of 24% or 42% absolute, depending on the metric, across all models and tasks, and yields CoTs that reason about the misspecified reward explicitly. Code is available at https://github.com/Lorenayannnnn/reward-misspecification-degrades-cot-monitorability.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Task (project management), Computer science, Key (lock), Reinforcement learning, Artificial intelligence, Code (set theory)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange