Newest papers

Reward Misspecification Degrades Chain-of-Thought Monitorability

Tianyi Yan, John Hewitt · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28303 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Pengawasan ketenagakerjaan (CoT) dari model bahasa (LM) adalah strategi kunci untuk mendeteksi dan mencegah perilaku buruk untuk keamanan AI. Oleh karena itu, sangat penting untuk mengidentifikasi praktik pengajaran penguatan (RL) yang menyebabkan ketenagakerjaan yang tidak dapat dipantau. Salah satu praktik tersebut mungkin tidak dapat dihindari: penghargaan salah spesifikasi. Sebagai lingkungan RL menjadi lebih kompleks, tidak mungkin untuk memiliki verifikasi sempurna, dan model dibayar untuk perilaku yang berbeda dari niat tugas yang sebenarnya. Kami menunjukkan bahwa penghargaan salah spesifikasi saja sudah merusak pengawasan. Kami membuat testbed dengan sepasang reward yang benar disesuaikan dengan niat tugas dan disesuaikan dengan reward yang memonitor perilaku yang berbeda, sehingga penghargaan hacking dapat dihasilkan dengan pelatihan mutlak dan dilabel secara terverifikasi.

Machine translation (Meta M2M100); the English original follows.

Abstract

Monitoring the chain-of-thought (CoT) of language models (LMs) is a key strategy for detecting and preventing misbehaviors for AI safety. As such, it is critical to identify reinforcement learning (RL) practices that lead to unmonitorable chain-of-thought. One such practice may be unavoidable: reward misspecification. As RL environments become more complex, it is infeasible to have perfect verification, and models are rewarded for behaviors that are different from the true task intent. We show that reward misspecification alone already degrades monitorability. We create a testbed with pairs of true reward aligned with task intent and misspecified rewards that favor a different behavior, so that reward hacking can be induced by training and labeled verifiably. Across three models and three tasks, CoT monitorability remains low as models' hack rates increase: the CoTs keep reasoning about the task intent while the outputs violate the intent and get rewarded. To mitigate unmonitorability without prior knowledge of the true reward or the hack, we collect rollouts with diverse rewards across training steps and use a LM to infer a reward description. We then inject the description into the input, which teaches the model to verbalize the reward. This reward description training improves monitorability by an average of 24% or 42% absolute, depending on the metric, across all models and tasks, and yields CoTs that reason about the misspecified reward explicitly. Code is available at https://github.com/Lorenayannnnn/reward-misspecification-degrades-cot-monitorability.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Task (project management), Computer science, Key (lock), Reinforcement learning, Artificial intelligence, Code (set theory)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange