Newest papers

Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift

Amarsaikhan Batjargal, Noriko Tomuro · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26416 · cited by 0 works (OpenAlex) · open-access PDF

German

Reinforcement Behavior Learning with Verifiable Rewards (RLVR) ist besonders gut geeignet für Code-Generation, weil erzeugte Programme durch Ausführung objektiv bewertet werden können. Execution liefert auch Verhaltensinformationen, die als zusätzliche Überwachung wiederverwendet werden können. Der Synergistic Policy and Reward Co-Evolving Framework (SPARK), der ursprünglich für Begründungsaufgaben vorgeschlagen wurde, nutzt diese Idee durch Recycling-Training-Rollouts und deren Bewertungsergebnisse, um ein Modell zu trainieren, um nicht nur Lösungen zu generieren, sondern auch zu bewerten und zu verbessern. Wir untersuchen, ob dieser Path-Pat-Co-Evolution-Ansatz effektiv zur Code-Generation übertragen wird und ob die Execution-

Machine translation (Meta M2M100); the English original follows.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is particularly well suited to code generation because generated programs can be objectively evaluated through execution. Execution also provides behavioral information that can be reused as additional supervision. The Synergistic Policy and Reward Co-Evolving Framework (SPARK), originally proposed for reasoning tasks, exploits this idea by recycling training rollouts and their evaluation results to train a model not only to generate solutions but also to evaluate and improve them. We investigate whether this policy-reward co-evolution approach transfers effectively to code generation and whether execution-grounded reflection provides an additional source of self-improvement. We compare SPARK-style rollout recycling with execution-only Group Relative Policy Optimization (GRPO) using Qwen2.5-Coder-3B-Instruct. Our results do not support either hypothesis under the configuration tested. Although co-evolution improves training performance, these gains do not generalize to held-out problems. Compared at the same stage of training, rollout recycling produces roughly 34x greater KL divergence from the reference policy than execution-only GRPO. Over training, trained tokens per GRPO sequence fall from 185 to 86 as training pass rate increases, while held-out performance declines or fails to sustain early gains. We characterize these patterns as single-adapter policy drift: the shared trainable parameters increasingly depart from pretrained code-generation behavior without corresponding gains in generalization. The results show that higher training reward alone is not evidence of generalizable code self-improvement and highlight the importance of held-out evaluation and drift diagnostics.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Exploit, Code (set theory), Divergence (linguistics), Reinforcement learning, Sequence (biology)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange