Newest papers

Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift

Amarsaikhan Batjargal, Noriko Tomuro · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26416 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

実行はまた、追加監督として再利用可能な行動情報を提供します。 コード生成のために最初に提案されたSynergistic Policy and Reward Co-Evolving Framework(SPARK)は、生成されたプログラムが実行を通じて客観的に評価され得るため、コード生成に特に適しています。 実行はまた、追加監督として再利用可能な行動情報を提供します。 このポリシーと報酬の共同進化アプローチがコード生成に効果的に移行するかどうかを調査し、実行に基づいた反省が自己改善のための追加の源を提供するかどうかを調査します。 我々は、SPARKスタイルのロールロードリサイクルと実行単独コード最適化(GRPO)のモデルをトレーニングすることにより、このモデルを

Machine translation (Meta M2M100); the English original follows.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is particularly well suited to code generation because generated programs can be objectively evaluated through execution. Execution also provides behavioral information that can be reused as additional supervision. The Synergistic Policy and Reward Co-Evolving Framework (SPARK), originally proposed for reasoning tasks, exploits this idea by recycling training rollouts and their evaluation results to train a model not only to generate solutions but also to evaluate and improve them. We investigate whether this policy-reward co-evolution approach transfers effectively to code generation and whether execution-grounded reflection provides an additional source of self-improvement. We compare SPARK-style rollout recycling with execution-only Group Relative Policy Optimization (GRPO) using Qwen2.5-Coder-3B-Instruct. Our results do not support either hypothesis under the configuration tested. Although co-evolution improves training performance, these gains do not generalize to held-out problems. Compared at the same stage of training, rollout recycling produces roughly 34x greater KL divergence from the reference policy than execution-only GRPO. Over training, trained tokens per GRPO sequence fall from 185 to 86 as training pass rate increases, while held-out performance declines or fails to sustain early gains. We characterize these patterns as single-adapter policy drift: the shared trainable parameters increasingly depart from pretrained code-generation behavior without corresponding gains in generalization. The results show that higher training reward alone is not evidence of generalizable code self-improvement and highlight the importance of held-out evaluation and drift diagnostics.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Exploit, Code (set theory), Divergence (linguistics), Reinforcement learning, Sequence (biology)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange