Newest papers

CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

Wenhao Yu, Zhengcheng Wang, Jiahong Liu, Zexuan Qiu, Minda Hu, Shaohang Wei, Jiafei Lyu, Junyan Lyu, Haobo Fu, Irwin King · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29965 · cited by 0 works (OpenAlex) · open-access PDF

French

L'apprentissage du renforcement conjoint et l'autodestillation combinent les récompenses des tâches avec la guidance dense des enseignants pour la formation des agents, mais leurs gradients peuvent en partie être annulés. Un coefficient de mélange contrôle l'échelle relative sans éliminer sélectivement les composants opposés. Nous présentons CORD, un cadre général pour l'optimisation consciente des conflits de deux objectifs. CORD sépare le choix de l'objectif protégé de la condition qui déclenche la projection et combine les gradients une fois par mini-batch d'optimisateur. Une formulation de point le plus proche produit une mise à jour de forme fermée et caractérise son effet de premier ordre sur l'objectif protégé avant les transform

Machine translation (Meta M2M100); the English original follows.

Abstract

Joint reinforcement learning and self-distillation combines task rewards with dense teacher guidance for agent training, but their gradients can partly cancel. A mixing coefficient controls relative scale without selectively removing opposing components. We present CORD, a general framework for conflict-aware optimization of two objectives. CORD separates the choice of protected objective from the condition that triggers projection, and combines the gradients once per optimizer mini-batch. A nearest-point formulation yields a closed-form update and characterizes its first-order effect on the protected objective before optimizer transformations. Gradient diagnostics connect conflict in agent training to lost local optimization progress and motivate selective correction. As a gradient-combination mechanism, CORD can be added to different joint training recipes while retaining their objectives. Across three such recipes, it improves ALFWorld success by 1.5-15.9 percentage points. On WebShop, it improves two of the three pairs, with SDAR rising from 60.7% to 64.6% success. Our analyses explain why protection direction and projection gate must be chosen together, and why the distillation objective and its mixing weight remain consequential after projection.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Mixing (physics), Joint (building), Distillation, Projection (relational algebra), Reinforcement learning, Computer science

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange