Newest papers

CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

Wenhao Yu, Zhengcheng Wang, Jiahong Liu, Zexuan Qiu, Minda Hu, Shaohang Wei, Jiafei Lyu, Junyan Lyu, Haobo Fu, Irwin King · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29965 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Joint reinforcement learning and self-destillation combines task rewards with dense teacher guidance for agent training, but their gradients can partially cancel. A mixing coefficient controls relative scale without selectively removing opposing components. We present CORD, a general framework for conflict-aware optimization of two objectives. CORD memisahkan pilihan objektif yang dilindungi dari kondisi yang memicu proyeksi, dan menggabungkan gradient sekali per mini-batch optimizer. Sebuah formulasi titik terdekat menghasilkan pembaruan bentuk tertutup dan menggambarkan efek pertama pada objektif yang dilindungi sebelum transformasi optimizer. Diagnosis gradient menghubungkan konflik dalam pelatihan agen ke optimalisasi lokal yang hilang dan memotivasi koreksi selektif. Sebagai mekanisme kombinasi gradient, CORD dapat ditambahkan ke resep pel

Machine translation (Meta M2M100); the English original follows.

Abstract

Joint reinforcement learning and self-distillation combines task rewards with dense teacher guidance for agent training, but their gradients can partly cancel. A mixing coefficient controls relative scale without selectively removing opposing components. We present CORD, a general framework for conflict-aware optimization of two objectives. CORD separates the choice of protected objective from the condition that triggers projection, and combines the gradients once per optimizer mini-batch. A nearest-point formulation yields a closed-form update and characterizes its first-order effect on the protected objective before optimizer transformations. Gradient diagnostics connect conflict in agent training to lost local optimization progress and motivate selective correction. As a gradient-combination mechanism, CORD can be added to different joint training recipes while retaining their objectives. Across three such recipes, it improves ALFWorld success by 1.5-15.9 percentage points. On WebShop, it improves two of the three pairs, with SDAR rising from 60.7% to 64.6% success. Our analyses explain why protection direction and projection gate must be chosen together, and why the distillation objective and its mixing weight remain consequential after projection.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Mixing (physics), Joint (building), Distillation, Projection (relational algebra), Reinforcement learning, Computer science

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange