Newest papers

CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

Wenhao Yu, Zhengcheng Wang, Jiahong Liu, Zexuan Qiu, Minda Hu, Shaohang Wei, Jiafei Lyu, Junyan Lyu, Haobo Fu, Irwin King · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29965 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

联合强化学习和自我蒸结合任务奖励与密集的教师指导为代理培训,但它们的梯度部分可以取消。混合系数控制相对规模而不选择性地删除对立的组件。我们介绍CORD,这是一种对冲突意识优化两个目标的一般框架。CORD将受保护的目标选择从触发投影的条件分开,并将梯度组合一次每个优化器小批次。最接近的点公式会产生封闭式更新,并在优化转型之前对受保护的目标产生第一类效果。 梯度诊断在代理培训中将冲突连接到失去的局部优化,并激励选择性纠正。 作为一个梯度组合机制,CORD可以添加到不同的联合训练配方中,同时保持其目标。 除了三种

Machine translation (Meta M2M100); the English original follows.

Abstract

Joint reinforcement learning and self-distillation combines task rewards with dense teacher guidance for agent training, but their gradients can partly cancel. A mixing coefficient controls relative scale without selectively removing opposing components. We present CORD, a general framework for conflict-aware optimization of two objectives. CORD separates the choice of protected objective from the condition that triggers projection, and combines the gradients once per optimizer mini-batch. A nearest-point formulation yields a closed-form update and characterizes its first-order effect on the protected objective before optimizer transformations. Gradient diagnostics connect conflict in agent training to lost local optimization progress and motivate selective correction. As a gradient-combination mechanism, CORD can be added to different joint training recipes while retaining their objectives. Across three such recipes, it improves ALFWorld success by 1.5-15.9 percentage points. On WebShop, it improves two of the three pairs, with SDAR rising from 60.7% to 64.6% success. Our analyses explain why protection direction and projection gate must be chosen together, and why the distillation objective and its mixing weight remain consequential after projection.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Mixing (physics), Joint (building), Distillation, Projection (relational algebra), Reinforcement learning, Computer science

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange