Newest papers

CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

Wenhao Yu, Zhengcheng Wang, Jiahong Liu, Zexuan Qiu, Minda Hu, Shaohang Wei, Jiafei Lyu, Junyan Lyu, Haobo Fu, Irwin King · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29965 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

共同強化学習と自己蒸発は、課題の報酬と、エージェント訓練のための密集した教師ガイドラインを組み合わせますが、それらのグレディエントは部分的にキャンセルすることができます。混合因子は、対立するコンポーネントを選択的に除去することなく、相対的なスケールを制御します。我々は、CORDを紹介し、2つの目標の対立意識の最適化のための一般的な枠組みです。CORDは、プロジェクションを引き起こす条件から保護されたオブジェクトの選択を分離し、グレディエントをミニバッチごとに一度組み合わせます。最も近いポイント式は、目標を維持しながら、さまざまな共同トレーニングレシピに追加することができます。その他の3つのレシピでは、

Machine translation (Meta M2M100); the English original follows.

Abstract

Joint reinforcement learning and self-distillation combines task rewards with dense teacher guidance for agent training, but their gradients can partly cancel. A mixing coefficient controls relative scale without selectively removing opposing components. We present CORD, a general framework for conflict-aware optimization of two objectives. CORD separates the choice of protected objective from the condition that triggers projection, and combines the gradients once per optimizer mini-batch. A nearest-point formulation yields a closed-form update and characterizes its first-order effect on the protected objective before optimizer transformations. Gradient diagnostics connect conflict in agent training to lost local optimization progress and motivate selective correction. As a gradient-combination mechanism, CORD can be added to different joint training recipes while retaining their objectives. Across three such recipes, it improves ALFWorld success by 1.5-15.9 percentage points. On WebShop, it improves two of the three pairs, with SDAR rising from 60.7% to 64.6% success. Our analyses explain why protection direction and projection gate must be chosen together, and why the distillation objective and its mixing weight remain consequential after projection.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Mixing (physics), Joint (building), Distillation, Projection (relational algebra), Reinforcement learning, Computer science

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange