Newest papers

A Fisher-Geometric View of Multi-Teacher On-Policy Distillation

Liu Kaiyuan, Zhuang Ziyuan, Ma Haoxuan, Yannian Gu, Song Mingfei, Rongxiang Weng, Ye Jieping · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26525 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

このドメインを統合して統合された一般主義者に統合した専門的な基盤モデルを再統合するには、トレーニング後の集中的な目標です。 二つのアプローチは共通の使用である: モデル再統合は、それは迅速だが、それはヒューリスティックとして扱われている、および多教師の政策上のスペクティュアルトレイダルディスチレーションは、分布に匹敵するが、重いローロウトコンピュータを要求し、クロスタスクグレディント紛争に苦しむ中心的な目標です。 我々は、両者が異なる順序で見られるドメインの単一のオブジェクトであることを示します。 それは、同一のトレイダルスピクティストの共有ベース内で、

Machine translation (Meta M2M100); the English original follows.

Abstract

Consolidating specialized foundation models into a unified generalist is a central goal of post-training. Two approaches are in common use: model merging, which is fast but treated as a heuristic, and multi-teacher on-policy distillation, which matches distributions but demands heavy rollout compute and suffers from cross-task gradient conflict. We show that the two are one objective seen at different orders. Within a shared basin of homologous specialists, the multi-teacher on-policy distillation objective is, to second order, exactly the Fisher-merging objective. Its minimizer is therefore the Fisher merge, a classic model-merging method given by a closed-form combination of the specialists, so merging is optimal, to second order, as a closed-form stage of distillation. Building on this identification, we propose Residual-Governed Branch-and-Merge(RGBM), a two-phase consolidation procedure. Concretely, a full-parameter Fisher merge is intractable on foundation models, so Phase~1 restricts the merge to the task-vector subspace. Extending tangent-space analyses of merging, we show that the same coefficients that mix task vectors also mix implicit rewards, which lets a residual program on unlabelled prompts solve for them. The program minimizes the worst per-domain residual and reports its energy-normalized values as the domain residual spectrum. Phase~2 lets this spectrum govern what may be added: RGBM passivates domains with low normalized residuals, trains isolated branches only for domains above the spectral gap, and writes back the admitted updates with calibrated shrinkage. On two open-source consolidation suites, the Phase~1 merge alone matches or outperforms fully trained multi-teacher distillation, and RGBM improves on it further with zero extra joint-training steps, whereas every un-gated write-back, including continued joint distillation from the merge, falls below it.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Merge (version control), Residual, Distillation, Computer science, Mathematics, Algorithm

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange