Newest papers

A Fisher-Geometric View of Multi-Teacher On-Policy Distillation

Liu Kaiyuan, Zhuang Ziyuan, Ma Haoxuan, Yannian Gu, Song Mingfei, Rongxiang Weng, Ye Jieping · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26525 · cited by 0 works (OpenAlex) · open-access PDF

Arabic

وتشمل هذه النماذج إعادة توزيع النماذج المتخصصة في المجال المفتوح إلى إعادة توزيع النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج المفتوحة إلى النماذج

Machine translation (Meta M2M100); the English original follows.

Abstract

Consolidating specialized foundation models into a unified generalist is a central goal of post-training. Two approaches are in common use: model merging, which is fast but treated as a heuristic, and multi-teacher on-policy distillation, which matches distributions but demands heavy rollout compute and suffers from cross-task gradient conflict. We show that the two are one objective seen at different orders. Within a shared basin of homologous specialists, the multi-teacher on-policy distillation objective is, to second order, exactly the Fisher-merging objective. Its minimizer is therefore the Fisher merge, a classic model-merging method given by a closed-form combination of the specialists, so merging is optimal, to second order, as a closed-form stage of distillation. Building on this identification, we propose Residual-Governed Branch-and-Merge(RGBM), a two-phase consolidation procedure. Concretely, a full-parameter Fisher merge is intractable on foundation models, so Phase~1 restricts the merge to the task-vector subspace. Extending tangent-space analyses of merging, we show that the same coefficients that mix task vectors also mix implicit rewards, which lets a residual program on unlabelled prompts solve for them. The program minimizes the worst per-domain residual and reports its energy-normalized values as the domain residual spectrum. Phase~2 lets this spectrum govern what may be added: RGBM passivates domains with low normalized residuals, trains isolated branches only for domains above the spectral gap, and writes back the admitted updates with calibrated shrinkage. On two open-source consolidation suites, the Phase~1 merge alone matches or outperforms fully trained multi-teacher distillation, and RGBM improves on it further with zero extra joint-training steps, whereas every un-gated write-back, including continued joint distillation from the merge, falls below it.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Merge (version control), Residual, Distillation, Computer science, Mathematics, Algorithm

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange