Runyang You, Yongqi Li, Tiezheng Yu, Wenjie Li · 2026 · arXiv
DOI: 10.48550/arXiv.2610.28048 · open-access PDF
On-policy distillation (OPD) mengkonsolidasikan kemampuan ahli menjadi satu siswa dengan mencocokkan distribusi lapisan akhir guru dan siswa pada peluncuran yang dihasilkan siswa. Kami menyelidiki apakah transisi prediksi mendalam, perubahan dalam prediksi antara lapisan, dapat memberikan pengawasan terkait tugas tambahan. Studi diagnostik kami menemukan bahwa kesepakatan guru-siswa yang lebih kuat dalam transisi menengah dikaitkan dengan koreksi jawaban yang lebih tinggi dalam pertanyaan, dengan jawaban yang benar menunjukkan transisi rata-rata KL yang lebih rendah di semua pasangan lapisan menengah yang diukur.
Machine translation (Meta M2M100); the English original follows.
On-policy distillation (OPD) consolidates expert capabilities into a single student by matching teacher and student final-layer distributions on student-generated rollouts. We investigate whether depthwise prediction transitions, the changes in predictions between layers, can provide additional task-relevant supervision. Our diagnostic study finds that stronger teacher--student agreement in intermediate transitions is associated with higher answer correctness within questions, with correct responses exhibiting lower mean transition KL across all measured intermediate layer pairs. These observa
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.28048, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange