Donghyun Jeong, Yoon-Yeong Kim · 2026 · arXiv
DOI: 10.48550/arXiv.2610.29355 · open-access PDF
Sparse Mixture-of-Experts (SMoE) Modelle verbessern die Skalierbarkeit, indem sie Eingangs-Token auf nur eine Untergruppe von Experten-Netzwerken rotieren, aber sie leiden oft unter Repräsentations-Kollaps, bei dem mehrere Experten in Richtung ähnlicher Funktionen konvergieren und ihre beabsichtigte Vielfalt verlieren. Frühere Arbeiten haben diese Beschränkung entweder durch die Verbesserung der Router-Architekturen oder durch die Einführung von Hilfsverlustbedingungen angegangen, die die Ähnlichkeit unter Experten implizit bestrafen.
Machine translation (Meta M2M100); the English original follows.
Sparse Mixture-of-Experts (SMoE) models improve scalability by routing input tokens to only a subset of expert networks, but they often suffer from representation collapse, in which multiple experts converge toward similar functions and lose their intended diversity. Prior work has addressed this limitation either by improving router architectures or by introducing auxiliary loss terms that implicitly penalize similarity among experts. However, router-based approaches remain vulnerable to routing collapses, and loss-based approaches can conflict with the task's original objective during optimi
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.29355, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange