Newest papers

GaP-MoE : Explicit Expert Specialization in Mixture-of-Experts via Orthogonal Gradient Accumulation and Projection

Donghyun Jeong, Yoon-Yeong Kim · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29355 · cited by 0 works (OpenAlex) · open-access PDF

Korean

스파르세 텍스트 혼합 전문가 (SMoE) 모델은 단지 전문가 네트워크의 하위 세트에 게이트 토큰을 라우팅함으로써 확장성을 향상시킵니다. 그러나 그들은 종종 표현 작업의 붕괴로 고통 받고 있으며, 여러 전문가가 유사한 기능을 향해 합병하여 의도된 다양성을 잃을 수 있습니다. 이전 작업은 라우터 아키텍처를 개선하거나 전문가들 사이의 유사성을 암시적으로 처벌하는 보조 손실 용어를 도입함으로써 확장성을 향상시켰습니다. 그러나 라우터 기반 접근 방식은 라우팅 붕괴에 취약하며 손실 기반 접근 방식은 최적화 과정에서 작업의 원래 목표와 충돌 할 수 있습니다. 이 작업에서 우리는 라우터 아키텍처에 GaP

Machine translation (Meta M2M100); the English original follows.

Abstract

Sparse Mixture-of-Experts (SMoE) models improve scalability by routing input tokens to only a subset of expert networks, but they often suffer from representation collapse, in which multiple experts converge toward similar functions and lose their intended diversity. Prior work has addressed this limitation either by improving router architectures or by introducing auxiliary loss terms that implicitly penalize similarity among experts. However, router-based approaches remain vulnerable to routing collapses, and loss-based approaches can conflict with the task's original objective during optimization. In this work, we propose GaP-MoE (Gradient Accumulation and Projection in MoE), which induces expert specialization by directly manipulating expert output gradients rather than modifying the loss function. At each training step, GaP-MoE stochastically selects one of the $TopK$ routed experts as a leader based on normalized routing scores, and projects the gradients of the remaining follower experts onto the orthogonal complement of the subspace spanned by the leader's recent gradients. This removes directional overlap between expert gradients without altering the task loss, allowing specialization to emerge as a direct consequence of the optimization dynamics rather than an externally imposed constraint. We evaluate GaP-MoE in both fine-tuning settings (BERT-based, on GLUE) and from-scratch pretraining settings (GPT-2-based, on WikiText-103, enwik8, and text8), and further confirm its compatibility as a drop-in module for existing SMoE architectures. Experimental results show that GaP-MoE achieves performance comparable to or better than standard SMoE.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Scalability, Subspace topology, Complement (music), Projection (relational algebra), Task (project management)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange