Newest papers

Competing Learning in Hybrid Architectures

Xingyu Shen, Yingfa Chen, Ziqing Qiao, Chaojun Xiao, Xu Han, Zhiyuan Liu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26830 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

多くの現代言語モデルは、線形時間の再現とソフトマックスの注意を組み込むハイブリッドアーキテクチャを採用しています。現代の再現層は非常に表現的な混合メカニズムを提供し、コンテキスト情報を固定サイズの状態に圧縮することによって高パフォーマンスを確保していますが、コンテキスト情報のリクエストでソフトマックスの注意を低性能にします。この論文では、さまざまな種類のミキサーが同じ機能を学ぶために競合する競合する競合学習現象を明らかにします。どの層が機能を学ぶと、それは他の層のための相応の学習信号を抑制します。レイヤー属性分析は、典型的なデータ配布では、現代の再現がソフトマックスの

Machine translation (Meta M2M100); the English original follows.

Abstract

Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval. In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete to learn the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers. Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Softmax function, Computer science, Artificial intelligence, Context (archaeology), Layer (electronics), Machine learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange