Newest papers

Competing Learning in Hybrid Architectures

Xingyu Shen, Yingfa Chen, Ziqing Qiao, Chaojun Xiao, Xu Han, Zhiyuan Liu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26830 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

许多现代语言模型采用了混合架构,其中包括线性时间重复和软max注意力。现代重复层提供了高度表达式的混合机制,同时通过将背景信息压缩到固定尺寸的状态来确保高输出,但在背景信息检索中低效的 softmax 注意力。在这篇论文中,我们揭示了一个竞争式学习现象,不同类型的混合器在学习相同功能时竞争,而当任何层学习函数时,它抑制了对其他层的相应学习信号。 层分配分析显示,在典型的数据分布中,现代重复学习会比软max 注意力更快地检索功能,导致注意力在大多数情况下无法检索。 在训练中中停止重复更新有助于学习注意力检索。 基于这些结果,我们提出了一个阶段化的

Machine translation (Meta M2M100); the English original follows.

Abstract

Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval. In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete to learn the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers. Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Softmax function, Computer science, Artificial intelligence, Context (archaeology), Layer (electronics), Machine learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange