Newest papers

Competing Learning in Hybrid Architectures

Xingyu Shen, Yingfa Chen, Ziqing Qiao, Chaojun Xiao, Xu Han, Zhiyuan Liu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26830 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Muchos modelos de lenguaje modernos han adoptado arquitecturas híbridas que incorporan tanto la recurrencia lineal-tiempo como la atención softmax. Las capas de recurrencia modernas proporcionan un mecanismo de mezcla altamente expresivo al asegurar un alto rendimiento comprimiendo la información contextual en un estado de tamaño fijo, pero subperforman la atención softmax en la recuperación de información contextual. En este artículo, revelamos un fenómeno de aprendizaje competitivo donde diferentes tipos de mezcladores compiten para aprender la misma función, y cuando cualquier capa aprende una función, suprime el mensaje de aprendizaje correspondiente para otras capas. El análisis de atribución de capas revela que bajo las distribuciones de datos típicas, la recurrencia moderna aprende funciones de recuperación más rápido que la atención softmax, causando la atención a no aprender la recuperación en

Machine translation (Meta M2M100); the English original follows.

Abstract

Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval. In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete to learn the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers. Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Softmax function, Computer science, Artificial intelligence, Context (archaeology), Layer (electronics), Machine learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange