Newest papers

Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder

Cheng Liang, Tengfei Zhang, Wenjie DU, Jianyu Wu, Changlian Ma, Shuchen Pu, Ya Zhang, Yanfeng Wang, Zhouhan Lin · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28489 · cited by 0 works (OpenAlex) · open-access PDF

Korean

Looped Context Transformers는 공유 레이어를 반복적으로 적용함으로써 효과적인 깊이를 증가시키고, 효과적인 매개 변수를 증가시키지 않고도 계산을 확대하는 방법을 제공합니다. 그러나, 전체 스택 루프는 같은 반복적 계산 내에서 컨텍스트 건설 및 반복적 상태 정화 (KV) 캐시를 구현하며, 역할을 명시적으로 구별하지 않고 효과적인 매개 변수 (AsymLoop)를 소개합니다. 단일 패스 컨텍스트 인코딩을 단일 상태 정화에서 분리합니다. 단일 패스 원인 인코더는 계층별 글로벌 키 값 (KV) 캐시를 생성합니다. 무게가있는 반복적 인 디코더는 진화하는 상태에서 로컬 KV를 생성합니다. 단일 소프트맥스 내에서 단일

Machine translation (Meta M2M100); the English original follows.

Abstract

Looped Transformers increase effective depth by repeatedly applying shared layers, offering a way to scale computation without increasing parameter count. However, full-stack looping places context construction and iterative state refinement within the same recurrent computation, without explicitly distinguishing their roles. We introduce the Asymmetric Looped Transformer (AsymLoop), which separates single-pass context encoding from recurrent state refinement. A single-pass causal encoder produces layer-specific Global key-value (KV) caches that remain fixed across iterations, while a weight-tied recurrent decoder generates Local KV from its evolving state. Joint attention over both sources under a single softmax allows each update to integrate persistent contextual information with the current intermediate state. For training, we apply a heavy-tail-guided layerwise learning-rate method to the encoder and decoder. Experiments on FineWeb-Edu across four model scales, from 60M to 1B, show consistently lower validation perplexity and higher average downstream accuracy than Full Loop under matched effective depth and training token budgets. At 1B, AsymLoop improves average zero-shot accuracy across seven commonsense reasoning benchmarks from 49.58% to 50.85%. Further experiments extend these benefits to longer training budgets, an OLMo-style backbone, and sparse mixture-of-experts models. These results support asymmetric context encoding and state refinement as an effective design for recurrent language models.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Perplexity, Softmax function, Computer science, Encoder, Language model, Algorithm

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange