Newest papers

BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing

Hayden Prairie, Jacob M. Ryan, Taylor Berg-Kirkpatrick, Daniel Y. Fu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31633 · cited by 0 works (OpenAlex) · open-access PDF

German

Während der Inferenz erzeugt jede Wiederholung unabhängige KV-Caches, wodurch das Gedächtnis linear wächst. Um diese Herausforderung anzugehen, haben die vorherigen Arbeiten die Mechanismen der KV-Sharing vorgeschlagen (d. h. das Teilen von KV-Caches über die Tiefe), wobei in einigen Fällen der KV-Caches der endgültigen Wiederholung nicht konsequent dargestellt werden kann. In diesem Papier untersuchen wir, wie die KV-Caches mit den früheren Wiederholungen ohne Qualitätsverlust erlaubt werden können, eine Eigenschaft, die wir als KV-Sharing bezeichnen. Zur Bewältigung dieser Herausforderung sind die Mechanismen der KV-Sharing (d. h. das Teilen von KV-Caches

Machine translation (Meta M2M100); the English original follows.

Abstract

Looped architectures scale quality while keeping parameter count fixed by repeatedly passing activations through the same block of layers. During inference, each recurrence generates independent KV-caches, causing memory to grow linearly. To address this challenge, prior work has proposed KV-sharing (i.e., sharing KV-caches across depth), finding that, in some cases, the KV-cache of the final recurrence can be shared with the earlier recurrences without quality loss, a property we denote as KV-shareability. However, the mechanisms underlying KV-shareability remain poorly understood, with even identically trained looped architectures not consistently exhibiting it. In this paper, we study how KV-shareability can be enabled in looped models. We propose a two-stage method: in the first stage, we use stochastic depth during pretraining, which induces contractive residual dynamics that drive representations toward a common limit across recurrences. In the second stage, we propose BeNice!, a post-training method that broadcasts the final recurrence's KV-cache into the first recurrence, training it to produce compatible residuals while preserving the original model's quality. As a result, BeNice! enables KV-shareability while matching standard finetuning's unshared validation perplexity. Using BeNice!, we show that KV-sharing is complementary to alternative KV-compression methods, composing with memory-efficient sequence mixers (e.g., multi-latent attention, Kimi delta attention) and low-precision KV-cache quantization (e.g., fp8, mxfp8, nvfp4). Finally, in simulated low-memory, long-context inference, KV-sharing improves cache hit rates by up to 31.4%, increases throughput by up to 23.6%, and reduces p90 time-to-first-token by up to 83.6%.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Quantization (signal processing), Computer science, Residual, Property (philosophy), Block (permutation group theory), Algorithm

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange