Newest papers

Rethinking Sequence Compression with Test-time Regression

Sihan Yang, Jie Huang, Hanxuan Li, Yifan Guo, Weigao Sun, Yiran Zhong, Weiyang Liu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31574 · cited by 0 works (OpenAlex) · open-access PDF

French

La compression de séquence est une opération fondamentale et un bouchon d'information critique dans les modèles de langage efficaces et multimodaux, et pourtant les approches prédominantes basées sur le regroupement ou les projections apprises restent sous-performantes. Nous revoyons la compression de séquence par le biais de l'objectif de la régression du temps de test: les jetons d'entrée définissent un problème de régression dont les coefficients fixés forment la représentation comprimée. Important, le problème de régression est appris de bout en bout par la propagation arrière, tandis que ses coefficients sont calculés efficacement sous forme fermée pendant le passage en avant. Cela conduit à TTR-Compressor, une alternative générale au regroupement conventionnel

Machine translation (Meta M2M100); the English original follows.

Abstract

Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. Importantly, the regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional pooling and projection-based compression. We evaluate TTR-Compressor as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models. On block-wise sparse attention, TTR-Compressor improves RULER performance over attention-based compressors by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens. These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Pooling, Sequence (biology), Computer science, Compression (physics), Data compression, Regression

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange