Newest papers

Rethinking Sequence Compression with Test-time Regression

Sihan Yang, Jie Huang, Hanxuan Li, Yifan Guo, Weigao Sun, Yiran Zhong, Weiyang Liu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31574 · cited by 0 works (OpenAlex) · open-access PDF

Korean

시퀀스 압축은 효율적인 언어 및 멀티 모드 모델에서 근본적인 작업이며 중요한 정보 병점이며, 그럼에도 불구하고 폴링 또는 배운 예측에 근거한 주도적인 접근 방식은 여전히 실패합니다. 우리는 시험 시간 회귀의 렌즈를 통해 시퀀스 압축을 재검토합니다 : 입력 토큰은 압축 요소가 압축 된 표현을 형성하는 회귀 문제를 정의합니다. 중요한 것은 압축 문제는 백프로파케이션을 통해 백프로파케이션을 통해 끝에서 끝으로 배운 반면 그 요소는 닫힌 형태로 효율적으로 계산됩니다. 이것은 전통적인 폴링 및 프로젝션 기반 압축에 대한 일반적인 대안 인 TTR-Compressor로 이어집니다. 우리는 TTR-Compressor를 블록

Machine translation (Meta M2M100); the English original follows.

Abstract

Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. Importantly, the regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional pooling and projection-based compression. We evaluate TTR-Compressor as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models. On block-wise sparse attention, TTR-Compressor improves RULER performance over attention-based compressors by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens. These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Pooling, Sequence (biology), Computer science, Compression (physics), Data compression, Regression

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange