Newest papers

TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning

Liana Patel, Rishabh Ranjan, Ion Gabriel Stoica, Matei Zaharia, Carlos Guestrin · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26754 · cited by 0 works (OpenAlex) · open-access PDF

Korean

다른 한편으로는 언어 모델은 각 모델 레이어를 통해 모든 입력 토큰을 처리하여 긴 컨텍스트 입력에 대한 상당한 처리 비용을 발생시킵니다. 우리는 토큰 절단을위한 비 차별화 할 수없는 Top-K 운영자에도 불구하고 끝에서 끝까지 학습을 허용하기 위해 빠르고 메모리 효율적인 프리플을 위해 긴 컨텍스트 입력을 통해 끝에서 끝까지 학습 가능한 토큰 절단을 허용하기 위해 기존 훈련 된 모델을 확장하는 작은 drop-in 구성 요소인 TokenGate를 도입합니다. 우리는 토큰을 사용하여 토큰 중요성 점수를 사용하여 토큰 점수를 재중계하는 엔드-to-엔드 학습 토큰 절단을 허용하는 메커니즘을 도입합니다. 다양한 작업을 통해, TokenGate는

Machine translation (Meta M2M100); the English original follows.

Abstract

Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters. On the other hand, language models process all input tokens through each model layer, resulting in significant processing costs for long-context inputs. We introduce TokenGate, a small, drop-in component that extends existing trained models to enable end-to-end learnable token pruning over long-context inputs for fast and memory-efficient prefill. To enable end-to-end learning despite the non-differentiable top-K operator for token pruning, we introduce token-gated attention, a mechanism inspired by Mixture-of-Experts that re-weights attention logits using token importance scores. Across diverse tasks, TokenGate achieves $1.4$-$11.3\times$ faster time-to-first token (TTFT) and $1.5$-$3.4\times$ lower peak memory at no accuracy degradation for prefill in transformers. Compared to the best-performing prior methods for token-pruning, TokenGate gives up to $6.4\times$ faster prefill and $2\times$ lower peak memory. TokenGate natively extends both standard transformers (e.g., Qwen3) and hybrid-attention models (e.g., Qwen3.5) with simple lightweight adaptation comparable to standard parameter-efficient fine-tuning. We make our code available at: https://github.com/tokengate-models/TokenGate.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Security token, Computer science, Pruning, Artificial intelligence, Process (computing), Language model

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange