Newest papers

BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models

Ning Zhang, Shihao Wang, Jinrui Zhang, Chaodong Xiao, Lei Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28500 · cited by 0 works (OpenAlex) · open-access PDF

German

Für die effiziente Bereitstellung von BitCLfix-Language-Modellen (LLMs) reduziert BitCLNet die Methoden von Ultra-Low-Bit-Backbones die Gewichtsspeicherung, den Speicherverkehr und die Linear-Layer-Ausführungskosten, während diese Methoden der Multi-Token-Vorhersage (MTP) die aufeinanderfolgenden Dekodierungsschritte reduzieren, indem sie mehrere zukünftige Token parallel vorhersagen. Obwohl die direkte Kombination dieser Methoden auf vergleichbaren Qualitäten die Dekodierung nicht beschleunigen kann, schlagen wir BitMTP, ein algorithmisches Co-Design-Framework für effiziente MTPs auf ultra-Low-Bit LLM-Backbones, vor, während die breitere Berech

Machine translation (Meta M2M100); the English original follows.

Abstract

For efficient deployment of large language models (LLMs), methods of ultra-low-bit backbones reduce weight storage, memory traffic, and linear-layer execution cost, while multi-token prediction (MTP) methods reduce sequential decoding steps by predicting multiple future tokens in parallel. Although complementary, the direct combination of these methods cannot accelerate decoding at comparable quality: masked drafting lacks intermediate token input supplied in next-token training, while wider computation and causal key-value (KV) updates can offset the savings from fewer steps. To address these challenges, we propose BitMTP, an algorithm-system co-design framework for efficient MTP on ultra-low-bit LLM backbones. A joint next-token prediction and MTP objective equips a single model with both autoregressive (AR) generation and block-causal drafting. Confidence-only decoding, speculative-only decoding, and hybrid decoding provide different quality-throughput operating points without using separate draft and verifier models. Accepted-prefix alignment keeps the committed prefix, materialized causal KV boundary, and CUDA Graph replay state consistent for continued generation. BitMTP uses this contract to select width-specialized low-bit execution and reuse captured graphs over shared KV storage. On NVIDIA H20, BitMTP with hybrid decoding achieves five-task mean speedups of 2.2x on BitNet b1.58-2B and 2.1-2.4x on 1.7B-4B Bonsai models at comparable task quality. On RTX 3090 and Jetson Orin, it further achieves up to 2.0x in five-task mean throughput and 3.3x on MBPP+, respectively. These results demonstrate that BitMTP accelerates ultra-low-bit LLM generation across backbones, model sizes, and GPU types. The source code is released at https://github.com/PolyU-VCLab/BitMTP

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Decoding methods, Security token, Reuse, Language model, Offset (computer science)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange