Newest papers

BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models

Ning Zhang, Shihao Wang, Jinrui Zhang, Chaodong Xiao, Lei Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28500 · cited by 0 works (OpenAlex) · open-access PDF

Russian

Для эффективного развертывания BitCLfix с помощью больших языковых моделей (LLM) методы Ultra-low-bit backbones снижают вес хранения, трафик памяти и затраты на выполнение линейного слоя, в то время как методы Multi-token prediction (MTP) уменьшают последовательные шаги декодирования, прогнозируя несколько будущих токенов параллельно. Хотя в дополнение к этим вызовам, прямая комбинация этих методов не может ускорить декодирование по сопоставимому качеству: в маскированном проектировании не хватает ввода промежуточных токенов, поставляемых в обучение следующего токена, в то время как более широкие вычисления и обновления причинного ключа MB (KV) могут компенсировать сбережения от меньшего

Machine translation (Meta M2M100); the English original follows.

Abstract

For efficient deployment of large language models (LLMs), methods of ultra-low-bit backbones reduce weight storage, memory traffic, and linear-layer execution cost, while multi-token prediction (MTP) methods reduce sequential decoding steps by predicting multiple future tokens in parallel. Although complementary, the direct combination of these methods cannot accelerate decoding at comparable quality: masked drafting lacks intermediate token input supplied in next-token training, while wider computation and causal key-value (KV) updates can offset the savings from fewer steps. To address these challenges, we propose BitMTP, an algorithm-system co-design framework for efficient MTP on ultra-low-bit LLM backbones. A joint next-token prediction and MTP objective equips a single model with both autoregressive (AR) generation and block-causal drafting. Confidence-only decoding, speculative-only decoding, and hybrid decoding provide different quality-throughput operating points without using separate draft and verifier models. Accepted-prefix alignment keeps the committed prefix, materialized causal KV boundary, and CUDA Graph replay state consistent for continued generation. BitMTP uses this contract to select width-specialized low-bit execution and reuse captured graphs over shared KV storage. On NVIDIA H20, BitMTP with hybrid decoding achieves five-task mean speedups of 2.2x on BitNet b1.58-2B and 2.1-2.4x on 1.7B-4B Bonsai models at comparable task quality. On RTX 3090 and Jetson Orin, it further achieves up to 2.0x in five-task mean throughput and 3.3x on MBPP+, respectively. These results demonstrate that BitMTP accelerates ultra-low-bit LLM generation across backbones, model sizes, and GPU types. The source code is released at https://github.com/PolyU-VCLab/BitMTP

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Decoding methods, Security token, Reuse, Language model, Offset (computer science)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange