Newest papers

BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models

Ning Zhang, Shihao Wang, Jinrui Zhang, Chaodong Xiao, Lei Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28500 · cited by 0 works (OpenAlex) · open-access PDF

French

Pour un déploiement efficace de BitCLfix via les grands modèles linguistiques (LLM), les méthodes de backbones ultra-low-bit réduisent le volume de stockage, le trafic de mémoire et les coûts d'exécution de la couche linéaire, tandis que ces méthodes de prédiction de jetons multiples (MTP) réduisent les étapes de décodage séquentiel en prédisant des jetons futurs multiples en parallèle. Bien que complémentaires, la combinaison directe de ces méthodes sur le contrat ne peut pas accélérer le décodage à une qualité comparable: le dessin masqué manque d'entrée de jeton intermédiaire fournie dans la prochaine formation de jetons, tandis que le calcul plus large et les mises à jour de la valeur de clé

Machine translation (Meta M2M100); the English original follows.

Abstract

For efficient deployment of large language models (LLMs), methods of ultra-low-bit backbones reduce weight storage, memory traffic, and linear-layer execution cost, while multi-token prediction (MTP) methods reduce sequential decoding steps by predicting multiple future tokens in parallel. Although complementary, the direct combination of these methods cannot accelerate decoding at comparable quality: masked drafting lacks intermediate token input supplied in next-token training, while wider computation and causal key-value (KV) updates can offset the savings from fewer steps. To address these challenges, we propose BitMTP, an algorithm-system co-design framework for efficient MTP on ultra-low-bit LLM backbones. A joint next-token prediction and MTP objective equips a single model with both autoregressive (AR) generation and block-causal drafting. Confidence-only decoding, speculative-only decoding, and hybrid decoding provide different quality-throughput operating points without using separate draft and verifier models. Accepted-prefix alignment keeps the committed prefix, materialized causal KV boundary, and CUDA Graph replay state consistent for continued generation. BitMTP uses this contract to select width-specialized low-bit execution and reuse captured graphs over shared KV storage. On NVIDIA H20, BitMTP with hybrid decoding achieves five-task mean speedups of 2.2x on BitNet b1.58-2B and 2.1-2.4x on 1.7B-4B Bonsai models at comparable task quality. On RTX 3090 and Jetson Orin, it further achieves up to 2.0x in five-task mean throughput and 3.3x on MBPP+, respectively. These results demonstrate that BitMTP accelerates ultra-low-bit LLM generation across backbones, model sizes, and GPU types. The source code is released at https://github.com/PolyU-VCLab/BitMTP

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Decoding methods, Security token, Reuse, Language model, Offset (computer science)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange