Newest papers

BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models

Ning Zhang, Shihao Wang, Jinrui Zhang, Chaodong Xiao, Lei Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28500 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Para el despliegue eficiente de BitCLfix por medio de los grandes modelos de lenguaje (LLMs), los métodos de backbones de BitCLNet reducen el peso del contrato, el tráfico de memoria y el coste de ejecución de la capa lineal, mientras que estos métodos de predicción de token múltiple (MTP) reducen los pasos de descifrado secuencial mediante la predicción de los tokens futuros múltiples en paralelo. Aunque complementarios, la combinación directa de estos métodos no puede acelerar el descifrado a una calidad comparable: el diseño enmascarado carece de la entrada de token intermedio suministrada en el próximo entrenamiento de token, mientras que la computación más amplia y las actualizaciones de valor clave causal (KV) de MBI pueden compensar los ahorros de menos pasos. Para

Machine translation (Meta M2M100); the English original follows.

Abstract

For efficient deployment of large language models (LLMs), methods of ultra-low-bit backbones reduce weight storage, memory traffic, and linear-layer execution cost, while multi-token prediction (MTP) methods reduce sequential decoding steps by predicting multiple future tokens in parallel. Although complementary, the direct combination of these methods cannot accelerate decoding at comparable quality: masked drafting lacks intermediate token input supplied in next-token training, while wider computation and causal key-value (KV) updates can offset the savings from fewer steps. To address these challenges, we propose BitMTP, an algorithm-system co-design framework for efficient MTP on ultra-low-bit LLM backbones. A joint next-token prediction and MTP objective equips a single model with both autoregressive (AR) generation and block-causal drafting. Confidence-only decoding, speculative-only decoding, and hybrid decoding provide different quality-throughput operating points without using separate draft and verifier models. Accepted-prefix alignment keeps the committed prefix, materialized causal KV boundary, and CUDA Graph replay state consistent for continued generation. BitMTP uses this contract to select width-specialized low-bit execution and reuse captured graphs over shared KV storage. On NVIDIA H20, BitMTP with hybrid decoding achieves five-task mean speedups of 2.2x on BitNet b1.58-2B and 2.1-2.4x on 1.7B-4B Bonsai models at comparable task quality. On RTX 3090 and Jetson Orin, it further achieves up to 2.0x in five-task mean throughput and 3.3x on MBPP+, respectively. These results demonstrate that BitMTP accelerates ultra-low-bit LLM generation across backbones, model sizes, and GPU types. The source code is released at https://github.com/PolyU-VCLab/BitMTP

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Decoding methods, Security token, Reuse, Language model, Offset (computer science)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange