Newest papers

BitMTP: When Multi-Token Prediction Meets Low-Bit Large Language Models

Ning Zhang, Shihao Wang, Jinrui Zhang, Chaodong Xiao, Lei Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28500 · cited by 0 works (OpenAlex) · open-access PDF

Korean

BitCLfix 1.7은 대형 언어 모델 (LLMs)의 효율적인 배포를 위해 BitCLNet를 선택합니다.Ultra-low-bit backbones의 방법은 체중 저장, 메모리 트래픽 및 선형 레이어 실행 비용을 감소시키는 반면, 다중 토큰 예측 (MTP) 방법은 여러 미래 토큰을 동시에 예측함으로써 연속적인 디코딩 단계를 감소시킵니다. 비록 이러한 도전에 대한 직접적인 조합은 비교할 수있는 품질의 디코딩을 가속화 할 수는 없지만, 마스크 드래핑은 다음 토큰 훈련에서 제공되는 중간 토큰 입력이 부족하지만, 광범위한 계산 및 MB의 원인 키 값 (KV) 업데이트는 여러 단계에서 저축을 보상 할 수

Machine translation (Meta M2M100); the English original follows.

Abstract

For efficient deployment of large language models (LLMs), methods of ultra-low-bit backbones reduce weight storage, memory traffic, and linear-layer execution cost, while multi-token prediction (MTP) methods reduce sequential decoding steps by predicting multiple future tokens in parallel. Although complementary, the direct combination of these methods cannot accelerate decoding at comparable quality: masked drafting lacks intermediate token input supplied in next-token training, while wider computation and causal key-value (KV) updates can offset the savings from fewer steps. To address these challenges, we propose BitMTP, an algorithm-system co-design framework for efficient MTP on ultra-low-bit LLM backbones. A joint next-token prediction and MTP objective equips a single model with both autoregressive (AR) generation and block-causal drafting. Confidence-only decoding, speculative-only decoding, and hybrid decoding provide different quality-throughput operating points without using separate draft and verifier models. Accepted-prefix alignment keeps the committed prefix, materialized causal KV boundary, and CUDA Graph replay state consistent for continued generation. BitMTP uses this contract to select width-specialized low-bit execution and reuse captured graphs over shared KV storage. On NVIDIA H20, BitMTP with hybrid decoding achieves five-task mean speedups of 2.2x on BitNet b1.58-2B and 2.1-2.4x on 1.7B-4B Bonsai models at comparable task quality. On RTX 3090 and Jetson Orin, it further achieves up to 2.0x in five-task mean throughput and 3.3x on MBPP+, respectively. These results demonstrate that BitMTP accelerates ultra-low-bit LLM generation across backbones, model sizes, and GPU types. The source code is released at https://github.com/PolyU-VCLab/BitMTP

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Decoding methods, Security token, Reuse, Language model, Offset (computer science)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange