Newest papers

SQuAT: Spherical Lattice Vector Quantization-Aware Training

Sebastian Loeschcke, Ivan Koryakovskiy, Tycho F. A. van der Ouderaa, Riccardo Del Chiaro, Mart van Baalen, Paul N. Whatmough, Markus Nagel · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31412 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

对低位大语言模型(LLMs)提供强劲的速率扭曲交换(SOQ),特别是基于线程的VQ指令量化(VQ-QAT),但它们的行为在定量化意识训练(VQ-QAT)下仍然不太理解。我们将线程VQ-QAT扩展到数十亿参数的LLM和数十亿美元的标记训练模式,并引入SQuAT,一种以单独的直径估计器(STE)训练的形状获取的形式获取的VQ-QAT方法。SQuAT将重量分解为方向和大小:标准化的方向被量化使用$E_$8衍生的球代码书,而获取量化是独立地通过低位量化量化器(QQAT)进行的。我们表明

Machine translation (Meta M2M100); the English original follows.

Abstract

Vector quantization (VQ), in particular lattice-based methods, offer strong rate-distortion trade-offs for low-bit large language models (LLMs), yet their behavior under quantization-aware training (VQ-QAT) remains poorly understood. We scale lattice VQ-QAT to billion-parameter LLMs and multi-billion-token training regimes, and introduce SQuAT, a shape-gain VQ-QAT approach trained with a separate straight-through estimator (STE) for the shape and gain. SQuAT decomposes weights into direction and magnitude: normalized directions are quantized using an $E_8$-derived spherical codebook, while gains are quantized independently with low-bit scalar quantizers. We show that, unlike a standard STE applied to the full quantizer, which ignores the shape-gain structure, these separate STEs induce tangent-space updates for directions, yielding an implicit form of spherical (Riemannian) optimization. To enable scalability, we develop a distributed exact nearest-neighbor assignment scheme, in which workers quantize disjoint weight blocks and communicate only discrete indices, reducing VQ-QAT training time by 12.8$\times$ and allowing us to scale it to 10B tokens. Across instruction-tuned LLaMA-3.2-1B/3B, Qwen3-4B, and base Qwen3-1.7B, SQuAT consistently outperforms standard VQ-QAT STE under identical quantizers and initialization, and achieves SOTA VQ-QAT accuracy.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Disjoint sets, Lattice (music), Estimator, Training (meteorology), Quantization (signal processing), Mathematics

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange