Sebastian Loeschcke, Ivan Koryakovskiy, Tycho F. A. van der Ouderaa, Riccardo Del Chiaro, Mart van Baalen, Paul N. Whatmough, Markus Nagel · 2026 · arXiv
DOI: 10.48550/arXiv.2610.31412 · open-access PDF
Vektorquantization (VQ), insbesondere gridbasierte Methoden, bieten starke Rate-Distortion-Kompromisse für Low-Bit-Big-Sprachmodelle (LLMs), aber ihr Verhalten unter Quantization-aware-Training (VQ-QAT) bleibt schlecht verstanden. Wir skalieren gridbasierte VQ-QAT zu Milliarden-Parameter-LLMs und Multi-Billion-Token-Training-Regimes und führen SQuAT ein, einen formgewinnenden VQ-QAT-Ansatz, der mit einem separaten Straight-Through-Estimator (STE) für Form und Gewinn trainiert wird.
Machine translation (Meta M2M100); the English original follows.
Vector quantization (VQ), in particular lattice-based methods, offer strong rate-distortion trade-offs for low-bit large language models (LLMs), yet their behavior under quantization-aware training (VQ-QAT) remains poorly understood. We scale lattice VQ-QAT to billion-parameter LLMs and multi-billion-token training regimes, and introduce SQuAT, a shape-gain VQ-QAT approach trained with a separate straight-through estimator (STE) for the shape and gain. SQuAT decomposes weights into direction and magnitude: normalized directions are quantized using an $E_8$-derived spherical codebook, while gai
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.31412, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange