Newest papers

Low-Bit N-Gram Embeddings for Language Models

Stephen Chung, Wenyu Du · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31646 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Muchos modelos de lenguaje de extracto abierto recientes utilizan tablas de embalaje de n-gram para mejorar el rendimiento. Estas tablas almacenan representaciones aprendidas de secuencias de token cortas, como pares o triples de tokens consecutivos, que el modelo recupera durante el procesamiento. Sin embargo, pueden requerir una memoria sustancial: DeepSeek-V4.1-Flash, por ejemplo, asigna 196 mil millones de parámetros a su memoria de Engram n-gram. Por lo tanto, investigamos si este coste de memoria se puede reducir al mismo tiempo que se conserva la mejora del rendimiento que proporcionan las tablas. Proponemos almacenar las embalajes de n-gram a baja precisión y investigar sistemáticamente diferentes métodos de cuantización y todos los bitocations en tablas. A través de las tablas

Machine translation (Meta M2M100); the English original follows.

Abstract

Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory. We therefore investigate whether this memory cost can be reduced while preserving the performance improvement that the tables provide. We propose storing n-gram embeddings at low precision and systematically investigate different quantization methods and bit allocations across tables. Across nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables save approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from adding the tables. Using fewer bits leads to a larger drop in performance, although we find that a short fine-tuning phase recovers a significant part of this loss. In addition, reinvesting the saved memory in larger tables can improve performance beyond that of the original full-precision tables while still using less memory. These table memory savings also extend to large existing models: on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables save 47-55% of table memory, with perplexity increases of less than 1.2% and largely preserved downstream task performance. These findings establish low-bit precision as a practical way to reduce the memory requirements of n-gram embeddings and use the saved capacity to improve model performance.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Perplexity, Language model, Table (database), Task (project management), Embedding

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange