Newest papers

Low-Bit N-Gram Embeddings for Language Models

Stephen Chung, Wenyu Du · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31646 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Banyak model bahasa extplex terbuka baru-baru ini menggunakan tabel embedding n-gram untuk meningkatkan kinerja. Tabel ini menyimpan representasi yang dipelajari dari urutan token pendek, seperti pasangan atau tiga token berturut-turut, yang diperoleh model selama proses. Namun, mereka dapat membutuhkan memori substansial: DeepSeek-V4.1-Flash, misalnya, mendistribusikan 196 miliar parameter ke memori Engram n-gram. Oleh karena itu kami menyelidiki apakah biaya memori ini dapat dikurangi sambil mempertahankan peningkatan kinerja yang disediakan tabel. Kami menyarankan menyimpan embeddings n-gram dengan presisi rendah dan secara sistematis menyelidiki metode kuantizasi yang berbeda dan semua bitocations di seluruh tabel. Melalui nanochat, Llama, dan model keluarga GPT-2 yang dilatih dari awal, tabel empat-bit ini dapat menghemat sekitar 75% dari

Machine translation (Meta M2M100); the English original follows.

Abstract

Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory. We therefore investigate whether this memory cost can be reduced while preserving the performance improvement that the tables provide. We propose storing n-gram embeddings at low precision and systematically investigate different quantization methods and bit allocations across tables. Across nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables save approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from adding the tables. Using fewer bits leads to a larger drop in performance, although we find that a short fine-tuning phase recovers a significant part of this loss. In addition, reinvesting the saved memory in larger tables can improve performance beyond that of the original full-precision tables while still using less memory. These table memory savings also extend to large existing models: on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables save 47-55% of table memory, with perplexity increases of less than 1.2% and largely preserved downstream task performance. These findings establish low-bit precision as a practical way to reduce the memory requirements of n-gram embeddings and use the saved capacity to improve model performance.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Perplexity, Language model, Table (database), Task (project management), Embedding

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange