Newest papers

GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation

Quang Anh Kiet Tran, Nam Anh Tran, Trinh Manh Hung, Thanh Danh Le, Khang Nhat Hoang Vo, Huynh Dai Nhan Tran · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30067 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

Os codificadores pré-treinados vietnamitas melhoraram rapidamente, mas os mais fortes são grandes, modelos de vocabulário multilíngue ou dependem de segmentação de palavras externas, o que os torna caros de implantar. Nós apresentamos GoiBERT, uma família de dois codificadores vietnamitas compactos com parâmetros 26M e 49M que operam diretamente em texto bruto. GoiBERT aprende de dois professores complementares. Um codificador vietnamita do mesmo nome ensina primeiramente representações de propósito geral. Um professor muito maior fornece orientação de nível de tarefas através de rótulos suaves que são gerados sem nunca rotular seus próprios dados de treinamento. Ao longo de oito benchmarks que cobrem classificação de texto, análise de sentimento baseada em aspectos, inferência de linguagem

Machine translation (Meta M2M100); the English original follows.

Abstract

Vietnamese pre-trained encoders have improved rapidly, but the strongest ones are large, multilingual-vocabulary models or depend on external word segmentation, which makes them costly to deploy. We present GoiBERT, a family of two compact Vietnamese encoders with 26M and 49M parameters that operate directly on raw text. GoiBERT learns from two complementary teachers. A same-tokenizer Vietnamese encoder first teaches general-purpose representations. A much larger teacher then provides task-level guidance through soft labels that are generated without ever labeling its own training data. Across eight benchmarks covering text classification, aspect-based sentiment analysis, natural language inference, and named entity recognition, GoiBERT-large achieves the highest average score among non-teacher models. It does so with fewer than half the parameters of the strongest compact baseline and twice its throughput. GoiBERT-base retains 99% of that baseline's performance at a quarter of its size. Our analysis shows that compression is nearly lossless for sentiment and classification tasks but remains costly for inference and entity recognition, which points to clear directions for future work. Our code and models are available at: https://github.com/Nhantran65/GoiBERT.git

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Vietnamese, Computer science, Encoder, Natural language processing, Artificial intelligence, Lossless compression

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange