Newest papers

GoiBERT: Compact Vietnamese Language Models via Two-Stage Knowledge Distillation

Quang Anh Kiet Tran, Nam Anh Tran, Trinh Manh Hung, Thanh Danh Le, Khang Nhat Hoang Vo, Huynh Dai Nhan Tran · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30067 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

越南预先训练的编码器已经快速改进,但最强大的是大型、多语言词汇模型或依赖于外部单词细分,这使得部署成本昂贵。我们介绍了GoiBERT,一个拥有26M和49M参数的两家紧凑的越南编码器家族,直接在原文上运作。GoiBERT从两个互补的教师那里学习。同一个代码器的越南编码器首先教授一般用途的代表性。一个更大的教师然后通过软标签提供任务水平的指导,而没有创建自己的培训数据标签。通过八个基准,涵盖文本分类,基于方面情绪分析,自然语言推断,命名实体识别,GoiBERT-large首先在非教师模型中获得最高的平均分数。它通过强

Machine translation (Meta M2M100); the English original follows.

Abstract

Vietnamese pre-trained encoders have improved rapidly, but the strongest ones are large, multilingual-vocabulary models or depend on external word segmentation, which makes them costly to deploy. We present GoiBERT, a family of two compact Vietnamese encoders with 26M and 49M parameters that operate directly on raw text. GoiBERT learns from two complementary teachers. A same-tokenizer Vietnamese encoder first teaches general-purpose representations. A much larger teacher then provides task-level guidance through soft labels that are generated without ever labeling its own training data. Across eight benchmarks covering text classification, aspect-based sentiment analysis, natural language inference, and named entity recognition, GoiBERT-large achieves the highest average score among non-teacher models. It does so with fewer than half the parameters of the strongest compact baseline and twice its throughput. GoiBERT-base retains 99% of that baseline's performance at a quarter of its size. Our analysis shows that compression is nearly lossless for sentiment and classification tasks but remains costly for inference and entity recognition, which points to clear directions for future work. Our code and models are available at: https://github.com/Nhantran65/GoiBERT.git

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Vietnamese, Computer science, Encoder, Natural language processing, Artificial intelligence, Lossless compression

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange