Newest papers

Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation

Hongcan Guo, Yian Zhao, Feng Wang, Shen Nie, Guoqiang Wei, Yan Zeng, Hengshuang Zhao · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31305 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

我们提出Cola2,一个框架,共同研究连续的语义压缩,重构和生成之间的平衡,以及连续的语义互适应,但它的有效性取决于隐形空间压缩的内容,以及隐形发电机如何使用它。我们提出Cola2,一个框架,该框架共同研究连续的语义压缩,连接重建和生成之间的平衡,以及连续的语义互适应。Cola2动态地将文本分为语义范围,通过辅助性损失的负面反馈来控制压缩率,并结合重建和生成监督,以学习紧凑的、预测性隐形表示。一个区块链伪装的散发解解码器被训练在清洁、噪声和落下的后期任务条件下,而分类器自由指导则调整了语义重建

Machine translation (Meta M2M100); the English original follows.

Abstract

Hierarchical language generation separates continuous semantic modeling from discrete text realization, but its effectiveness depends on what the latent space compresses and how the discrete generator uses it. We propose Cola2, a framework that jointly studies continuous semantic compression, the balance between reconstruction and generation, and continuous-discrete co-adaptation. Cola2 dynamically segments text into semantic spans, controls the compression rate through auxiliary-loss-free negative feedback, and combines reconstruction and generation supervision to learn compact, predictive latent representations. A block-causal masked diffusion decoder is trained with clean, noisy, and dropped latent conditions, while classifier-free guidance adjusts the balance between semantic reconstruction and autonomous generation. Token-prediction gradients update the encoder through the latents, whereas Flow Matching prior learning is decoupled from representation learning by stop-gradient. During inference, generated text is re-encoded under a fixed span layout to keep the latent history consistent with the actual text context. Experiments on five downstream tasks with about 210 billion training tokens validate the core designs and show competitive, unsaturated scaling. With 346.03 billion source tokens, an unconverged checkpoint reaches a 60.10 average across eight tasks and surpasses Llama-3-8B on HellaSwag and StoryCloze despite its reported pretraining budget being over 43 times larger. These results demonstrate the potential of continuous-latent hierarchical language generation.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Language model, Artificial intelligence, Generator (circuit theory), Natural language processing, Encoder

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange