Newest papers

Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation

Hongcan Guo, Yian Zhao, Feng Wang, Shen Nie, Guoqiang Wei, Yan Zeng, Hengshuang Zhao · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31305 · cited by 0 works (OpenAlex) · open-access PDF

Korean

잠재적 인 고정 언어 생성은 간단한 텍스트 실현에서 지속적인 서란틱 모델링을 분리하지만, 그 효과는 잠재적 인 공간을 압축하고 분리 된 생성자가 그것을 사용하는 방법에 따라 달라집니다. 우리는 Cola2를 제안합니다. 지속적인 서란틱 압축, 재구성 및 생성 사이의 균형을 공동으로 연구하는 프레임 워크. Cola2는 문자를 시멘틱 범위로 동적으로 분할하고, 보조 손실없는 부정적인 피드백을 통해 압축 속도를 조절하고, 컴팩트하고 예측적 인 늦은 표현을 배우기 위해 재구성 및 생성 감독을 결합합니다. Block-causal masked diffusion decoder는 깨끗하고 시끄럽고 늦은 작업 조건으로 훈련됩니다. Cola

Machine translation (Meta M2M100); the English original follows.

Abstract

Hierarchical language generation separates continuous semantic modeling from discrete text realization, but its effectiveness depends on what the latent space compresses and how the discrete generator uses it. We propose Cola2, a framework that jointly studies continuous semantic compression, the balance between reconstruction and generation, and continuous-discrete co-adaptation. Cola2 dynamically segments text into semantic spans, controls the compression rate through auxiliary-loss-free negative feedback, and combines reconstruction and generation supervision to learn compact, predictive latent representations. A block-causal masked diffusion decoder is trained with clean, noisy, and dropped latent conditions, while classifier-free guidance adjusts the balance between semantic reconstruction and autonomous generation. Token-prediction gradients update the encoder through the latents, whereas Flow Matching prior learning is decoupled from representation learning by stop-gradient. During inference, generated text is re-encoded under a fixed span layout to keep the latent history consistent with the actual text context. Experiments on five downstream tasks with about 210 billion training tokens validate the core designs and show competitive, unsaturated scaling. With 346.03 billion source tokens, an unconverged checkpoint reaches a 60.10 average across eight tasks and surpasses Llama-3-8B on HellaSwag and StoryCloze despite its reported pretraining budget being over 43 times larger. These results demonstrate the potential of continuous-latent hierarchical language generation.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Language model, Artificial intelligence, Generator (circuit theory), Natural language processing, Encoder

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange