Newest papers

Hierarchical Token Merging with Recursive Autoencoders for Unsupervised Grammar Induction

Murat Guney Kemal, Robert Lieck · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26695 · cited by 0 works (OpenAlex) · open-access PDF

Korean

Gruperarchic 토크 아브스트레이션은 많은 데이터 모드에 걸쳐 복잡한 시퀀스를 구성적으로 여러 레벨에서 표현할 수 있도록하는 기본 원칙입니다. 그러한 계층을 데이터에서 직접 학습하는 것은 명시적 인 구조적 감독이없는 데 중앙적인 도전으로 남아 있습니다. 우리는 의미있는 계층 구조가 압축에서만 나타날 수 있는지 여부를 조사합니다. 자동 코더가 복구를 통해 압축 된 계층을 배우는 반면,이 목표는 반드시 의미있는 구성 구조를 촉진하지 않습니다. 우리는 통제되지 않은 문법적 인덱싱을 통해이 도전을 연구합니다. 기본 계층이 명시적으로 평가 될 수 있는지 여부를 조사합니다. 우리는 문법적 인덱싱 자동 코더를 소개합니다. 학습을 사용하여 역동적 인덱싱을위한

Machine translation (Meta M2M100); the English original follows.

Abstract

Hierarchical abstraction is a fundamental principle across many data modalities, enabling complex sequences to be represented compositionally at multiple levels. Learning such hierarchies directly from data, without explicit structural supervision, remains a central challenge in representation learning. We investigate whether meaningful hierarchical structure can emerge from compression alone. While autoencoders learn compressed representations through reconstruction, this objective does not necessarily favour meaningful compositional structure. We study this challenge through unsupervised grammar induction, where the underlying hierarchy can be evaluated explicitly. We introduce a hierarchical token-merging autoencoder that recursively combines adjacent tokens into increasingly abstract representations, compressing entire sequences through a variational information bottleneck. The model jointly learns representations and hierarchical structure by marginalising over possible binary trees using dynamic programming. To address training instability, we combine independently trained models through unsupervised consensus and distillation. Experiments on a synthetic, ambiguous grammar demonstrate the emergence of meaningful hierarchical structure without structural supervision. The resulting model achieves 93% grammatical parse-validity and 93% exact reconstruction on held-out sequences, compared with 42% parse-validity for the strongest trivial baseline. These findings establish a proof of concept for hierarchical token merging as a mechanism for unsupervised structure discovery and abstraction learning.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Hierarchy, Autoencoder, Artificial intelligence, Abstraction, Representation (politics)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange