Newest papers

Hierarchical Token Merging with Recursive Autoencoders for Unsupervised Grammar Induction

Murat Guney Kemal, Robert Lieck · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26695 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

複数のレベルで複雑なセクションを構成的に表すことを可能にする複数のデータモダリティの基礎原理です。 明示的な構造監督なしにデータから直接このような階層を学ぶことは、表示学習における中心的な課題です。 我々は、有意義な階層構造が圧縮単独から生じるかどうかを調査します。 自動編集者が複雑な構成を再構築することで圧縮された表記を学ぶ一方で、これらの目標は必ずしも有意義な構成構造を好むものではありません。 我々は、この課題を監督されていない文法誘導を通じて研究します。 基礎的な階層構造が明確に評価され得る場所です。 我々は、プログラミング学習のための可能なバイナリトレ

Machine translation (Meta M2M100); the English original follows.

Abstract

Hierarchical abstraction is a fundamental principle across many data modalities, enabling complex sequences to be represented compositionally at multiple levels. Learning such hierarchies directly from data, without explicit structural supervision, remains a central challenge in representation learning. We investigate whether meaningful hierarchical structure can emerge from compression alone. While autoencoders learn compressed representations through reconstruction, this objective does not necessarily favour meaningful compositional structure. We study this challenge through unsupervised grammar induction, where the underlying hierarchy can be evaluated explicitly. We introduce a hierarchical token-merging autoencoder that recursively combines adjacent tokens into increasingly abstract representations, compressing entire sequences through a variational information bottleneck. The model jointly learns representations and hierarchical structure by marginalising over possible binary trees using dynamic programming. To address training instability, we combine independently trained models through unsupervised consensus and distillation. Experiments on a synthetic, ambiguous grammar demonstrate the emergence of meaningful hierarchical structure without structural supervision. The resulting model achieves 93% grammatical parse-validity and 93% exact reconstruction on held-out sequences, compared with 42% parse-validity for the strongest trivial baseline. These findings establish a proof of concept for hierarchical token merging as a mechanism for unsupervised structure discovery and abstraction learning.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Hierarchy, Autoencoder, Artificial intelligence, Abstraction, Representation (politics)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange