Newest papers

Copying Under Repetition: From Prefix Matching to Position Hashing

Anxin Guo, Tianlong Huang, Chenxiao Yang, Zhiyuan Li · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26722 · cited by 0 works (OpenAlex) · open-access PDF

Korean

오랜 기간 동안 복제된 원본 LLM은 거의 실수없이 무작위 텍스트를 복사 할 수 있지만 종종 반복되는 문자의 긴 블록을 복제하거나 분쇄합니다. 반복되는 문자열에서 로컬 콘텐츠를 충족시키는 것은 더 이상 복제가 계속되어야하는지 또는 언제 중지되어야하는지 신뢰할 수 없습니다. 그럼에도 불구하고 우리는 $O(\log\log N)$-비트 정밀도와 $O(\log N\log N)$-비트를 초과하여 $O(\log N\log N)$-비트를 초과하는 두 층 변환기를 사용하여 관심 가치와 출력 (VO-rotations)의 RoPE 스타일 회전을 사용하여 원본 모듈 내에서 격차를 인코딩 할 수 있습니다. 중국 Rinder Theorem

Machine translation (Meta M2M100); the English original follows.

Abstract

Frontier LLMs can copy random text almost flawlessly, yet they often overrun or truncate long blocks of repeated characters. In repeated strings, matching local content no longer reliably identifies where copying should continue or when it should stop. Nevertheless, we show that a two-layer transformer with $O(\log\log N)$-bit precision and $O(\log N/\log\log N)$ width can copy every string up to a chosen design horizon $N$ and terminate correctly under arbitrary repetition. We use RoPE-style rotations of attention values and outputs (VO-rotations) with small prime periods to encode distances from delimiter anchors modulo those primes. By the Chinese Remainder Theorem, these periodic encodings jointly distinguish all required distances, allowing one retrieval rule to select either the next source token or the end marker. In contrast, we prove a width-independent impossibility result for copying under sequential finite-precision softmax accumulation. It applies to transformers whose only positional input is an additive attention bias that eventually becomes constant, including NoPE and a recursive implementation of ALiBi. Once an identical-token run exceeds the resulting saturation threshold, further repetitions leave the logits unchanged. If this threshold grows sublinearly with $N$, these models cannot both complete a long all-identical copy and stop correctly, a pattern resembling the termination errors of frontier LLMs. In synthetic copying experiments, baseline transformer models fail on long repeated strings even within the training range. Trained VO models, by contrast, generalize beyond the training lengths with near-perfect accuracy up to length $\approx5005$, the selected primes' period product.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Prefix, Security token, Copying, Computer science, Algorithm, Suffix

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange