Newest papers

Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

Jens Lehmann, Sahar Vahdati · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30499 · cited by 0 works (OpenAlex) · open-access PDF

Korean

일반적으로 강력한 인텔리전스 시스템은 인간이 모든 새로운 작업을 위해 소프트웨어 스카우트를 재 설계 할 필요없이 효율적으로 새로운 기술을 습득합니다. 그러나 언어 모델은 거의 독립적으로 작동하지 않습니다. 스카우트는 모델에 호출하고 인식, 메모리, 도구 사용, 복구 및 추론 할당을 효율적으로 수행하는 상징적 인 작업 프로그램입니다. ARC-AGI-3는 에이전트가 익숙하지 않은 환경을 탐구하고 비행기에서 목표를 파악하고 그들이 가는 동안 학습을 계속해야하는 상호 작용 기준입니다. 그 위에서, 인간이 지시하는 시스템은 이미 공공 게임에 걸쳐 천장 점수를보고, 복구하고 추론 할 수있는 효과적인 디자인이 존재한다는 것을 알고 있습니다. 우리는 경계 모델이 그

Machine translation (Meta M2M100); the English original follows.

Abstract

A generally intelligent system acquires new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet language models rarely act alone. A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf. ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. On it, human-steered systems already report ceiling scores across the public games, so we know effective designs exist. We ask whether frontier models can build and improve such a harness on their own. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE), which combines level completion with action efficiency relative to human baselines, and record dollar-denominated inference cost as a practical resource measure. Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]). Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance. Human-steered systems demonstrate further headroom. These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Task (project management), Inference, Coding (social sciences), Risk analysis (engineering), Adaptation (eye)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange