Newest papers

Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

Jens Lehmann, Sahar Vahdati · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30499 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

通常,即使是强大的间隔检测系统也有效地获得了新技能,而不需要人类为每个新任务重新设计其软件架构。然而,语言模型很少单独行动。 轴承是任务间隔的象征性程序,该程序对模型进行调试,处理感知,记忆,工具使用,恢复和推断分配,因此我们知道有效的设计是否存在。 ARC-AGI-3 是一个交互式基准,在其中代理人必须探索陌生的环境,在飞机上找到他们的目标,并在他们之间继续学习。 在其上,人为控制的系统已经报告了公共游戏的天花板分数,所以我们知道有效的设计是否存在。 我们询问边缘模型是否可以建立和改进这样的轴承。 通过固定编码的起始轴承模型,每个模型都构

Machine translation (Meta M2M100); the English original follows.

Abstract

A generally intelligent system acquires new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet language models rarely act alone. A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf. ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. On it, human-steered systems already report ceiling scores across the public games, so we know effective designs exist. We ask whether frontier models can build and improve such a harness on their own. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE), which combines level completion with action efficiency relative to human baselines, and record dollar-denominated inference cost as a practical resource measure. Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]). Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance. Human-steered systems demonstrate further headroom. These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Task (project management), Inference, Coding (social sciences), Risk analysis (engineering), Adaptation (eye)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange