Newest papers

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

Soham Dan, Jeff Da, Sanyam Bhutani, Miguel Romero Calvo, Iván Guevara, Fernando Crabedo, José Bernal, Julio Reyna, Luis Leandro Monsalve Mendez, Ying Liu, Daniel Yue Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.24206 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

SWE-Bench Pro 是编码代理商的长视野软件工程任务的广泛使用基准。随着代理商的改进,他们达到以前代理商从未触及的任务问题,并以原始工具所未预期的方式与评估环境进行互动。我们介绍了 SWE-Bench Pro V2,一个修订的公共分区,以保持基准准准确,因为能力提前。代理商审核,专家评审和盲目重新实施修复或退休了 731 项任务中的 480 项: 23 名合同工程师花了 404 小时,每个工作在他们专业写的语言中,每一个修订的指示都是由第二位工程师验证,从单独的文本解决任务。 一个硬化协议限制了网络访问终点模型,并在清洁容器中重新分级了每个

Machine translation (Meta M2M100); the English original follows.

Abstract

SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance. Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch. On the 642 V2 tasks frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%. We release the benchmark and distill lessons for maintaining agent benchmarks over time.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Computer science, Coding (social sciences), Task (project management), Software, Software engineering

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange