Newest papers

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

Soham Dan, Jeff Da, Sanyam Bhutani, Miguel Romero Calvo, Iván Guevara, Fernando Crabedo, José Bernal, Julio Reyna, Luis Leandro Monsalve Mendez, Ying Liu, Daniel Yue Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.24206 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

SWE-Bench Pro es un benchmark ampliamente utilizado de tareas de ingeniería de software de largo horizonte para agentes de codificación. A medida que los agentes mejoran, alcanzan problemas de tareas antes agentes nunca tocados e interactúan con el entorno de evaluación de maneras que el arnés original no anticipaba. Presentamos SWE-Bench Pro V2, una división pública revisada construida para mantener el benchmark exacto a medida que avanzan las capacidades. Las auditorías de agentes, la adjudicación de expertos y la reimplantación ciega reparan o retiran 480 de las 731 tareas: 23 ingenieros contratados pasan 404 horas, cada uno trabajando en un idioma que escriben profesionalmente, y cada instrucción revisada fue verificada por un segundo ingeniero resolviendo la tarea desde el texto

Machine translation (Meta M2M100); the English original follows.

Abstract

SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance. Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch. On the 642 V2 tasks frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%. We release the benchmark and distill lessons for maintaining agent benchmarks over time.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Computer science, Coding (social sciences), Task (project management), Software, Software engineering

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange