Tianqi Liu, Yandong Guan, Songze Li, Wenkai Wang, Gang Yu, Chengfei Lv · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.26394 · cited by 0 works (OpenAlex) · open-access PDF
Los agentes de codificación se encargan cada vez más de tareas de desarrollo en repositorios empresariales maduros, pero los benchmarks existentes ofrecen una visión limitada de si cumplen con los Documentos de Requisitos del Producto (PRD) completos y donde se rompe la entrega. La resolución de problemas y el desarrollo desde cero dejan no explotadas las tareas incrementales de cumplimiento del PRD, mientras que las pruebas aisladas proporcionan evidencia incompleta de los requisitos de negocio e integración. Introducimos PACE, un benchmark de 43 tareas empresariales con 750 puntos de código auditados por expertos. Combina la reconstrucción de requisitos, la construcción de criterios y la evaluación basada en evidencias para medir el cumplimiento del PRD y conectar las brechas de entrega al comportamiento del agente. A través de
Machine translation (Meta M2M100); the English original follows.
Coding agents increasingly undertake development tasks in mature enterprise repositories, yet existing benchmarks offer limited insight into whether they fulfill complete Product Requirement Documents (PRDs) and where delivery breaks down. Issue resolution and development from scratch leave incremental PRD fulfillment underexplored, while isolated tests provide incomplete evidence of business and integration requirements. We introduce PACE, a benchmark of 43 enterprise tasks with 750 expert-audited Code Points. It combines requirement reconstruction, criterion construction, and evidence-based evaluation to measure PRD fulfillment and connect delivery gaps to agent behavior. Across 11 model--scaffold configurations, most scores fall between 52.1 and 78.1; GPT-5.6-Sol achieves the highest complete-delivery rate of 14.0\%. Incomplete implementation and missing integration are the principal bottlenecks: producing individual components does not ensure a complete delivery. Evaluation achieves 95.2\% agreement with expert consensus, with lower case-level variation across judging records than repeated executions. Beyond model comparison, the benchmark provides actionable feedback for harness refinement: diagnosis guides targeted improvements, and successive harness releases raise mean performance from 59.4 to 63.0 using the same model and task suite. Code is available at https://github.com/TianqiLiu1/PACE-Bench.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Computer science, Coding (social sciences), Benchmark (surveying), Software engineering, Task (project management), Principal (computer security)
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.26394, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange