Tianqi Liu, Yandong Guan, Songze Li, Wenkai Wang, Gang Yu, Chengfei Lv · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.26394 · cited by 0 works (OpenAlex) · open-access PDF
Os agentes de codificação cada vez mais assumem tarefas de desenvolvimento em repositórios de empresas maduras, mas os benchmarks existentes oferecem insights limitados sobre se eles cumprem documentos de requisitos de produto completos (PRDs) e onde a entrega cai. Resolução de problemas e desenvolvimento a partir do zero deixam o cumprimento de PRD incremental não explorado, enquanto os testes isolados fornecem evidências incompletas de requisitos de negócios e integração. Introduzimos o PACE, um benchmark de 43 tarefas de negócios com 750 pontos de código auditados por especialistas. Ele combina reconstrução de requisitos, construção de critérios e avaliação baseada em evidências para medir o cumprimento do PRD e conectar lacunas de entrega ao comportamento do agente. Ao longo de 11 configurações de modelo-scaffold, a maioria das
Machine translation (Meta M2M100); the English original follows.
Coding agents increasingly undertake development tasks in mature enterprise repositories, yet existing benchmarks offer limited insight into whether they fulfill complete Product Requirement Documents (PRDs) and where delivery breaks down. Issue resolution and development from scratch leave incremental PRD fulfillment underexplored, while isolated tests provide incomplete evidence of business and integration requirements. We introduce PACE, a benchmark of 43 enterprise tasks with 750 expert-audited Code Points. It combines requirement reconstruction, criterion construction, and evidence-based evaluation to measure PRD fulfillment and connect delivery gaps to agent behavior. Across 11 model--scaffold configurations, most scores fall between 52.1 and 78.1; GPT-5.6-Sol achieves the highest complete-delivery rate of 14.0\%. Incomplete implementation and missing integration are the principal bottlenecks: producing individual components does not ensure a complete delivery. Evaluation achieves 95.2\% agreement with expert consensus, with lower case-level variation across judging records than repeated executions. Beyond model comparison, the benchmark provides actionable feedback for harness refinement: diagnosis guides targeted improvements, and successive harness releases raise mean performance from 59.4 to 63.0 using the same model and task suite. Code is available at https://github.com/TianqiLiu1/PACE-Bench.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Computer science, Coding (social sciences), Benchmark (surveying), Software engineering, Task (project management), Principal (computer security)
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.26394, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange