Tianqi Liu, Yandong Guan, Songze Li, Wenkai Wang, Gang Yu, Chengfei Lv · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.26394 · cited by 0 works (OpenAlex) · open-access PDF
代码代理人越来越多地在成熟企业存储库中承担开发任务,但现有基准提供了有限的洞察力,以确定他们是否完成完整的产品要求文件(PRDs)和在哪里交付打破。从零开始,发行解决方案和开发让增量PRD实现未被探索,而孤立测试提供了业务和整合要求的不完整证据。我们引入了PACE,其中包括43个企业任务的基准,有750个专家审核的代码点。它结合了要求重建,标准构建和基于证据的评估,以测量PRD满足并将交付差距连接到代理人行为。在11个模型 -scaffold配置中,大多数的分数在52.1和78.1之间;GPT-5.6-Solive实现了14.0%的
Machine translation (Meta M2M100); the English original follows.
Coding agents increasingly undertake development tasks in mature enterprise repositories, yet existing benchmarks offer limited insight into whether they fulfill complete Product Requirement Documents (PRDs) and where delivery breaks down. Issue resolution and development from scratch leave incremental PRD fulfillment underexplored, while isolated tests provide incomplete evidence of business and integration requirements. We introduce PACE, a benchmark of 43 enterprise tasks with 750 expert-audited Code Points. It combines requirement reconstruction, criterion construction, and evidence-based evaluation to measure PRD fulfillment and connect delivery gaps to agent behavior. Across 11 model--scaffold configurations, most scores fall between 52.1 and 78.1; GPT-5.6-Sol achieves the highest complete-delivery rate of 14.0\%. Incomplete implementation and missing integration are the principal bottlenecks: producing individual components does not ensure a complete delivery. Evaluation achieves 95.2\% agreement with expert consensus, with lower case-level variation across judging records than repeated executions. Beyond model comparison, the benchmark provides actionable feedback for harness refinement: diagnosis guides targeted improvements, and successive harness releases raise mean performance from 59.4 to 63.0 using the same model and task suite. Code is available at https://github.com/TianqiLiu1/PACE-Bench.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Computer science, Coding (social sciences), Benchmark (surveying), Software engineering, Task (project management), Principal (computer security)
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.26394, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange