Ruikai Shi, Zhengran Zeng, Rui Xie, Shikun Zhang, Wei Ye · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.25308 · cited by 0 works (OpenAlex) · open-access PDF
随着软件代理人越来越高效地生成和修改代码,评估他们的改变和发现缺陷的能力变得重要。这样的评估需要可执行的项目环境来调查和可靠的任务缺陷引用,由可重复行为支持。现有的代码审查基准为此组合提供有限的支持。我们提出了一个系统,它从历史的引入 - 修复委托对和通过验证版本的测试中构建可执行的复制和基于证据的回归实例。该系统从修复侧的变化中提取候选人测试,然后使用受限制的代理人将其移植并配置版本特定的环境,同时保留生产代码和测试预期。独立复制和基于证据的审查验证了在引入版本和通过验证版本上是否失败的测试。我们的接受的实例
Machine translation (Meta M2M100); the English original follows.
As software agents increasingly generate and modify code, evaluating their ability to review changes and discover defects becomes important. Such evaluation needs executable project environments for investigation and reliable defect references supported by reproducible behavior. Existing code review benchmarks provide limited support for this combination. We present a system that constructs executable regression instances from historical introducing--fixing commit pairs and developer tests. The system extracts candidate tests from fixing-side changes, then uses constrained agents to transplant them and configure version-specific environments while preserving production code and test expectations. Independent replay and evidence-based review verify whether tests that fail on the introducing version and pass on the fixing version support the proposed regression relationship. Accepted instances retain both versions' replay recipes, validated tests, and execution evidence. Repair-supported analysis and human review add root-cause locations, tested repairs, and causal explanations. These instances support bug detection and localization in code review, and enable bug repair and regression-test generation on the same defects. The system constructs 172 validated instances from 522 candidates across 11 Python projects. We evaluate three software agents with ten models and additional reasoning-effort settings on 83 manually selected, enriched instances across the three tasks. Precise root-line localization remains challenging, while baseline combinations achieve repair and regression-test generation success rates of up to 68.7% and 77.1%, respectively. Leading configurations differ by task, and higher task performance does not necessarily yield greater API-cost efficiency. Our approach provides reusable executable evidence for studying agents' defect discovery, repair, and testing capabilities.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Executable, Regression testing, Computer science, Python (programming language), Commit, Software engineering
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.25308, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange