Ruikai Shi, Zhengran Zeng, Rui Xie, Shikun Zhang, Wei Ye · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.25308 · cited by 0 works (OpenAlex) · open-access PDF
Da Software-Task-Agenten immer effizienter Code generieren und modifizieren, wird die Bewertung ihrer Fähigkeit, Änderungen zu überprüfen und Defekte zu entdecken, wichtig. Eine solche Bewertung benötigt ausführbare Projektumgebungen für die Untersuchung und zuverlässige Aufgabenfehlerreferenzen, die durch reproduzierbares Verhalten unterstützt werden. Bestehende Code-Review-Benchmarks bieten eine begrenzte Unterstützung für diese Kombination. Wir präsentieren ein System, das ausführbare Replay- und evidenzbasierte Revisionseinheiten aus historischen Einführungs- Fix-Commit-Paaren und Entwickler-Tests konstruiert. Das System extrahiert Kandidaten-Tests aus Fix-Seiten-Veränderungen und verwendet dann eingeschränkte Agenten, um sie zu transplantieren und Version
Machine translation (Meta M2M100); the English original follows.
As software agents increasingly generate and modify code, evaluating their ability to review changes and discover defects becomes important. Such evaluation needs executable project environments for investigation and reliable defect references supported by reproducible behavior. Existing code review benchmarks provide limited support for this combination. We present a system that constructs executable regression instances from historical introducing--fixing commit pairs and developer tests. The system extracts candidate tests from fixing-side changes, then uses constrained agents to transplant them and configure version-specific environments while preserving production code and test expectations. Independent replay and evidence-based review verify whether tests that fail on the introducing version and pass on the fixing version support the proposed regression relationship. Accepted instances retain both versions' replay recipes, validated tests, and execution evidence. Repair-supported analysis and human review add root-cause locations, tested repairs, and causal explanations. These instances support bug detection and localization in code review, and enable bug repair and regression-test generation on the same defects. The system constructs 172 validated instances from 522 candidates across 11 Python projects. We evaluate three software agents with ten models and additional reasoning-effort settings on 83 manually selected, enriched instances across the three tasks. Precise root-line localization remains challenging, while baseline combinations achieve repair and regression-test generation success rates of up to 68.7% and 77.1%, respectively. Leading configurations differ by task, and higher task performance does not necessarily yield greater API-cost efficiency. Our approach provides reusable executable evidence for studying agents' defect discovery, repair, and testing capabilities.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Executable, Regression testing, Computer science, Python (programming language), Commit, Software engineering
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.25308, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange