Ruikai Shi, Zhengran Zeng, Rui Xie, Shikun Zhang, Wei Ye · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.25308 · cited by 0 works (OpenAlex) · open-access PDF
소프트웨어 에이전트가 점점 더 효율적으로 코드를 생성하고 수정함에 따라 변경 사항을 검토하고 결함을 발견하는 능력을 평가하는 것이 중요합니다. 그러한 평가는 수행 가능한 프로젝트 환경을 조사하고 재현 가능한 행동에 의해 지원되는 신뢰할 수있는 작업 결함 참조를 위해 실행할 수 있습니다. 기존의 코드 검토 벤치마크는이 조합에 대한 제한된 지원을 제공합니다. 우리는 역사적인 소개-fixing 커미팅 쌍과 개발자 테스트에서 실행 가능한 재생 및 증거 기반 검토 인스턴스를 구축하는 시스템을 제시합니다. 시스템은 제안된 회귀 관계를 지원하는지 여부를 검토합니다. 수행 가능한 인스턴스는 재생 조리법, 인스턴스 시험 및 실행 증거를 개조하고 생산 코드
Machine translation (Meta M2M100); the English original follows.
As software agents increasingly generate and modify code, evaluating their ability to review changes and discover defects becomes important. Such evaluation needs executable project environments for investigation and reliable defect references supported by reproducible behavior. Existing code review benchmarks provide limited support for this combination. We present a system that constructs executable regression instances from historical introducing--fixing commit pairs and developer tests. The system extracts candidate tests from fixing-side changes, then uses constrained agents to transplant them and configure version-specific environments while preserving production code and test expectations. Independent replay and evidence-based review verify whether tests that fail on the introducing version and pass on the fixing version support the proposed regression relationship. Accepted instances retain both versions' replay recipes, validated tests, and execution evidence. Repair-supported analysis and human review add root-cause locations, tested repairs, and causal explanations. These instances support bug detection and localization in code review, and enable bug repair and regression-test generation on the same defects. The system constructs 172 validated instances from 522 candidates across 11 Python projects. We evaluate three software agents with ten models and additional reasoning-effort settings on 83 manually selected, enriched instances across the three tasks. Precise root-line localization remains challenging, while baseline combinations achieve repair and regression-test generation success rates of up to 68.7% and 77.1%, respectively. Leading configurations differ by task, and higher task performance does not necessarily yield greater API-cost efficiency. Our approach provides reusable executable evidence for studying agents' defect discovery, repair, and testing capabilities.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Executable, Regression testing, Computer science, Python (programming language), Commit, Software engineering
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.25308, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange