Ruikai Shi, Zhengran Zeng, Rui Xie, Shikun Zhang, Wei Ye · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.25308 · cited by 0 works (OpenAlex) · open-access PDF
À medida que os agentes de tarefas de software continuam a gerar e modificar código cada vez mais eficientemente, a avaliação de sua capacidade de revisar alterações e descobrir defeitos torna-se importante. Tal avaliação precisa de ambientes de projeto executáveis para investigação e referências de defeitos confiáveis suportadas por comportamento reprodutível. Os benchmarks de revisão de código existentes fornecem suporte limitado para esta combinação. Nós apresentamos um sistema que constrói instâncias de regressão executáveis de pares de compromissos históricos de introdução - fixação e testes de desenvolvedores. O sistema extrai testes de candidato de mudanças de desempenho de fixação e, em seguida, usa agentes restritos para os transplantar e configurar ambientes específicos de versão, enquanto preserva o código de produção e as expectativas de teste
Machine translation (Meta M2M100); the English original follows.
As software agents increasingly generate and modify code, evaluating their ability to review changes and discover defects becomes important. Such evaluation needs executable project environments for investigation and reliable defect references supported by reproducible behavior. Existing code review benchmarks provide limited support for this combination. We present a system that constructs executable regression instances from historical introducing--fixing commit pairs and developer tests. The system extracts candidate tests from fixing-side changes, then uses constrained agents to transplant them and configure version-specific environments while preserving production code and test expectations. Independent replay and evidence-based review verify whether tests that fail on the introducing version and pass on the fixing version support the proposed regression relationship. Accepted instances retain both versions' replay recipes, validated tests, and execution evidence. Repair-supported analysis and human review add root-cause locations, tested repairs, and causal explanations. These instances support bug detection and localization in code review, and enable bug repair and regression-test generation on the same defects. The system constructs 172 validated instances from 522 candidates across 11 Python projects. We evaluate three software agents with ten models and additional reasoning-effort settings on 83 manually selected, enriched instances across the three tasks. Precise root-line localization remains challenging, while baseline combinations achieve repair and regression-test generation success rates of up to 68.7% and 77.1%, respectively. Leading configurations differ by task, and higher task performance does not necessarily yield greater API-cost efficiency. Our approach provides reusable executable evidence for studying agents' defect discovery, repair, and testing capabilities.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Executable, Regression testing, Computer science, Python (programming language), Commit, Software engineering
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.25308, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange