Newest papers

VeeReview: Measuring Build-Verify Pairs in Agent-Authored Pull Requests from a V-Model Perspective

Gabriel Dautovi, Jie Wu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29143 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

Contexto verificado: Revisão de um mecanismo de relatório válido é o que requer a verificação de uma solicitação de intervalo do laboratório. Os benchmarks do agente de codificação e os estudos dos resultados da solicitação de pull não avaliam diretamente essa relação. Objetivo: Testamos se os pares de compilação e verificação do V-Modelo podem medir a evidência de verificação em solicitações de pull e as comparações de suporte entre as alterações propostas. Método: VeeReview classifica quatro pares de compilação e verificação com base em sinais de repositório e um juiz de modelo de linguagem. Três pares usam julgamentos de modelo restritos por critérios deterministas, e uma regra sobre as solicitações de pull e a árvore de repositório decide sobre as taxas de

Machine translation (Meta M2M100); the English original follows.

Abstract

Context: Reviewing a pull request requires verification evidence that addresses the proposed change. Coding agent benchmarks and studies of pull request outcomes do not directly assess this relationship. Objective: We test whether the V-Model's build and verify pairs can measure verification evidence in pull requests and support comparisons between agent-authored and human-authored changes. Method: VeeReview classifies four build and verify pairs using repository signals and a language-model judge. Three pairs use model judgements constrained by deterministic signals, and a rule over the diff and repository tree decides Module Design-Unit Test. We evaluated VeeReview against a 25-pull request reference set labelled by two raters and applied it to 1,454 matched pairs of agent and human pull requests from 741 repositories. Results: VeeReview assigned lower build-only rates to agent-authored pull requests on all four pairs, contrary to the preregistered hypothesis (mixed-effects odds ratios 0.52 to 0.86). Architecture-Integration was not significant. Standardization for linked issues, shipped tests and size reduced matched-rate differences by 34% to 60%, and the rule-decided pair's standardized interval included zero. The instrument did not meet its preregistered calibration criterion: agreement with adjudicated labels was 4% to 60% against a required 80%, and inter-rater agreement was weak (kappa = 0.154). Conclusions: The rule-decided pair shows that a process measure judged by a language model partly reflects how a pull request is presented rather than the verification work behind it. We report the cohort differences as observations made with this instrument, and we argue that such measures need a mechanically decided component and a validated human reference before they can support comparisons of verification practice.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Coding (social sciences), Standardization, Perspective (graphical), Set (abstract data type), Measure (data warehouse)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange