Newest papers

VeeReview: Measuring Build-Verify Pairs in Agent-Authored Pull Requests from a V-Model Perspective

Gabriel Dautovi, Jie Wu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29143 · cited by 0 works (OpenAlex) · open-access PDF

Korean

본 보고서의 유효한 측면: 이 유효한 간격을 반영하는 것은 제안된 변화를 해결하는 검증 증거가 필요합니다. 코딩 에이전트 벤치마크 및 끌 요청 결과의 연구는 직접적으로이 관계를 평가하지 않습니다. 목적: 우리는 V-모델의 구축 및 검증 쌍이 끌 요청에서 검증 증거를 측정할 수 있는지 여부를 테스트하고, 디프와 레포터 나무에 대한 규칙은 에이전트 작성 및 인간 작성 변경 사이의 비교를 결정합니다. 메서드: VeeReview는 2 명의 평가자에 의해 표시된 4 개의 구축 및 검증 옵션 쌍을 분류하고 1,454 명의 언어 모델 판사를 적용했습니다. 3 명의 쌍은 741 개의 레포터에서 유효한 모델

Machine translation (Meta M2M100); the English original follows.

Abstract

Context: Reviewing a pull request requires verification evidence that addresses the proposed change. Coding agent benchmarks and studies of pull request outcomes do not directly assess this relationship. Objective: We test whether the V-Model's build and verify pairs can measure verification evidence in pull requests and support comparisons between agent-authored and human-authored changes. Method: VeeReview classifies four build and verify pairs using repository signals and a language-model judge. Three pairs use model judgements constrained by deterministic signals, and a rule over the diff and repository tree decides Module Design-Unit Test. We evaluated VeeReview against a 25-pull request reference set labelled by two raters and applied it to 1,454 matched pairs of agent and human pull requests from 741 repositories. Results: VeeReview assigned lower build-only rates to agent-authored pull requests on all four pairs, contrary to the preregistered hypothesis (mixed-effects odds ratios 0.52 to 0.86). Architecture-Integration was not significant. Standardization for linked issues, shipped tests and size reduced matched-rate differences by 34% to 60%, and the rule-decided pair's standardized interval included zero. The instrument did not meet its preregistered calibration criterion: agreement with adjudicated labels was 4% to 60% against a required 80%, and inter-rater agreement was weak (kappa = 0.154). Conclusions: The rule-decided pair shows that a process measure judged by a language model partly reflects how a pull request is presented rather than the verification work behind it. We report the cohort differences as observations made with this instrument, and we argue that such measures need a mechanically decided component and a validated human reference before they can support comparisons of verification practice.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Coding (social sciences), Standardization, Perspective (graphical), Set (abstract data type), Measure (data warehouse)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange