Newest papers

VerusEval: Evaluating LLM-Generated Verus Specifications Beyond Verifier Acceptance

Ding Wang, Chao Jiang, Dugang Liu, Zhiwu Xu, Cheng Wen, Zhong Ming, Yuwei Liu, Xinyi Wan, Lin Huang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26464 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

Revisão de contratos de grande escala (LLMs) são cada vez mais usados para gerar contratos de benus para programas de Rust, juntamente com as provas que os estabelecem. No entanto, a aceitação do verificador, a medida comum de sucesso, mostra apenas que a implementação satisfaz o contrato gerado, por isso deixa aberto se o contrato captura o comportamento pretendido. A avaliação da geração de especificações exige, portanto, uma metodologia de avaliação, não uma única métrica. Nós apresentamos o VerusEval, um quadro que avalia contratos gerados ao longo de quatro dimensões, incluindo verificações nativas de Verus para trivialidade semântica, suites de entrada e saída validados (I/O) e equivalencia de contratos corretos, e que os relaciona de modo que a força e os pontos

Machine translation (Meta M2M100); the English original follows.

Abstract

Large language models (LLMs) are increasingly used to generate Verus contracts for Rust programs together with the proofs that establish them. Yet verifier acceptance, the common measure of success, shows only that the implementation satisfies the generated contract, so it leaves open whether the contract captures the intended behavior. Assessing specification generation therefore requires an evaluation methodology, not a single metric. We present VerusEval, a framework that evaluates generated contracts along six dimensions, including Verus-native checks for semantic triviality, validated input-output (I/O) suites, and contract equivalence, and that relates them so that the strength and blind spots of each become measurable. Evaluating 13,716 artifacts from four workflows, five LLMs, and two prompting settings on 762 tasks, we find that acceptance overstates quality: only 42.3\% of the accepted artifacts complete all three semantic checks without a flag, and across workflows and models, higher acceptance does not consistently come with more correct contracts. I/O evaluation rejects 23.6\% of the accepted artifacts, yet only 63.2\% of those that pass it are proved equivalent. Of the accepted artifacts flagged by contract comparison, 84.1\% have a rejected completeness obligation, and the direction of a failure prioritizes inspection of the generated contract or the reference; it does not by itself establish which artifact is defective. Screening and review further identify 36 candidate defects among 61 selected reference contracts, and repairing one of them makes 10 of its 11 failing artifacts equivalent. Based on these findings, we derive four actionable lessons for evaluation, generation, and benchmark maintenance.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Completeness (order theory), Artifact (error), Benchmark (surveying), Mathematical proof, Workflow

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange