Newest papers

From Candidate Admission to Task-Level Return: Auditing LLM-Generated Code under Execution Contracts

Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi, Naohiko Matsuda · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.24609 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Aturan penerimaan mempertahankan program yang dihasilkan, sementara kebijakan pemilihan mengembalikan satu kandidat per tugas. Kami mengaudit bagaimana keputusan ini berinteraksi dengan kontrak eksekusi dalam dua kohor HumanEvalPlus yang dibekukan secara terpisah, masing-masing terdiri dari 32 tugas, empat sampel per tugas per profil dan dua profil yang diserahkan. Di bawah kontrak eksekusi inline, layar struktural dan self-testing mengakui 4-9 pas-suite yang diperluas per 80-107 penerimaan dan mengecualikan 14-43 pas-suite yang ditandatangani pertama kali dan abstinensi tugas tetap tidak berubah. Dalam kolam asli, kondisi pas-modul terbatas menambahkan 20 dan 14 pas-admission untuk satu profil dalam kohor masing-masing, ditambah satu non-pas follow-up, dan mengurangi pas-suite yang dikecualikan menjadi 14

Machine translation (Meta M2M100); the English original follows.

Abstract

Admission rules retain generated programs, whereas selection policies return one candidate per task. We audit how these decisions interact with execution contracts in two separately frozen, disjoint HumanEvalPlus cohorts, each comprising 32 tasks, four samples per task per profile and two served profiles. Under the inline execution contract, a structural-and-self-test screen admits 4-9 expanded-suite nonpasses per 80-107 admissions and excludes 14-43 suite passes per 128 planned attempts. A restricted module-supply condition adds 20 and 14 passing admissions for one profile in the respective cohorts, plus one follow-up nonpass, and reduces the excluded suite passes to 14-23. All fixed first-admitted outcomes and abstentions remain unchanged. In the original pool, the added passes enter already all-passing admitted pools; in the follow-up, four of 24 common sample orders lose one passed task. Selected-subset diagnostics separately address registration-sensitive completion and reference-relative assertion disagreement, accounting for input domains and comparator tolerances. In these high-pass pools, candidate retention and task-level return diverge and are reported separately. Results are conditional on the saved pools, contracts and pinned suites; downstream training effects are unmeasured.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Audit, Suite, Task (project management), Computer science, Disjoint sets, Sample (material)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange