Newest papers

From Candidate Admission to Task-Level Return: Auditing LLM-Generated Code under Execution Contracts

Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi, Naohiko Matsuda · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.24609 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

入学規則は生成されたプログラムを保持し、選択方針は任務ごとに1人の候補者を返します。我々は、これらの決定が2つの別々に凍結され、分離されたHumanEvalPlusコホートで実行契約とどのように相互作用するかを監査し、それぞれ32の任務、プロフィールごとに4つのサンプル、および2つのサービスプロフィールで4つのサンプルを含みます。それぞれのコホートの1つのインライン実行契約の下で、構造的および自己テストスクリーンは、80-107入学ごとに4〜9の拡張されたスイート非パスを許可し、128の計画試みごとに14〜43のスイートパスを排除します。オリジナルのプルプールでは

Machine translation (Meta M2M100); the English original follows.

Abstract

Admission rules retain generated programs, whereas selection policies return one candidate per task. We audit how these decisions interact with execution contracts in two separately frozen, disjoint HumanEvalPlus cohorts, each comprising 32 tasks, four samples per task per profile and two served profiles. Under the inline execution contract, a structural-and-self-test screen admits 4-9 expanded-suite nonpasses per 80-107 admissions and excludes 14-43 suite passes per 128 planned attempts. A restricted module-supply condition adds 20 and 14 passing admissions for one profile in the respective cohorts, plus one follow-up nonpass, and reduces the excluded suite passes to 14-23. All fixed first-admitted outcomes and abstentions remain unchanged. In the original pool, the added passes enter already all-passing admitted pools; in the follow-up, four of 24 common sample orders lose one passed task. Selected-subset diagnostics separately address registration-sensitive completion and reference-relative assertion disagreement, accounting for input domains and comparator tolerances. In these high-pass pools, candidate retention and task-level return diverge and are reported separately. Results are conditional on the saved pools, contracts and pinned suites; downstream training effects are unmeasured.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Audit, Suite, Task (project management), Computer science, Disjoint sets, Sample (material)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange