Newest papers

Validating Autoformalized Specifications via Contrapuntal Languages

Yihan Dai, Xuanlin Liu, Haoxiang Jia, Tianchi Li, Xin Zhang, Sergey Mechtaev · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30059 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

由于用户的意图通常是用自然语言而不是形式逻辑表达的,所以规格必须首先从中得到正确的,例如,通过LLM基于的完全程序自动化2%,但自动化规格本身是不可信任的,所以一个关键的挑战是验证他们是否捕捉用户的意图而不依赖于昂贵的方言,如人类的审查。在这种无对点的语言设置中,一种自然的策略是通过模型输出进行投票,但由于模型特定的错误往往是相关的,输出之间的协议可能反映了共享错误而不是正确性。我们的关键见解是,这些错误可以通过代表性多样性进行排除。我们享受这种多样性:对任务特定的公式语言的对比,但由于模型特定的错误往往是相关的,所以输出之间的协议可能反映

Machine translation (Meta M2M100); the English original follows.

Abstract

Formal specifications can make AI-generated code mechanically checkable. Because user intent is typically expressed in natural language rather than formal logic, specifications must first be derived from it, e.g., via LLM-based autoformalization. Yet autoformalized specifications are themselves untrusted, so a key challenge is validating whether they capture user intent without relying on a costly oracle such as human review. In this oracle-free setting, a natural strategy is to vote among model outputs, but because model errors are often correlated, agreement among outputs may reflect shared mistakes rather than correctness. Our key insight is that these errors can be decorrelated through representational diversity. We achieve this diversity with contrapuntal languages: pairs of task-specific formal languages, each designed to express the intended specification, yet admitting no structure-preserving translation between them. An LLM formalizes the intent independently in each language, and we accept a specification only if a solver proves the two formalizations equivalent. To construct such languages, we present a synthesis algorithm that reuses fragments of sampled seed specifications as syntactic sugar over a base specification language. We implement the approach for first-order logic specifications and evaluate it on HumanEval+ and LiveCodeBench using GPT-5.4 mini and Qwen3.6-Flash with and without thinking. Over the strongest baseline, our method increases the proportion of accepted specifications that are correct, i.e., complete and sound, by 12.0% on HumanEval+ and 42.9% on LiveCodeBench. The accepted specifications judged complete reject 98.6% of buggy program variants on HumanEval+ and 96.2% on LiveCodeBench, compared with 92.8% and 78.3% for the strongest baseline.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Programming language, Oracle, Formal specification, Key (lock), Construct (python library)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange