Newest papers

What Do RTL Repair Benchmarks Actually Measure?

Yeyin Eva Zhu, Rishabh Ranawat · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28595 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

修复的RTL行为基准应该衡量一个模型是否可以修复破碎的RTL根据规格。为此目的,我们构建和哈希连接RTLRepairDataset,5784(破碎,固定的)训练对;RTLRepairBench,459在一个被评估的群组中提供规格的显而易见效果的相反迹象;重新设置的行为突变;以及支持模拟和正式等同性的评估工具。然后我们使用这个基础设施来识别两个测量混淆。首先,模拟和正式总数给出在一个评估的群组中提供规格的显而易见效应的相反迹象;半重复评估的属性几乎覆盖了所有模拟形式的分歧,以重新设置和第一个周期进行对比。第二,在

Machine translation (Meta M2M100); the English original follows.

Abstract

RTL repair benchmarks should measure whether a model can fix broken RTL given a specification. To this end, we construct and hash-bind RTLRepairDataset, 5,784 (broken, fixed) training pairs; RTLRepairBench, 459 held-out cases with 374 syntactic and 85 behavioral mutations; and an evaluation harness supporting simulation and formal equivalence. We then use this infrastructure to identify two measurement confounds. First, simulation and formal scoring give opposite signs for the apparent effect of providing a specification in one evaluated cohort; reset-aware rescoring attributes almost all of the simulation-formal disagreement to reset and first-cycle comparison semantics. Second, on 274 mined model failures, the specification-only minus specification-conditioned recovery point estimate ranges from -8.5 to +18.3 percentage points across six models. Thus, a conventional repair score need not isolate whether the model used the faulty RTL. We translate these findings into evaluation controls: report initialization, reset, first-cycle, and unknown-value semantics; disclose oracle coverage and undecided cases; and include a specification-only baseline. Together, the resources and audit provide both a benchmark for behavioral repair and a protocol for diagnosing what contributes to its measured score. We release the benchmark, the training set, construction code, sanitized aggregate evidence, and reproduction materials at https://github.com/Evaaaz/RTLRepairBench.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Benchmark (surveying), Reset (finance), Oracle, Construct (python library), Measure (data warehouse)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange