What Do RTL Repair Benchmarks Actually Measure?

Yeyin Eva Zhu, Rishabh Ranawat · 2026 · arXiv

DOI: 10.48550/arXiv.2610.28595 · open-access PDF

Korean

RTL 수리 벤치마크는 특정 사양에 따라 모델이 부러진 RTL를 수정할 수 있는지 측정해야 합니다. 이를 위해 우리는 5,784 (부러진, 고정된) 훈련 쌍인 RTLRepairDataset를 구축하고 해시 링크; RTLRepairBench, 374 구문과 85 행동 돌연변이를 포함한 459 건의 케이스; 시뮬레이션과 형식적 동등성을 지원하는 평가 유틸리티를 구축합니다. 우리는이 인프라를 사용하여 두 가지 측정 혼란을 식별합니다. 첫째, 시뮬레이션과 형식적 점수는 평가된 한 코호트에서 사양을 제공하는 분명한 효과에 대한 반대 신호를 제공합니다.

Machine translation (Meta M2M100); the English original follows.

Abstract

RTL repair benchmarks should measure whether a model can fix broken RTL given a specification. To this end, we construct and hash-bind RTLRepairDataset, 5,784 (broken, fixed) training pairs; RTLRepairBench, 459 held-out cases with 374 syntactic and 85 behavioral mutations; and an evaluation harness supporting simulation and formal equivalence. We then use this infrastructure to identify two measurement confounds. First, simulation and formal scoring give opposite signs for the apparent effect of providing a specification in one evaluated cohort; reset-aware rescoring attributes almost all of t

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Is this paper real? This page was built from OpenAlex and arXiv records, not generated. Software can verify any reference in one call: GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.28595, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txt

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange