Luong Tuan Thanh, Abhijit Sanyal · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.30924 · cited by 0 works (OpenAlex) · open-access PDF
Penilaian Interval Intellectual Intelligence (AI) yang lebih rendah memerlukan bukti tentang apakah suite pengujian mengekspos cacat tertentu dan apakah aturan referensi mereka dapat diandalkan. Kami memperkenalkan REGATTA (Regulatory Evaluation of Agents via Two-stage Testing and Assurance), sebuah kerangka kerja pengukuran dengan tiga instrumen. Daftar pemeriksaan yang diindikasikan oleh sumber 125-item mencakup lima sel industri-per-jurisdiksi di seluruh Amerika Serikat dan Vietnam. Sebuah korpus referensi 25-fault mencakup lima kategori, dan skor kode saat ini mencakup deteksi cacat suite-level pada tahap desain dan eksekusi. Kami memeriksa empat protokol pengukuran pada 180 suite yang dihasilkan (5,400 skenario) dari tiga generator LLM. Sebuah audit sumber menemukan 24 item yang menyesatkan aturan yang berlaku, 46 kesalahan eksek
Machine translation (Meta M2M100); the English original follows.
Enterprise AI-agent evaluation needs evidence about whether test suites expose specified faults and whether their reference rules are trustworthy. We introduce REGATTA (Regulatory Evaluation of Agents via Two-stage Testing and Assurance), a measurement framework with three instruments. A 125-item source-cited checklist spans five industry-by-jurisdiction cells across the United States and Vietnam. A 25-fault corpus covers five categories, and a harness scores suite-level fault detection at the design and execution stages. We examine four measurement diagnostics on 180 generated suites (5,400 scenarios) from three LLM generators. A source audit finds 24 items misstating the rule in force, 46 additional citation defects, 31 items with only non-binding, unidentifiable, or unverifiable sources, and 24 verified clean. Only initially substance-flagged items received author review; all 24 clean rows received AI review only. Results therefore describe the instruments as written, not legal compliance. Item coverage varies, execution-stage fault profiles are stable across generators, and the historical stages agree on $66$--$69\%$ of fault--suite pairs. A current-model US-insurance retest finds stage disagreement above repeated-execution variation, but stage agreement $.536$ falls below pooled chance $.549$, failing V3's lower inequality. Blind adjudication by one expert supports the design-stage judging protocol ($κ=.72$, 95% interval $[.52,.88]$); 40 current-model execution probes give $κ=.75$. The instruments, audit, and reusable code are released.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Computer science, Software engineering, Audit, Protocol (science), Checklist, Adjudication
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.30924, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange