Newest papers

SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges

Ethan Ludwick, Yu-Chung Hsiao · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28886 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Los jueces de clasificación SANBen son cada vez más mediadores de evaluación de benchmark, selección de modelos, curado de datos y capacitación. Sus decisiones son útiles solo cuando cada juez responde de manera fiable a los cambios que afectan a su criterio de puntuación. Los métodos existentes de prueba de fiabilidad perturban las entradas con efectos esperados conocidos sobre las puntuaciones de jueces. Introducimos SANE, un marco de prueba basado en agentes que combina las recetas de perturbación de los autores humanos con la generación de agentes sintéticos, dejando una receta de puntuación específica para los jugadores fuera de la línea de rendimiento automatizada. Esto limita su uso a las perturbaciones fijas y los tipos de puntuación previamente asumidos. Introducimos SANE, un marco de prueba basado en ag

Machine translation (Meta M2M100); the English original follows.

Abstract

LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion. Existing methods for reliability testing perturb inputs with known expected effects on judge scores. Prominent LLM-judge testing harnesses combine human-authored perturbation recipes with synthetic test generation, leaving scorer-specific recipe ideation outside the automated pipeline. This limits their use to fixed perturbations and pre-assumed scorer types. We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators as reusable agent instructions. Given a scorer specification, implementation, and representative inputs, the pipeline first identifies scorer-specific capabilities, then automatically derives perturbation recipes and expected score relations for each. SANE also defines a scorer-independent validation framework for generated judge tests. It determines whether each recipe agrees with scorer intent and whether each instantiated case realizes the recipe and supports its expected relation. Because SANE generates a recipe inventory tailored to each scorer, the number of aligned recipes grows as scorers are added. Across five domains, SANE produces 3.7 times as many validated test families as the Judge Reliability Harness (JRH) at comparable end-to-end validity and detects scorer deficiencies in every domain, including professionalism and toxicity, where JRH detects none. On faithfulness, SANE achieves strong human alignment in judge ranking: its test-suite scores rank 11 judge models consistently with human-labeled FaithBench performance (Spearman rho = 0.852), while JRH does not (rho = 0.100).

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Recipe, Computer science, Random testing, Benchmark (surveying), Test (biology), Reliability (semiconductor)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange