Newest papers

SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges

Ethan Ludwick, Yu-Chung Hsiao · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28886 · cited by 0 works (OpenAlex) · open-access PDF

Korean

LLM 순위 판사 SANBen은 점점 더 벤치마크 평가, 모델 선택, 데이터 코레이션 및 훈련을 중재합니다. 판사의 결정은 각 판사가 점수 기준에 영향을 미치는 변화를 신뢰할 수있게 반응하는 경우에만 유용합니다. 신뢰성 테스트를위한 기존 방법은 판사 프레임에 알려진 예상 효과가있는 판사 프레임을 방해합니다. 유명한 LLM 판사 테스트 프레임은 인간이 작성한 혼란 요법과 합성 테스트 제조 지침을 결합하여 점수 특정 요법 아이디어를 자동화 된 파이프 라인 밖으로 남겨줍니다. 이것은 최초의 파이프 라인은 점수 특정 능력을 추가하는지 여부를 포함하여 점수 특정 요법에 부족한 점수를 제공

Machine translation (Meta M2M100); the English original follows.

Abstract

LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion. Existing methods for reliability testing perturb inputs with known expected effects on judge scores. Prominent LLM-judge testing harnesses combine human-authored perturbation recipes with synthetic test generation, leaving scorer-specific recipe ideation outside the automated pipeline. This limits their use to fixed perturbations and pre-assumed scorer types. We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators as reusable agent instructions. Given a scorer specification, implementation, and representative inputs, the pipeline first identifies scorer-specific capabilities, then automatically derives perturbation recipes and expected score relations for each. SANE also defines a scorer-independent validation framework for generated judge tests. It determines whether each recipe agrees with scorer intent and whether each instantiated case realizes the recipe and supports its expected relation. Because SANE generates a recipe inventory tailored to each scorer, the number of aligned recipes grows as scorers are added. Across five domains, SANE produces 3.7 times as many validated test families as the Judge Reliability Harness (JRH) at comparable end-to-end validity and detects scorer deficiencies in every domain, including professionalism and toxicity, where JRH detects none. On faithfulness, SANE achieves strong human alignment in judge ranking: its test-suite scores rank 11 judge models consistently with human-labeled FaithBench performance (Spearman rho = 0.852), while JRH does not (rho = 0.100).

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Recipe, Computer science, Random testing, Benchmark (surveying), Test (biology), Reliability (semiconductor)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange