Newest papers

Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models

Daniel Robert Kling Alexander, Catherine Louise Kling · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.10506 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Muchas de las preguntas que ahora se plantean a los modelos de idiomas grandes no tienen una respuesta correcta para puntuar contra: qué política vale la pena, qué opción debe elegir un usuario, cómo ponderar los valores competidores. La economía de preferencia declarada ha enfrentado este problema durante décadas. juzga las respuestas de la encuesta sin conocer el valor verdadero, a través de un marco de validez y conceptos relacionados: contenido, construcción y validez de criterio, fiabilidad, compatibilidad de incentivos y consecuencia. Argumentamos que este marco es un método general para evaluar modelos de idiomas, y exponemos lo que cada concepto significa para la evaluación de LLM. Demostramos el enfoque utilizando una encuesta de valoración económica de preferencia declarada de calidad del agua publicada (Vossler et al. 2023) administrada a seis modelos. En esta

Machine translation (Meta M2M100); the English original follows.

Abstract

Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of \$75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Valuation (finance), Test (biology), Set (abstract data type), Incentive, Economics, Contingent valuation

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange