Newest papers

Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models

Daniel Robert Kling Alexander, Catherine Louise Kling · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.10506 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

大規模な言語モデルに現在置かれている質問の多くは、何のポリシーが価値があるか、どのオプションをユーザーが選ぶべきか、競合価値を重ねる方法について、正しい答えを持たない: 何のポリシーが価値があるか、何のポリシーが価値があるか、何十年もの間この問題に直面してきました。 それは、真の価値を知らずに調査の答えを判断し、有効性の枠組みと関連する概念を通じて:コンテンツ、構築、および基準の有効性、信頼性、奨励互換性、および結果性。我々は、この枠組みは言語モデルを評価するための一般的な方法であると主張し、我々はLLM評価のためのそれぞれの概念が何を意味するかを明らかに

Machine translation (Meta M2M100); the English original follows.

Abstract

Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of \$75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Valuation (finance), Test (biology), Set (abstract data type), Incentive, Economics, Contingent valuation

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange