Daniel Robert Kling Alexander, Catherine Louise Kling · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.10506 · cited by 0 works (OpenAlex) · open-access PDF
Banyak pertanyaan yang sekarang ditempatkan pada model bahasa besar tidak memiliki jawaban yang benar untuk skor terhadap: kebijakan apa yang layak, opsi mana yang harus dipilih pengguna, bagaimana menimbang nilai kompetitif. Ekonomi preferensi telah menghadapi masalah ini selama beberapa dekade. Ini menilai jawaban survei tanpa mengetahui nilai sebenarnya, melalui kerangka validitas dan konsep terkait: konten, konstruksi, dan validitas kriteria, keandalan, kompatibilitas insentif, dan konsekuensi. Kami berpendapat bahwa kerangka ini adalah metode umum untuk mengevaluasi model bahasa, dan kami menetapkan apa yang berarti setiap konsep untuk evaluasi LLM. Kami menunjukkan pendekatan menggunakan survei preferensi ekonomi yang diterbitkan pada kualitas air (Vossler et al. 2023) yang diadministrasikan ke enam model. Dalam aplikasi ekonomi ini, tes validitas mengambil bentuk prediksi dari teori ekonomi: permintaan harus turun, dan keinginan untuk membayar harus menangg
Machine translation (Meta M2M100); the English original follows.
Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of \$75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Valuation (finance), Test (biology), Set (abstract data type), Incentive, Economics, Contingent valuation
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.10506, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange