Newest papers

DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems

Xialie Zhuang, Mingcong Lei, Zunhai Su, Ri Liu, Jianjin Li, Peixian Ma, Jia Pan, Ngai Wong · 2026 · arXiv

DOI: 10.48550/arXiv.2610.28522 · open-access PDF

Japanese

決定モデルは、LLMシステムにおけるルーティング、ガードレイル、出力検証をサポートします。しかし、項目の精度だけでは、モデルがコントロールされた変数間で正しい決定を下すか、または予測に信頼性の高い信頼を付与するかを確立することはできません。我々は、48のタスクと7つの意思決定の役割で40033の項目を含む診断基準であるDecisionPivotを導入します。

Machine translation (Meta M2M100); the English original follows.

Abstract

Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or assign reliable confidence to their predictions. We introduce DecisionPivot, a diagnostic benchmark comprising 48 tasks and 40,033 items across seven decision roles. Its core track contains 7,800 items from 13 generated task families, organized into contrast groups with computed answers verified by independently implemented solvers. The complementary reference track contains 32,233 items from 35

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange