Xialie Zhuang, Mingcong Lei, Zunhai Su, Ri Liu, Jianjin Li, Peixian Ma, Jia Pan, Ngai Wong, Guanhua Chen, Chuan-Wei Kuo · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.28522 · cited by 0 works (OpenAlex) · open-access PDF
Entscheidungsmodelle mit feinen Wahrscheinlichkeitsvorhersagen unterstützen Routing, Guardrails und Ausgangsüberprüfung in LLM-Systemen. Die Punktgenauigkeit allein kann jedoch nicht feststellen, ob Modelle die richtigen Entscheidungen über kontrollierte Variationen treffen oder zuverlässigen Vertrauen in ihre Vorhersagen verleihen. Wir stellen DecisionPivot ein, ein diagnostisches Benchmark, das 48 Aufgaben und 40.033 Elemente in sieben Entscheidungsrollen umfasst. Sein Kernspur enthält 7.800 Elemente aus 13 generierten Aufgabenfamilien, die in Kontrastgruppen mit computergestützten Antworten organisiert sind, die von unabhängig durchgeführten Lösern überprüft werden. Der ergänzende Referenzspur enthält 32.233 Elemente aus 35 öffentlichen Testsätzen, wobei
Machine translation (Meta M2M100); the English original follows.
Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or assign reliable confidence to their predictions. We introduce DecisionPivot, a diagnostic benchmark comprising 48 tasks and 40,033 items across seven decision roles. Its core track contains 7,800 items from 13 generated task families, organized into contrast groups with computed answers verified by independently implemented solvers. The complementary reference track contains 32,233 items from 35 public test sets, preserving their original evaluation metrics. We evaluate 40 decision models and 14 general models using documented, model-specific protocols, assessing item and contrast-group accuracy, probability ordering, calibration, and empirical selective coverage. Our results reveal substantial differences obscured by aggregate scores: two 27B models separated by only 2.6 points on the reference track achieve contrast-group accuracies of 70.1% and 44.2%. Across 2,400 binary contrast pairs, Perplexity Decider correctly ranks the positive member above the negative member in 92.3% of pairs but answers both members correctly in only 74.1%, exposing a gap between probability ordering and decision correctness. Models with similar item accuracy also differ in the reliability of their high-confidence predictions. On a separate 1,296-item subset, GLM-5.3 achieves 93.5\% contrast-group accuracy at a higher reasoning setting, compared with 81.5% at its lowest effort. By linking errors to pivotal details and difficulty levels, DecisionPivot provides fine-grained diagnostics of decision failures that aggregate evaluation alone cannot reveal.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Computer science, Benchmark (surveying), Task (project management), Aggregate (composite), Artificial intelligence, Machine learning
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.28522, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange