Atharva Thakur, Abulhair Saparov, Bruno Ribeiro · 2026 · arXiv
DOI: 10.48550/arXiv.2610.28340 · open-access PDF
Wir stellen DeclineCraft ein, einen dynamischen Abstinenz-Benchmark, dessen unbeantwortete Fragen auf Minecraft-Feature-Additions nach einem Corpus-Cut-off abzielen, mit Audits der möglichen parametrischen Exposition. Spätere Cut-offs unterstützen die Bewertung zukünftiger Modelle. Über 27 LLMs messen wir Correct Abstention Rate (CAR; wie oft Modelle unzureichende Kenntnisse über unbeantwortete Fragen erkennen), Answerbare Versuchsrate und Antwortgenauigkeit.
Machine translation (Meta M2M100); the English original follows.
Reliable retrieval-augmented generation (RAG) requires recognizing when neither retrieved documents nor parametric knowledge supports an answer. We introduce DeclineCraft, a dynamic abstention benchmark whose unanswerable questions target Minecraft feature additions after a corpus cut-off, with audits of possible parametric exposure. Later cut-offs support evaluation of future models. Across 27 LLMs, we measure Correct Abstention Rate (CAR; how often models acknowledge insufficient knowledge on unanswerable questions), Answerable Attempt Rate and Answer Accuracy. We find that (1) single-shot r
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.28340, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange