Newest papers

DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge

Atharva Thakur, Abulhair Saparov, Bruno Ribeiro · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28340 · cited by 0 works (OpenAlex) · open-access PDF

French

Nous introduisons DeclineCraft, un référentiel d'abstention dynamique dont les questions irresponsables visent les ajouts de fonctionnalités de Minecraft après un coup de corpus CAR, avec des audits sur l'exposition paramétrique possible. Les coupes ultérieures prennent en charge l'évaluation des modèles futurs. Au-delà de 27 LLM, nous mesurons le taux d'abstention correct (CAR; combien de fois les modèles reconnaissent des connaissances insuffisantes sur les questions irresponsables), le taux d'essai de réponse et l'exactitude des réponses CAR. Nous constatons que (1) le recours à un seul coup sur les modèles futurs augmente considérablement l'évaluation de 38 des 40 configurations (modèle et param

Machine translation (Meta M2M100); the English original follows.

Abstract

Reliable retrieval-augmented generation (RAG) requires recognizing when neither retrieved documents nor parametric knowledge supports an answer. We introduce DeclineCraft, a dynamic abstention benchmark whose unanswerable questions target Minecraft feature additions after a corpus cut-off, with audits of possible parametric exposure. Later cut-offs support evaluation of future models. Across 27 LLMs, we measure Correct Abstention Rate (CAR; how often models acknowledge insufficient knowledge on unanswerable questions), Answerable Attempt Rate and Answer Accuracy. We find that (1) single-shot retrieval significantly raises CAR in 38 of 40 configurations (model and thinking setting) versus no retrieval. (2) Iterative RAG, in which the model self-assesses its context and can request up to two further retrieval rounds, significantly lowers CAR versus single-shot RAG in 30 of 40 configurations (mean 46.4%), with little change in Answer Accuracy (68.5% to 69.3%). Adding query rewriting and reranking to hybrid (dense & sparse) retrieval lowers CAR for all 12 models tested. (3) Larger models answer more accurately (Spearman $ρ= +0.60$ with total parameters), but neither model size nor release date shows a significant association with CAR. (4) Without retrieval, enabling thinking on the same weights lowers CAR significantly for 8 of 13 models (by 14% points on average), and the responses shift from acknowledged uncertainty to premise denial.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Context (archaeology), Benchmark (surveying), Parametric statistics, Feature (linguistics), Premise

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange