Newest papers

DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge

Atharva Thakur, Abulhair Saparov, Bruno Ribeiro · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28340 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

DeclineCraft, un benchmark de abstención dinámico cuyas preguntas irresponsables se dirigen a las adiciones de características de Minecraft después de un corpus cut-off, con auditorías sobre la posible exposición paramétrica. Los cortes posteriores respaldan la evaluación de los modelos futuros. A través de 27 LLM, medimos la tasa de abstención correcta (CAR; con qué frecuencia los modelos reconocen el conocimiento insuficiente sobre las preguntas irresponsables), la tasa de intento de respuesta y la exactitud de respuesta de CAR. Encontramos que (1) la recuperación de disparo único de CAR aumenta significativamente en 38 de las 40 configuraciones (modelo y configuración) frente a la no-recuperación. (2) la tasa de abstención Iterativa de RAG, en la que el modelo se autoevalúa y puede solicitar

Machine translation (Meta M2M100); the English original follows.

Abstract

Reliable retrieval-augmented generation (RAG) requires recognizing when neither retrieved documents nor parametric knowledge supports an answer. We introduce DeclineCraft, a dynamic abstention benchmark whose unanswerable questions target Minecraft feature additions after a corpus cut-off, with audits of possible parametric exposure. Later cut-offs support evaluation of future models. Across 27 LLMs, we measure Correct Abstention Rate (CAR; how often models acknowledge insufficient knowledge on unanswerable questions), Answerable Attempt Rate and Answer Accuracy. We find that (1) single-shot retrieval significantly raises CAR in 38 of 40 configurations (model and thinking setting) versus no retrieval. (2) Iterative RAG, in which the model self-assesses its context and can request up to two further retrieval rounds, significantly lowers CAR versus single-shot RAG in 30 of 40 configurations (mean 46.4%), with little change in Answer Accuracy (68.5% to 69.3%). Adding query rewriting and reranking to hybrid (dense & sparse) retrieval lowers CAR for all 12 models tested. (3) Larger models answer more accurately (Spearman $ρ= +0.60$ with total parameters), but neither model size nor release date shows a significant association with CAR. (4) Without retrieval, enabling thinking on the same weights lowers CAR significantly for 8 of 13 models (by 14% points on average), and the responses shift from acknowledged uncertainty to premise denial.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Context (archaeology), Benchmark (surveying), Parametric statistics, Feature (linguistics), Premise

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange