Newest papers

DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge

Atharva Thakur, Abulhair Saparov, Bruno Ribeiro · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28340 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

A geração de pré-recuperação confiável e aumentada da mesma data (RAG) requer reconhecimento quando nem os documentos recuperados nem o conhecimento paramétrico suportam uma resposta. Introduzimos o DeclineCraft, um benchmark de abstenção dinâmico cujas perguntas irrespondíveis visam as adições de recursos do Minecraft após um corte de corpus, com auditorias de possível exposição paramétrica. Os descontos posteriores suportam a avaliação de modelos futuros. Ao longo de 27 LLM, medimos a taxa de abstenção correta (CAR; com que frequência os modelos reconhecem conhecimento insuficiente sobre perguntas não respondidas), a taxa de tentativa de resposta e a precisão de resposta do CAR. Descobrimos que (1) a recuperação de um único tiro aumenta significativamente em 38 de 40 configurações (modelo

Machine translation (Meta M2M100); the English original follows.

Abstract

Reliable retrieval-augmented generation (RAG) requires recognizing when neither retrieved documents nor parametric knowledge supports an answer. We introduce DeclineCraft, a dynamic abstention benchmark whose unanswerable questions target Minecraft feature additions after a corpus cut-off, with audits of possible parametric exposure. Later cut-offs support evaluation of future models. Across 27 LLMs, we measure Correct Abstention Rate (CAR; how often models acknowledge insufficient knowledge on unanswerable questions), Answerable Attempt Rate and Answer Accuracy. We find that (1) single-shot retrieval significantly raises CAR in 38 of 40 configurations (model and thinking setting) versus no retrieval. (2) Iterative RAG, in which the model self-assesses its context and can request up to two further retrieval rounds, significantly lowers CAR versus single-shot RAG in 30 of 40 configurations (mean 46.4%), with little change in Answer Accuracy (68.5% to 69.3%). Adding query rewriting and reranking to hybrid (dense & sparse) retrieval lowers CAR for all 12 models tested. (3) Larger models answer more accurately (Spearman $ρ= +0.60$ with total parameters), but neither model size nor release date shows a significant association with CAR. (4) Without retrieval, enabling thinking on the same weights lowers CAR significantly for 8 of 13 models (by 14% points on average), and the responses shift from acknowledged uncertainty to premise denial.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Context (archaeology), Benchmark (surveying), Parametric statistics, Feature (linguistics), Premise

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange