Newest papers

RAG-NIMBF: A Non-Invasive, Modular Benchmarking Framework for Automated RAG Configuration Analysis

Niclas Hart, Nicolas Weeger · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.27816 · cited by 0 works (OpenAlex) · open-access PDF

German

Retrieval-Augmented Generation (RAG) ist das dominante Paradigma für die Ausführung von Rohrprodukten von großen Sprachmodellen (LLM) in externe Kenntnisse geworden. Jedoch stehen Praktizierende mit einem großen Konfigurationsraum konfrontiert, der Chunking, Embedding, Retrieval, Reranking, Prompting und Generation Modelle umfasst, und es gibt begrenzte Anleitung darüber, wie diese Komponenten interagieren. Wir präsentieren RAG-NIMBF, ein modulares Benchmarking-Framework, das die bestehenden RAG-Pipelines als eine nicht-invasive Bewertungsschicht beinhaltet. Es bewertet den Konfigurationsraum durch kontrollierte Ablation und Generation mit End-to-End-Experiment-Tracking. Die Be

Machine translation (Meta M2M100); the English original follows.

Abstract

Retrieval-Augmented Generation (RAG) has become the dominant paradigm for grounding the outputs of large language models (LLM) in external knowledge. However, practitioners are faced with a vast configuration space comprising chunking, embedding, retrieval, reranking, prompting and generation models, and there is limited guidance on how these components interact. We present RAG-NIMBF, a modular benchmarking framework that attaches to existing RAG pipelines as a non-invasive evaluation layer. It evaluates the configuration space through controlled ablation with end-to-end experiment tracking. The evaluation combines LLM-based answer quality scores, retrieval correctness measures, and runtime indicators. This study validates the framework across multiple generation models on the SQuAD dataset. This process reveals non-obvious interactions and demonstrates how the experiments are tracked. The findings indicate that the retrieval balance of relevance and diversity substantially enhances the relevance and coverage of retrieved context in comparison to a conventional similarity search, with MMR raising nDCG@5 by +4 to +11 percentage points across the swept models. More detailed prompt instructions meaningfully increase how faithfully generated answers stick to the retrieved evidence, especially for capable and reasoning models (+10 to +13pp on Qwen3.6-27B, GPT-4o-mini and Qwen3.5-think, but negligible on the 20B generator). Overall, RAG-NIMBF transforms the configuration space of RAG into reproducible, comparable experiments. By attaching to existing production pipelines in a non-invasive way, it enables practitioners to systematically evaluate and improve deployed systems without disrupting existing workflows. This replaces ad hoc choices with evidence-based decisions.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Correctness, Benchmarking, Modular design, Context (archaeology), Relevance (law)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange