Newest papers

How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions

Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias, Thaís Gaudêncio, Yuri Malheiros, Barufaldi · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31606 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Tujuan: Untuk mendeskripsikan protokol evaluasi pengambilan bukti yang menyadari kurva untuk sistem pengambilan klinis yang diperluas (RAG) atas literatur manajemen lesi payudara yang valid, yang menangani ketergantungan skor pengambilan pada cakupan anotasi. Metode: Corpus 18 dokumen dan benchmark pengambilan 18 pertanyaan, dengan satu Yes/No atau pertanyaan WH-per dokumen, digunakan. Retrieval menggabungkan embeddings padat, BM25, dan reranking cross-encoder. Ukuran Chunk dari 128 hingga 1024 token, kelipatan dari 0 hingga 20%, dan empat konfigurasi retriever dievaluasi. Relevansi didefinisikan oleh spans bukti yang dipaparkan dengan semua bukti yang mengandung potongan, menangani potongan-potongan overlapping dengan bukti yang setara tanpa relevansi. Retrieval

Machine translation (Meta M2M100); the English original follows.

Abstract

Purpose: To describe an evidence-aware retrieval evaluation protocol for a clinical retrieval-augmented generation (RAG) system over breast lesion management literature, addressing the dependence of retrieval scores on annotation coverage. Methods: An 18-document corpus and an 18-query development benchmark, with one Yes/No or WH-question per document, were used. Retrieval combined dense embeddings, BM25, and cross-encoder reranking. Chunk sizes of 128 to 1024 tokens, overlaps of 0 to 20%, and four retriever configurations were evaluated. Relevance was defined by evidence spans mapped to all containing chunks, treating overlapping chunks with equivalent evidence as relevant. Retrieval was evaluated using R-Precision, Recall@K, MRR, and nDCG@K. Separately, generation was assessed on a 512-token index at K=1 to 5 using DeepEval claim-level metrics, with deepseek-v4-pro as the LLM judge, and BLEU/ROUGE metrics. Results: The selected hybrid retriever, using 384-token chunks and 20% overlap, retrieved accepted evidence in the top three results for 17 of 18 queries. Recall@3, R-Precision, MRR@3, and nDCG@3 each reached 0.94. Ground-truth curation increased evidence spans from 25 to 37 and accepted chunk targets from 36 to 51 without changing retrieval rankings. Conclusion: Cross-encoder reranking achieved the highest retrieval scores. Expanded relevance annotations increased recognized evidence without changing rankings, highlighting annotation sensitivity. The small development set and retrieval-informed curation require independent validation, including radiologist assessment of evidence sufficiency.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Annotation, Computer science, Information retrieval, Relevance (law), Protocol (science), Set (abstract data type)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange