Newest papers

How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions

Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias, Thaís Gaudêncio, Yuri Malheiros, Barufaldi · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31606 · cited by 0 works (OpenAlex) · open-access PDF

Korean

목적: 유효한 유방 손상 관리 문서에 대한 클리닉 수신 증폭 (RAG) 순위 시스템을위한 증거 인식 복구 평가 프로토콜을 설명하기 위해, 항목 커버에 대한 복구 점수의 의존성을 해결합니다. 방법: 18 문서 코르푸스와 18 쿼리 개발 벤치마크, 문서당 한 개의 예/아니오 또는 WH-문제가 포함 된 증거 범위에 의해 정의되었습니다. Retrieval은 밀접한 삽입, BM25 및 크로스 코더 리 랜킹을 결합했습니다. 128 ~ 1024 토큰, 0 ~ 20 %의 중복 및 4 개의 리 랜버 구성이 평가되었습니다. Relevance는 증거 범위가 모든 증거를 포함하는 증거를 매핑하여 증거에 의해 정의

Machine translation (Meta M2M100); the English original follows.

Abstract

Purpose: To describe an evidence-aware retrieval evaluation protocol for a clinical retrieval-augmented generation (RAG) system over breast lesion management literature, addressing the dependence of retrieval scores on annotation coverage. Methods: An 18-document corpus and an 18-query development benchmark, with one Yes/No or WH-question per document, were used. Retrieval combined dense embeddings, BM25, and cross-encoder reranking. Chunk sizes of 128 to 1024 tokens, overlaps of 0 to 20%, and four retriever configurations were evaluated. Relevance was defined by evidence spans mapped to all containing chunks, treating overlapping chunks with equivalent evidence as relevant. Retrieval was evaluated using R-Precision, Recall@K, MRR, and nDCG@K. Separately, generation was assessed on a 512-token index at K=1 to 5 using DeepEval claim-level metrics, with deepseek-v4-pro as the LLM judge, and BLEU/ROUGE metrics. Results: The selected hybrid retriever, using 384-token chunks and 20% overlap, retrieved accepted evidence in the top three results for 17 of 18 queries. Recall@3, R-Precision, MRR@3, and nDCG@3 each reached 0.94. Ground-truth curation increased evidence spans from 25 to 37 and accepted chunk targets from 36 to 51 without changing retrieval rankings. Conclusion: Cross-encoder reranking achieved the highest retrieval scores. Expanded relevance annotations increased recognized evidence without changing rankings, highlighting annotation sensitivity. The small development set and retrieval-informed curation require independent validation, including radiologist assessment of evidence sufficiency.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Annotation, Computer science, Information retrieval, Relevance (law), Protocol (science), Set (abstract data type)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange