Newest papers

A Visual Grounding Benchmark for Retinal OCT with Multimodal Large Language Models

Ronald Fecso, José Morano, Botond Fazekas, Paul Widmann-Sedlnitzky, Dorota Himel, Laetita Hinterhuber, Virgínia Mares, Andreas Pollreisz, Hrvoje Bogunović · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31205 · cited by 0 works (OpenAlex) · open-access PDF

Russian

Мултимодальные фино-специфические модели большого языка (MLLMs) могут ответить на клинические вопросы о медицинских изображениях, но надежное использование требует визуального заземления: модели должны локализовать патологию, о которой они сообщают, а не полагаться на неверные сокращения. Может ли MLLMs визуально заземлять клинические результаты в сетчатой оптической кохерентной томографии (OCT), наиболее широко используемой модели визуализации в офтальмологии, остается в значительной степени неисследованной. Чтобы устранить этот пробел, мы вводим RetinaGroundBench, первый ориентир для визуального заземления биомаркеров в сетчатой ОКТ. Он включает в себя 815 Б-сканов, полностью отмеченных офтальмологами с

Machine translation (Meta M2M100); the English original follows.

Abstract

Multimodal large language models (MLLMs) can answer clinical questions about medical images, but trustworthy use requires visual grounding: models should localize the pathology they report rather than rely on spurious shortcuts. Whether MLLMs can visually ground clinical findings in retinal optical coherence tomography (OCT), the most widely used imaging modality in ophthalmology, remains largely unexplored. To address this gap, we introduce RetinaGroundBench, the first benchmark for biomarker visual grounding in retinal OCT. It comprises 815 B-scans exhaustively annotated by ophthalmologists with 4,537 bounding boxes covering 13 biomarkers across 8 diseases. Because MLLMs often fragment a single lesion into multiple partial predictions, we propose AggIoU, a metric that evaluates whether the union of overlapping predicted boxes covers the annotated region. Across 27 open-source MLLMs (0.8-38B parameters), we find that visual grounding remains largely unsolved: the best general-purpose model achieves a macro F1 of just 23.7%, with consistently low recall, despite reaching up to 73% balanced accuracy on biomarker presence questions. To narrow this gap, we developed RetGrounder, an MLLM fine-tuned using an OCT-specific strategy integrating grounding and textual information. Despite its relatively small size (8B), RetGrounder improves grounding F1 to 33.9%, a 43% relative improvement. Our benchmark provides a reference point for measuring progress in the field of OCT diagnosis with MLLMs.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Benchmark (surveying), Artificial intelligence, Metric (unit), Modality (human–computer interaction), Spurious relationship

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange