Newest papers

Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination

Miguel Fernandez-de-Retana, Unai Zulaika, Rubén Sánchez, Aritz Bilbao-Jayo, Aitor Almeida · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31586 · cited by 0 works (OpenAlex) · open-access PDF

German

Die Genauigkeit der medizinischen Untersuchungen ist ein häufiger Beweis für die klinische Verwendung von Sprachmodellen, aber ein Gesundheitssystem, das beabsichtigt, ein offenes Modell auf seiner eigenen Hardware zu betreiben, muss wesentlich mehr wissen: ob das Modell erkennt, wenn es wahrscheinlich falsch ist, ob die medizinische Fine-Tuning diese Fähigkeit verbessert, ob seine korrekten Antworten auf klinischen klinischen Erwägungen beruhen oder ob es tatsächlich die Bilder verwendet, die es gibt. Wir untersuchen diese Fragen für vier offene Modelle von 8 bis 27 Milliarden Parametern auf der spanischen MIR-Residenzprüfung, wobei 1,552 Fragen aus acht Ausgaben unter den eigenen Regeln der Prüfung erzielt werden. Wenn jede Frage beantwortet wird, würden die drei größeren Mod

Machine translation (Meta M2M100); the English original follows.

Abstract

Accuracy on medical examinations is common evidence offered for the clinical use of language models, yet a health system that intends to run an open model on its own hardware needs to know considerably more: whether the model recognizes when it is likely to be wrong, whether medical fine-tuning improves this ability, whether its correct answers rest on sound clinical reasoning, or whether it actually uses the images it is given. We examine these questions for four open models of 8 to 27 billion parameters on the Spanish MIR residency examination, scoring 1,552 questions from eight editions under the exam's own rules. Answering every question, the three larger models would rank among roughly the first 600 to 3,000 candidates, with 41 to 43 specialties still open to them. Abstention, however, never pays, because even their least confident answers are right more often than the 25% break-even. Medical fine-tuning reduced overconfidence at the cost of accuracy, without improving how well confidence separates right from wrong answers. Moreover, 13-38% of correct answers rejected a distractor for a clinically unsound reason that confidence did not reflect, and removing or swapping the provided contextual images left accuracy unchanged.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Overconfidence effect, Computer science, Rank (graph theory), Artificial intelligence, Open data, Natural language processing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange