Newest papers

Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination

Miguel Fernandez-de-Retana, Unai Zulaika, Rubén Sánchez, Aritz Bilbao-Jayo, Aitor Almeida · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31586 · cited by 0 works (OpenAlex) · open-access PDF

Korean

의학 검사에 대한 정확성은 언어 모델의 임상 사용을 위해 제공되는 일반적인 증거이지만, 자체 하드웨어에서 개방형 모델을 실행하려는 건강 시스템은 모델이 잘못 될 가능성이 있는지, 의학 정렬이이 능력을 향상시키는지, 올바른 답변이 선명한 임상 추론에 의존하는지, 또는 실제로 제공되는 이미지를 사용하는지에 대해 상당히 더 많은 것을 알아야합니다. 우리는 스페인 MIR 거주 시험에 8 ~ 27 억 매개 변수의 4 개 개 개의 개방형 모델에 대한이 질문을 검토하고, 시험의 자체 규칙에 따라 8 개의 에디션에서 1,552 개의 질문을 점수합니다. 각 질문에 대답하면 세 개의 큰 모델이 처음 600 ~ 3,000 명의 후보자들 사이에

Machine translation (Meta M2M100); the English original follows.

Abstract

Accuracy on medical examinations is common evidence offered for the clinical use of language models, yet a health system that intends to run an open model on its own hardware needs to know considerably more: whether the model recognizes when it is likely to be wrong, whether medical fine-tuning improves this ability, whether its correct answers rest on sound clinical reasoning, or whether it actually uses the images it is given. We examine these questions for four open models of 8 to 27 billion parameters on the Spanish MIR residency examination, scoring 1,552 questions from eight editions under the exam's own rules. Answering every question, the three larger models would rank among roughly the first 600 to 3,000 candidates, with 41 to 43 specialties still open to them. Abstention, however, never pays, because even their least confident answers are right more often than the 25% break-even. Medical fine-tuning reduced overconfidence at the cost of accuracy, without improving how well confidence separates right from wrong answers. Moreover, 13-38% of correct answers rejected a distractor for a clinically unsound reason that confidence did not reflect, and removing or swapping the provided contextual images left accuracy unchanged.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Overconfidence effect, Computer science, Rank (graph theory), Artificial intelligence, Open data, Natural language processing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange