Newest papers

Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination

Miguel Fernandez-de-Retana, Unai Zulaika, Rubén Sánchez, Aritz Bilbao-Jayo, Aitor Almeida · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31586 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

医学检查的准确性是用于语言模型的临床使用的常见证据,但一个计划在其自身硬件上运行一个开放模型的医疗系统需要知道更多:模型是否认识到它可能错误时,医学细节调整是否提高了这种能力,其正确的答案是否依赖于健全的临床推理,或者是否实际上使用它所提供的图像。我们对西班牙MIR居住考试的8至27亿参数的4个开放模型的这些问题进行审查,在考试本身的规则下获得了1552个问题。回答每一个问题时,三大模型将排在首批600至3000名候选人中,41至43个专业仍然对他们开放。然而,弃权从未支付,因为至少他们的自信的答案往往比25%更为正确。医学细节

Machine translation (Meta M2M100); the English original follows.

Abstract

Accuracy on medical examinations is common evidence offered for the clinical use of language models, yet a health system that intends to run an open model on its own hardware needs to know considerably more: whether the model recognizes when it is likely to be wrong, whether medical fine-tuning improves this ability, whether its correct answers rest on sound clinical reasoning, or whether it actually uses the images it is given. We examine these questions for four open models of 8 to 27 billion parameters on the Spanish MIR residency examination, scoring 1,552 questions from eight editions under the exam's own rules. Answering every question, the three larger models would rank among roughly the first 600 to 3,000 candidates, with 41 to 43 specialties still open to them. Abstention, however, never pays, because even their least confident answers are right more often than the 25% break-even. Medical fine-tuning reduced overconfidence at the cost of accuracy, without improving how well confidence separates right from wrong answers. Moreover, 13-38% of correct answers rejected a distractor for a clinically unsound reason that confidence did not reflect, and removing or swapping the provided contextual images left accuracy unchanged.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Overconfidence effect, Computer science, Rank (graph theory), Artificial intelligence, Open data, Natural language processing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange