Mohammed Saqlain · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.29812 · cited by 0 works (OpenAlex) · open-access PDF
Los agentes de búsqueda de agentes de lenguaje se están convirtiendo en un componente estándar de los asistentes científicos para las ciencias de la vida, sin embargo, hay poca evidencia controlada sobre cuáles son las partes de su arnés. Executamos una ablación controlada en la verificación de reclamaciones biomédicas (SciFact) con la API de PubMed en vivo como única herramienta. Manteniendo el esqueleto y el formato de salida fijo, comparamos cuatro arnés: la respuesta de libro cerrado, la recuperación de disparo único utilizando la misma afirmación como herramienta de publicación, un agente de estilo ReAct que escribe sus propias consultas de PubMed, y un oráculo dado a los abstractos de oro, cada uno con dos esqueletos fronterizos eficientes. El ingenio de búsqueda
Machine translation (Meta M2M100); the English original follows.
Literature-searching language agents are becoming a standard component of scientific assistants for the life sciences, yet there is little controlled evidence on which parts of their harness matter. We run a controlled ablation on biomedical claim verification (SciFact) with the live PubMed API as the only tool. Holding the backbone and output format fixed, we compare four harnesses: closed-book answering, single-shot retrieval using the claim as the query, a ReAct-style agent that writes its own PubMed queries, and an oracle given the gold abstracts, each with two efficient frontier backbones. Naive single-shot retrieval finds the gold paper for only 6% of claims and turns overconfident errors into abstentions without improving accuracy. Letting the model formulate its own queries triples gold recall and yields the best verdicts (62% accuracy vs. 49-59% for the other realistic harnesses), although these differences are not significant at our sample size and agents use fewer than half of their search budget. With gold abstracts, the same models reach 79-86% accuracy, a significant 17-24 point gap showing that retrieval, not reasoning, is the main bottleneck. Finally, a citation audit shows that closed-book models cite PubMed IDs that almost always exist (327/335) but are almost never relevant: a cross-family LLM judge rates only 3 of 327 existing citations as relevant to the claim, versus 67% for citations produced by the search agent. Existence checks are therefore insufficient safeguards for scientific agents. We release code, prompts, and all cached tool calls at https://github.com/saqlain2204/real-pmids-wrong-papers.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Oracle, Computer science, Point (geometry), Gold standard (test), Information retrieval, Audit
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.29812, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange