Mohammed Saqlain · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.29812 · cited by 0 works (OpenAlex) · open-access PDF
文学代理人寻找语言代理人正在成为生命科学科学的科学助理的标准组成部分,但在他们的附件中存在的控制证据很少。我们在生物医学索赔验证(SciFact)上运行了控制的抽象,以实用的PubMed API作为唯一的工具。保持后骨和输出格式的标准组成部分,我们比较了四个抽象:封闭书答复,单击获取使用索赔作为查询的过度错误,一个ReAct式代理人写自己的PubMed查询,并以黄金抽象作为唯一的工具。保持后骨和输出格式的有效性,我们比较了四个抽象:仅为6%的索赔找到黄金纸,并将过度信任的错误转化为禁用,但没有提高准确性。
Machine translation (Meta M2M100); the English original follows.
Literature-searching language agents are becoming a standard component of scientific assistants for the life sciences, yet there is little controlled evidence on which parts of their harness matter. We run a controlled ablation on biomedical claim verification (SciFact) with the live PubMed API as the only tool. Holding the backbone and output format fixed, we compare four harnesses: closed-book answering, single-shot retrieval using the claim as the query, a ReAct-style agent that writes its own PubMed queries, and an oracle given the gold abstracts, each with two efficient frontier backbones. Naive single-shot retrieval finds the gold paper for only 6% of claims and turns overconfident errors into abstentions without improving accuracy. Letting the model formulate its own queries triples gold recall and yields the best verdicts (62% accuracy vs. 49-59% for the other realistic harnesses), although these differences are not significant at our sample size and agents use fewer than half of their search budget. With gold abstracts, the same models reach 79-86% accuracy, a significant 17-24 point gap showing that retrieval, not reasoning, is the main bottleneck. Finally, a citation audit shows that closed-book models cite PubMed IDs that almost always exist (327/335) but are almost never relevant: a cross-family LLM judge rates only 3 of 327 existing citations as relevant to the claim, versus 67% for citations produced by the search agent. Existence checks are therefore insufficient safeguards for scientific agents. We release code, prompts, and all cached tool calls at https://github.com/saqlain2204/real-pmids-wrong-papers.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Oracle, Computer science, Point (geometry), Gold standard (test), Information retrieval, Audit
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.29812, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange