Mohammed Saqlain · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.29812 · cited by 0 works (OpenAlex) · open-access PDF
문서 검색 에이전트 보안 언어 에이전트는 생명 과학을위한 과학 보조자의 표준 구성 요소가되고있다, 그러나 그들의 헬멧의 어떤 부분에 대한 통제 된 증거는 거의 없습니다. 우리는 라이브 PubMed API를 유일한 도구로 라이브 PubMed API를 사용하여 생물학적 주장 검증 (SciFact)에 대한 통제 된 ablation을 실행합니다. 뒷면과 출력 형식이 고정되어있는 각각의 효율적 인 경계 backbones를 보유하고 있습니다. 순진한 단일 샷 리트리벨은 문서 응답, 쿼리로 같은 주장을 사용하는 단일 샷 리트리벨, 자신의 PubMed 쿼리를 작성하는 ReAct 스타일 에이전트, 그리고 금 추론을 제공하는 an oracle를
Machine translation (Meta M2M100); the English original follows.
Literature-searching language agents are becoming a standard component of scientific assistants for the life sciences, yet there is little controlled evidence on which parts of their harness matter. We run a controlled ablation on biomedical claim verification (SciFact) with the live PubMed API as the only tool. Holding the backbone and output format fixed, we compare four harnesses: closed-book answering, single-shot retrieval using the claim as the query, a ReAct-style agent that writes its own PubMed queries, and an oracle given the gold abstracts, each with two efficient frontier backbones. Naive single-shot retrieval finds the gold paper for only 6% of claims and turns overconfident errors into abstentions without improving accuracy. Letting the model formulate its own queries triples gold recall and yields the best verdicts (62% accuracy vs. 49-59% for the other realistic harnesses), although these differences are not significant at our sample size and agents use fewer than half of their search budget. With gold abstracts, the same models reach 79-86% accuracy, a significant 17-24 point gap showing that retrieval, not reasoning, is the main bottleneck. Finally, a citation audit shows that closed-book models cite PubMed IDs that almost always exist (327/335) but are almost never relevant: a cross-family LLM judge rates only 3 of 327 existing citations as relevant to the claim, versus 67% for citations produced by the search agent. Existence checks are therefore insufficient safeguards for scientific agents. We release code, prompts, and all cached tool calls at https://github.com/saqlain2204/real-pmids-wrong-papers.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Oracle, Computer science, Point (geometry), Gold standard (test), Information retrieval, Audit
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.29812, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange