Newest papers

OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics

Gideon Shaked, Siddharth Nagisetty · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.21074 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

计算机生物学家越来越多地将实质性分析分配给AI代理人,在长视野、多回合会场,但大多数基准代理人的生物信息学能力都提出了一个单一的短任务,一个单击提示。我们介绍了OmicsBench,一个长视野、多回合的AI代理人的生物信息学基准。在其97个任务中,每一个都将依赖的子任务链接到一个导向的非循环图,为662个底层子任务总共。每个子任务都被定义为一个用户的转换和定义对一个答案密钥的分级,在12个模型上对不正确的答案进行可选的多回合生物信息学基准。在三个管道上,最慢的代理平均值为每一个任务3,6小时,并且没有组合接近

Machine translation (Meta M2M100); the English original follows.

Abstract

Computational biologists increasingly delegate substantial analyses to AI agents in long-horizon, multi-turn sessions, yet most benchmarks of agents' bioinformatics capabilities pose a single short task with a single-shot prompt. We present OmicsBench, a long-horizon, multi-turn bioinformatics benchmark for AI agents. Each of its 97 tasks links dependent subtasks into a directed acyclic graph, for 662 underlying subtasks in all. Each subtask is posed as one user turn and graded deterministically against an answer key, with optional in-context feedback on incorrect answers. Across 12 models on three harnesses, the slowest agent averages 3.6 hours per task, and no combination comes near saturating the benchmark, with mean scores of 0.43 to 0.57 out of a possible 1 without in-context feedback and 0.54 to 0.67 with it. Multi-turn evaluation also uncovers behavior that single-shot prompts cannot, as agents score lower on subtasks that depend on earlier subtasks, and in-context feedback helps agents recover some of that loss. Because deterministic scoring gives no qualitative account of how an agent failed, we also introduce an unsupervised failure mode discovery pipeline that describes, embeds, and clusters agents' failed attempts to surface emergent failure modes. Applied to OmicsBench, this pipeline exposes qualitative shortcomings in how agents carry out bioinformatics analyses, ranging from agents applying their own statistical filters in place of the specified ones to agents writing whole answers from memory.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Delegate, Benchmark (surveying), Pipeline (software), Computer science, Task (project management), Artificial intelligence

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange