Hossein A. Rahmani, Bin Wu, Chaeeun Lee, Fanghua Ye, Sahan Bulathwela, Sujay Kumar Jauhar, Marzieh Fadaee, Nick Craswell, Emine Yılmaz · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.26679 · cited by 0 works (OpenAlex) · open-access PDF
Die Bewertung der Planungsfähigkeit eines Agenten bleibt für Open-Domain-Aufgaben eine Herausforderung, wo die Verifizierung von Verfahren durch Ausführung oft unpraktisch ist. Die bestehenden Bewertungen beurteilen in der Regel nur den Endplan und unterscheiden keine Verfahrensfehler wie fehlende, falsche oder missordnete Aktionen. Wir führen STEPBENCH ein, ein Benchmark von 6.885 verifizierten Verfahren, die von wikiHow in 81 Open-Domain-Kategorien abgerufen wurden. Aus jedem Verfahren ergeben wir 5 Diagnoseaufgaben: Vier mit deterministischen Antworten, die enthalten, falsch oder verzerrt Schritte beinhalten, und einer, der die nächste Stufe-Generation erfordert. Wir bewerten zusätzlich die Vollplan-Generation,
Machine translation (Meta M2M100); the English original follows.
Evaluating an agent's planning ability remains challenging for open-domain tasks, where verifying procedures through execution is often impractical. Existing evaluations typically assess only the final plan and do not distinguish procedural errors such as missing, incorrect, or misordered actions. We introduce STEPBENCH, a benchmark of 6,885 verified procedures crawled from wikiHow across 81 open-domain categories. From each procedure, we derive 5 diagnostic tasks: four with deterministic answers involving withheld, incorrect, or shuffled steps, and one requiring next-step generation. We additionally evaluate full-plan generation to examine how diagnostic performance correlates with end-to-end plan validity. Using the same procedures across tasks lets us directly compare different planning capabilities. For generation, STEPEVAL identifies critical failures that prevent goal completion. Across 14 configurations of 12 models, we find substantial differences across planning capabilities, with step ordering proving the most difficult: the strongest model correctly orders only 63.0% of procedures, dropping to 33.1% for those with nine or more steps. Incorrect-step detection also predicts judged plan validity across models, suggesting a lower-cost alternative for evaluating generation quality. On STEPBENCH-Val, comprising 214 procedures published in 2026, comprehension scores and rankings remain stable, suggesting limited influence from memorisation.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Computer science, Plan (archaeology), Artificial intelligence, Machine learning, Benchmark (surveying), Comprehension
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.26679, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange