Newest papers

SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation

Lorenz Brehme, Benjamin Steenhoek, Ruth Breu · 2026 · arXiv

DOI: 10.48550/arXiv.2610.30047 · open-access PDF

Japanese

SWEベンチのようなベンチマークは、既存のGitHub問題を暗号化エージェントの入力として使用し、エージェントがタスクを成功に解決するかどうかを評価します。しかし、現実世界の使用事例では、プロンプトはしばしば既存のGitHub問題と同様に formulatedまたは構造化されていません。さらに、コーディングエージェントのセッションは単一のプロンプトに限定されていません。

Machine translation (Meta M2M100); the English original follows.

Abstract

Coding agents are typically evaluated using one-shot, well-formulated prompts. Benchmarks such as SWE-bench use existing GitHub issues as input for a coding agent and evaluate whether the agent successfully solves the task. However, in real-world use cases, prompts are often not as well-formulated or structured as existing GitHub issues. Additionally, a coding agent session is not limited to a single prompt; it typically involves a multi-turn interaction in which the user iteratively adds new tasks and requirements that the agent must fulfill. To benchmark these abilities, we developed SWE-DIV

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange