Newest papers

SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation

Lorenz Brehme, Benjamin Jeremiah Steenhoek, Ruth Breu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30047 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

编码代理通常是使用一次性,精确的提示进行评估的。 基准如 SWE-bench 使用现有的 GitHub 问题作为编码代理的输入,并评估代理是否成功解决了任务。 然而,在现实世界的框架使用情况下,提示往往不像现有的 GitHub 问题一样精确或结构化。 此外,编码代理会不会仅限于一个单一的提示; 它通常涉及多重互动,用户可以迭代添加新的任务和要求,代理必须满足。 为了衡量这些能力,我们开发了 SWE-DIVE,一个动态,交互式验证和评估框架,用于在整个多重转换环境中对编码代理的基准进行评估。 为了实现这一目标,我们引入了一个用户模拟器,可以模拟不同的行为

Machine translation (Meta M2M100); the English original follows.

Abstract

Coding agents are typically evaluated using one-shot, well-formulated prompts. Benchmarks such as SWE-bench use existing GitHub issues as input for a coding agent and evaluate whether the agent successfully solves the task. However, in real-world use cases, prompts are often not as well-formulated or structured as existing GitHub issues. Additionally, a coding agent session is not limited to a single prompt; it typically involves a multi-turn interaction in which the user iteratively adds new tasks and requirements that the agent must fulfill. To benchmark these abilities, we developed SWE-DIVE, a dynamic, interactive validation and evaluation framework for benchmarking coding agents in a multi-turn environment. To achieve this, we introduce a user simulator that can simulate different behavioral patterns and communicate with the coding agent. We also introduce a conversation validator that evaluates the user simulator's outputs to determine whether they fulfill the initial requirements and guide the agent toward successfully solving the task. The entire framework is embedded in a Harbor environment, which orchestrates the coding agent, user simulator, and conversation validator, and executes the test environment to determine whether the agent successfully solves the task. Our results and source code are available in our GitHub repository (https://github.com/lorenzbrehme/swe-dive_usersim).

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Coding (social sciences), Benchmarking, Conversation, Source code, Validator

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange