Newest papers

AgenTest: Mining Business Rules to Test LLM Agents for Business Logic Bugs

Lili Quan, L. Wang, Yuelin Wang, Mingfei Cheng, Chao Shen, Sen Chen, Xiaofei Xie · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28116 · cited by 0 works (OpenAlex) · open-access PDF

Russian

Агенты с большим языком (LLM) агенты доступа к данным все чаще выполняют бизнес-операции через инструменты и взаимодействуют напрямую с данными и услугами приложений. Помимо выполнения запросов пользователей, эти агенты должны подчиняться доменным правилам бизнеса, которые ограничивают, когда разрешены конкретные действия и какие данные или ресурсы могут быть затронуты. Нарушение таких правил может привести к ошибкам в бизнес-логике, даже когда задача кажется успешной. Обнаружение этих ошибок затруднено, потому что правила редко доступны в качестве явных, проверяемых базовых спецификаций. Они распространены по системным просьбам и определениям инструментов, в то время как другие остаются подразумеваемыми в области знаний. Существующие методы тестирования агентов обычно улучшают

Machine translation (Meta M2M100); the English original follows.

Abstract

Large language model (LLM) agents increasingly execute business operations through tools and interact directly with application data and services. Beyond completing user requests, these agents must obey domain-specific business rules that constrain when actions are permitted and what data or resources may be affected. Violating such rules can lead to business logic bugs, even when a task appears successful. Detecting these bugs is difficult because the rules are rarely available as explicit, testable specifications. They are distributed across system prompts and tool definitions, while others remain implicit in domain knowledge. Existing agent testing approaches typically evaluate task success, predefined policies, or known attack scenarios. This limits their ability to systematically uncover previously unspecified business-rule violations. We present AgenTest, which mines business rules from system prompts, tool definitions, and domain knowledge, refines them into Given-When-Then specifications, and systematically tests each specification. For each extracted specification, AgenTest instantiates its Given conditions by constructing a satisfying environment state and simulated user, exercises the agent through multi-turn interactions, and checks its Then conditions over messages, tool calls, and state changes using a hybrid oracle. On the airline, retail, and telecom agents of tau2-bench with two underlying models, AgenTest produces 551 refined specifications, and detects violations of 89 and 121 distinct specifications, respectively -- 3.4X and 2.5X as many as all baselines combined. Only AgenTest detects cases where an agent improperly accesses or modifies another user's data. Moreover, about 60% of bug-triggering runs cannot be detected from the final state alone, highlighting the need to check the full interaction against explicit business rules.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Task (project management), Domain (mathematical analysis), Business rule, State (computer science), Business logic

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange