Newest papers

AgenTest: Mining Business Rules to Test LLM Agents for Business Logic Bugs

Lili Quan, L. Wang, Yuelin Wang, Mingfei Cheng, Chao Shen, Sen Chen, Xiaofei Xie · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28116 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

大規模なエージェント言語モデル(LLM)のデータアクセスのエージェントは、ツールを通じてビジネス操作をどんどん実行し、アプリケーションデータとサービスと直接相互作用する。ユーザー要求を完了する以外に、これらのエージェントは、特定のアクションが許可され、特定のデータやリソースが影響を受ける場合に制限するドメイン特有のビジネスルールに従わなければなりません。既存のエージェントテストアプローチは、通常、タスク成功、事前定義のポリシー、または既知の攻撃シナリオを評価する可能性があります。これらのバグを検出することは、これらのルールが以前明示されていない、テスト可能な基本的な仕様として稀に利用可能であるため困難です。それらは、

Machine translation (Meta M2M100); the English original follows.

Abstract

Large language model (LLM) agents increasingly execute business operations through tools and interact directly with application data and services. Beyond completing user requests, these agents must obey domain-specific business rules that constrain when actions are permitted and what data or resources may be affected. Violating such rules can lead to business logic bugs, even when a task appears successful. Detecting these bugs is difficult because the rules are rarely available as explicit, testable specifications. They are distributed across system prompts and tool definitions, while others remain implicit in domain knowledge. Existing agent testing approaches typically evaluate task success, predefined policies, or known attack scenarios. This limits their ability to systematically uncover previously unspecified business-rule violations. We present AgenTest, which mines business rules from system prompts, tool definitions, and domain knowledge, refines them into Given-When-Then specifications, and systematically tests each specification. For each extracted specification, AgenTest instantiates its Given conditions by constructing a satisfying environment state and simulated user, exercises the agent through multi-turn interactions, and checks its Then conditions over messages, tool calls, and state changes using a hybrid oracle. On the airline, retail, and telecom agents of tau2-bench with two underlying models, AgenTest produces 551 refined specifications, and detects violations of 89 and 121 distinct specifications, respectively -- 3.4X and 2.5X as many as all baselines combined. Only AgenTest detects cases where an agent improperly accesses or modifies another user's data. Moreover, about 60% of bug-triggering runs cannot be detected from the final state alone, highlighting the need to check the full interaction against explicit business rules.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Task (project management), Domain (mathematical analysis), Business rule, State (computer science), Business logic

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange