Newest papers

When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems

Shihao Weng, L. Wang, Yang Feng, Mingfei Cheng, Jia Liu, Chao Shen, He Ye, Federica Sarro, Xiaofei Xie · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.24545 · cited by 0 works (OpenAlex) · open-access PDF

Korean

소프트웨어 메커니즘은 오랫동안 코드가 무엇을 해야 하는지에 대한 사양을 지정하고, 코드가 무엇을 보증해야 하는지 테스트하고 검증하는 프레임 워크를 고정했습니다. 에이전트 시스템은이 기초를 변경합니다 : 그들은 다음 도구를 선택하여 행동이 허용되는지 여부를 판단하는 것과 같은 논리의 일부를 대체합니다. 코드가 무엇을 보장해야하는지 테스트하고 검증합니다. 프로그램은 여전히이 논리가 무엇을 보장해야하는지에 의존합니다. 그러나 모델의 행동은 전체 코드에 의해 고정되어 있지 않으며, 실행에 따라 다릅니다. 그러므로 보증은 각 LLM 호출 사이트 주위의 프로그램에서 발생해야합니다. 약 17,428%의 인간 판단에 의해 유효한 다

Machine translation (Meta M2M100); the English original follows.

Abstract

Software quality has long rested on specifications that state what code must do, and on testing and verification that check that the code does it. Agentic systems change this foundation: they replace parts of their logic, such as choosing the next tool, judging whether an action is allowed, or writing code to run, with calls to a large language model (LLM). The program still relies on what this logic must guarantee, but the model's behavior is not fixed by code, varies across runs, and can be steered by its input. The guarantee must therefore come from the program around each LLM call site, as a contract of pre-conditions, prompt constraints, and post-conditions. We present an empirical study of these contracts in 496 LLM call sites of 40 popular open-source agentic systems (13.4 MLOC, four languages). Using a multi-agent pipeline validated by 17,428 human judgments, we trace where every model output flows, derive an expected contract for each call site without access to its code, and compare it with the contract that the code implements. We find that 48% of LLM outputs reach control flow or execution, and 73% of these never reach a person. Programs enforce 78% of their constraints in code, but in the prompt, 67% of the constraints are only requested from the model. Although 82% of call sites are checked before and after the call, 63% of the expected constraints are unmet and only 5.3% of call sites satisfy their whole contract; ten recurring patterns explain 99% of the gaps. 38 mechanisms behind these gaps trigger at runtime, and a small code fix that enforces the expected constraint stops all 38. Our results show that the presence of checks says little about their adequacy, and we derive implications for agent developers, framework maintainers, and researchers.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, TRACE (psycholinguistics), Code (set theory), Empirical research, Action (physics), Control flow

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange