Newest papers

When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems

Shihao Weng, L. Wang, Yang Feng, Mingfei Cheng, Jia Liu, Chao Shen, He Ye, Federica Sarro, Xiaofei Xie · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.24545 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

虽然 gforce 控制机制长期以来一直依赖于规格,规定代码必须执行,以及测试和验证代码必须执行的规格。代理系统改变了这个基础:它们取代了它们的逻辑的一部分,例如选择下一个工具,判断是否允许采取行动,或写代码运行每个LLM呼叫站点周围的程序,以呼叫大语言模型(LLM)进行。该程序仍然依赖于这个逻辑必须保证的内容,但该模型的行为并非由代码固定,而且可以通过其输入来控制,因此,保证必须来自每个LLM呼叫站点周围的程序,因为每个LLM呼叫站点的预条件,提示约束和后条件的合同。我们在496个LLM呼叫站点中对这些合同的实证运行进行了实证研究,它们的40个

Machine translation (Meta M2M100); the English original follows.

Abstract

Software quality has long rested on specifications that state what code must do, and on testing and verification that check that the code does it. Agentic systems change this foundation: they replace parts of their logic, such as choosing the next tool, judging whether an action is allowed, or writing code to run, with calls to a large language model (LLM). The program still relies on what this logic must guarantee, but the model's behavior is not fixed by code, varies across runs, and can be steered by its input. The guarantee must therefore come from the program around each LLM call site, as a contract of pre-conditions, prompt constraints, and post-conditions. We present an empirical study of these contracts in 496 LLM call sites of 40 popular open-source agentic systems (13.4 MLOC, four languages). Using a multi-agent pipeline validated by 17,428 human judgments, we trace where every model output flows, derive an expected contract for each call site without access to its code, and compare it with the contract that the code implements. We find that 48% of LLM outputs reach control flow or execution, and 73% of these never reach a person. Programs enforce 78% of their constraints in code, but in the prompt, 67% of the constraints are only requested from the model. Although 82% of call sites are checked before and after the call, 63% of the expected constraints are unmet and only 5.3% of call sites satisfy their whole contract; ten recurring patterns explain 99% of the gaps. 38 mechanisms behind these gaps trigger at runtime, and a small code fix that enforces the expected constraint stops all 38. Our results show that the presence of checks says little about their adequacy, and we derive implications for agent developers, framework maintainers, and researchers.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, TRACE (psycholinguistics), Code (set theory), Empirical research, Action (physics), Control flow

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange