Newest papers

Rewards for Budget-Aware Multi-Turn Agents That Adapt to Arbitrary Test-Time Constraints

Tom A. Lamb, Adam Goliński, Eleonora Gualdoni, Sinead A. Williamson, Michael Kirchhof · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28409 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Di seluruh model perilaku agen tunggal, sering diberikan anggaran inferensi dan diminta untuk memaksimalkan kesuksesan di dalamnya. Agen multi-turn harus mendistribusikan anggaran ini melalui banyak interaksi lingkungan. Harness eksternal dapat memperkuat anggaran, tetapi tetap tidak jelas apakah agen itu sendiri dapat mempelajari kapan dan berapa banyak untuk menghabiskan. Kami bertanya apakah agen dapat mempelajari alokasi ini di dalam anggaran yang ditentukan secara dinamis dan hadiah mana yang paling baik mengajarkan perilaku ini. Kami menyesuaikan hukuman panjang yang ada ke dalam anggaran yang mencakup berbagai jenis sumber daya, memperkenalkan formulasi hadiah baru, dan membandingkan 11 tujuan dalam rantai pelatihan bersama. Kami melatih model Qwen3.5 di 4B dan 9B di AppWorld dengan penghargaan token, panggilan API gratis, dan langkah-langkah lingkungan, dan

Machine translation (Meta M2M100); the English original follows.

Abstract

Large language model agents are often given an inference budget and asked to maximise success within it. A multi-turn agent has to allocate this budget across many environment interactions. An external harness can enforce the budget, but it remains unclear whether the agent itself can learn when and how much to spend. We ask whether agents can learn this allocation across dynamically specified budgets and which rewards best teach this behaviour. We adapt existing length penalties to budget constraints spanning multiple resource types, introduce new reward formulations, and compare eleven objectives in a shared training pipeline. We train Qwen3.5 models at 4B and 9B on AppWorld with budgets over tokens, API calls, and environment steps, and on the BabyAI gridworld with turn budgets. We train a separate policy for each environment and resource type, with each policy trained across budget values. Training with rewards that penalise overspending substantially improves within-budget success and natural compliance over untrained models and over training on task success alone, with agents learning to adapt their resource use to stated budgets and task difficulty. These gains extend to unseen budgets and, zero-shot, to a shifted task distribution and a new benchmark. Reward objectives trade off success within budget and natural compliance. A sigmoid penalty for overspending most often achieves the highest within-budget success at an individual budget, while a fixed negative reward for overspending generally yields the strongest natural compliance. Combining penalties for overspend frequency and magnitude gives the smallest average gap to the best method at each budget. Overall, we show that a single agent can learn to allocate resources across turns and adapt its behaviour to user-specified budgets using budget feedback without external budget-allocation controllers.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Task (project management), Ask price, Computer science, Budget constraint, Resource (disambiguation), Resource allocation

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange