Newest papers

Cache Engineering for Agents

Tianyu Fu, Ruoxi Wang, Zihan Min, Zhongqi Ren, Zhihao Li, Tengxuan Liu, Shengxu Mu, Hanling Zhang, Yu Wang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28577 · cited by 0 works (OpenAlex) · open-access PDF

Korean

Large Language Model Threshold (LLM) 에이전트는 점점 더 복잡한 작업을 처리하여 관찰 및 행동의 긴 경로를 탐색 할 수 있습니다. 이러한 경로가 성장함에 따라 키 값 (KV) 캐시 I/O 및 저장 상태가 지배적 인 비용이 될 수 있습니다. 경로는 LLM의 최대 컨텍스트 길이를 초과 할 수 있으며 활성 캐시 성장에 대한 제어가 너무 적게 유지되어 재예비 비용을 제한 할 때도 고통을받을 수 있습니다. 사전 작업은 컨텍스트 엔지니어링을 사용하여 캐시 크기를 줄이거나 역사를 요약함으로써 캐시 엔지니어링을 사용합니다. 그러나, 경로 사전을 수정하면 기존의 경로를 수정하지 않고 기존의 컨텍스트를 재예비하지

Machine translation (Meta M2M100); the English original follows.

Abstract

Large Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can also exceed the LLM's maximum context length, requiring control over active cache growth. Prior work uses context engineering to reduce cache size by trimming or summarizing history. However, modifying the trajectory prefix invalidates subsequent cached states and triggers costly re-prefilling of retained context. Task accuracy can also suffer when too little context is retained to limit re-prefill costs. We propose cache engineering as a new scheme. It allows agents to directly control their active KV cache without modifying the trajectory or re-prefilling retained context. Specifically, agents can proactively use drop to exclude selected cache from future attention and repos to adjust retained keys' position encodings. We formalize these cache operations and identify history dependence as a serving challenge: identical token sequences can yield different cache states under different cache operation sequences, making prefix matching insufficient for correct cache reuse. Thus, we co-design the AKV serving engine. It provides a stateless interface compatible with existing serving APIs, while internally managing stateful cache reuse, eviction, and restoration. Experiments show that a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7%, compared with conventional threshold-triggered summarization. With AKV, it achieves up to 1.2x the output throughput and a 1.5x speedup in P95 request latency. Recursive self-improvement of cache policies further reduces cost by 7.1% over expert design. Our code is at https://github.com/thu-nics/AKV.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Cache, Cache algorithms, Computer science, Cache invalidation, Cache coloring, Smart Cache

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange