Summary (Overview)

• This paper introduces cache engineering, a new scheme that allows LLM agents to directly control their active key-value (KV) cache without modifying the trajectory text or re-prefilling retained context, addressing the growing cost and accuracy challenges of long-horizon agent tasks.
• Three cache operations are formalized: append (standard KV addition), drop (excludes selected KV entries from future attention), and repos (adjusts position encodings of retained keys via RoPE rotations), enabling agents to shape their active cache programmatically.
• The authors co-design AKV, a serving system with a stateless interface compatible with existing APIs, internally managing stateful cache reuse, eviction, and restoration through an extended radix tree, drop-aware eviction, and segment-batch attention.
• Experiments across three LLMs and four agent benchmarks show that a simple tool-response rolling-drop policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7% compared to conventional threshold-triggered summarization, with up to 1.2× output throughput and 1.5× speedup in P95 latency.
• Recursive self-improvement (RSI) of cache policies further reduces cost by 7.1% over expert design, and exposing cache management as a tool improves accuracy on BCP benchmarks.

Introduction and Theoretical Foundation

Background and Motivation

Large Language Model (LLM) agents handle increasingly complex, long-horizon tasks with trajectories spanning hundreds of rounds of reasoning, tool use, and environment interactions. As trajectories grow, KV cache I/O and storage become the dominant cost—KV cache reads account for 82.4% of total API cost (Figure 13). Additionally, trajectories can exceed the LLM's maximum context length, degrading accuracy.

Prior work relies on context engineering—trimming or summarizing history—but this creates a dilemma:

• Modifying the trajectory prefix invalidates subsequent cached states, triggering costly re-prefilling of retained context
• Aggressive reductions risk losing useful information from intermediate rounds

Key Theoretical Insight

The paper identifies that cache states are history-dependent: identical token sequences can yield different cache states under different cache operation sequences. This makes standard prefix matching insufficient for correct cache reuse, since the token prefix alone no longer determines the cached states.

The authors propose cache engineering as an alternative: instead of modifying the text trajectory, agents directly manipulate the active KV cache, keeping the logical token sequence unchanged while controlling which KV entries participate in future attention.

Methodology

Cache Operations

Three basic cache operations are formalized:

1. Append – Standard cache update:

$$\operatorname{append} (x _ {t + 1}): \qquad \mathcal {C} \leftarrow [ \mathcal {C}; C _ {t + 1} ]$$

2. Drop – Removes selected KV states from attention:

$$\operatorname{drop} (P): \qquad \mathcal {C} \leftarrow \mathcal {C} \setminus \{C _ {i}: i \in P \}$$

3. Repos – Adjusts position encodings of retained keys via RoPE rotation:

$$K \leftarrow R (q) R (p) ^ {- 1} K$$

Cache Operation Sequence

A formal description of cache operations:

$$o _ {1: m} = \left(o _ {1}, \dots , o _ {m}\right), \quad o _ {j} \in \{\text { append } (x), \text { drop } (P), \text { repos } \}$$

This sequence is necessary because cache states are history-dependent—the same retained tokens can have different KV states depending on when drops and appends occurred.

AKV Serving System

AKV follows a stateless interface, stateful serving principle:

1. Extended radix tree: Adds drop nodes (Δ) to track cache operation history alongside token prefixes, enabling correct cache reuse across different operation histories.

2. Drop-aware eviction: Cache entries become evictable only when no ongoing computation or restoration needs them, prioritizing eviction of dropped cache over cache from completed requests.

3. Segment-batch attention: Restores missing cache states in a single prefill pass by splitting queries at drop trigger points, with each segment attending causally to its own tokens and active cache from earlier segments. FFN processes the full sequence as one batch to avoid repeated weight reads.

4. Reposition-aware management: Copies and rotates keys for repos operations, evicts positional versions independently, and extends segment-batch attention to handle position encoding variations.

Empirical Validation / Results

Benchmark Performance

Table 1: Benchmark scores (pass rate, %) and costs (cents) by model and management method

AKV achieves the highest average success rates across all three models and reduces inference cost by 27.7% relative to Standard and 21.9% relative to Text Engineering.

Serving Efficiency

Table 2: Serving performance of GPT-OSS-120B on Top-80 SWE-Verified at concurrency C=14

AKV achieves 1.36× and 1.51× speedups in mean and P95 request latency over Standard, with 1.53× input and 1.18× output throughput improvements.

Ablations and Policy Analysis

Table 3: Drop-target and repos ablations (GPT-OSS-120B, first 100 tasks)

Adding repos raises BCP's pass rate from 55% to 62%, and tool-response drop with repos achieves the highest pass rate on both benchmarks.

Recursive Self-Improvement (RSI)

RSI finds lower-cost cache policies than expert design: policies from iterations 3 and 6 reduce per-task cost by 15.6% and 17.8% relative to Standard, respectively, while maintaining accuracy on held-out tasks.

Theoretical and Practical Implications

Theoretical Contributions

1. Formalization of cache engineering: The paper provides a rigorous mathematical framework for cache operations and their trajectories, establishing that cache states are history-dependent and require both token and operation histories for correct reuse.

2. Decoupling position encodings: The repos operation demonstrates that position encoding indices can be managed independently of logical token indices, enabling agents to work beyond the model's position limit without re-prefilling.

3. Information preservation: Retained KV states can preserve information from dropped history used in their computation, providing a theoretical advantage over context engineering's information loss.

Practical Implications

1. Serving system design: AKV demonstrates that cache engineering can be supported efficiently in production serving systems through:

   • Stateless interfaces compatible with existing APIs
   • Extended radix trees for history-dependent cache reuse
   • Segment-batch attention for single-pass cache restoration

2. Agent performance: Cache engineering improves both accuracy and cost simultaneously, breaking the traditional trade-off between re-prefill cost and information retention.

3. Programmable cache management: Agents can now define, search, and refine cache policies through RSI or by exposing cache management as a tool, enabling adaptive strategies beyond fixed heuristics.

Conclusion

This paper introduces cache engineering as a new paradigm for managing LLM agent context, enabling agents to directly control active KV states while reusing retained caches without recomputation. The co-designed AKV serving system supports this through a stateless interface, extended radix tree, and segment-batch attention, achieving:

• Improved task accuracy (average +1.7 points)
• Reduced inference costs (average -27.7%)
• Higher serving throughput (1.18× output) and lower latency (1.51× P95 speedup)

Future directions include:

• Further exploration of cache engineering strategies
• Training agents under this scheme
• Extending serving support to linear attention models