# Cache Engineering for Agents

> Cache engineering lets LLM agents manipulate active KV cache directly, cutting inference cost by 27.7% while raising benchmark accuracy by 1.7 points.

- **Source:** [arXiv](https://arxiv.org/abs/2610.28577)
- **Published:** 2026-10-23
- **Permalink:** https://picx.dev/p/cX5DVA
- **Whiteboard:** https://picx.dev/p/cX5DVA/image

## Summary

## Summary (Overview)

- This paper introduces **cache engineering**, a new scheme that allows LLM agents to directly control their active key-value (KV) cache without modifying the trajectory text or re-prefilling retained context, addressing the growing cost and accuracy challenges of long-horizon agent tasks.
- Three cache operations are formalized: **append** (standard KV addition), **drop** (excludes selected KV entries from future attention), and **repos** (adjusts position encodings of retained keys via RoPE rotations), enabling agents to shape their active cache programmatically.
- The authors co-design **AKV**, a serving system with a stateless interface compatible with existing APIs, internally managing stateful cache reuse, eviction, and restoration through an extended radix tree, drop-aware eviction, and segment-batch attention.
- Experiments across three LLMs and four agent benchmarks show that a simple tool-response rolling-drop policy improves average benchmark scores by **1.7 points** and reduces average cost by **27.7%** compared to conventional threshold-triggered summarization, with up to **1.2× output throughput** and **1.5× speedup in P95 latency**.
- Recursive self-improvement (RSI) of cache policies further reduces cost by **7.1%** over expert design, and exposing cache management as a tool improves accuracy on BCP benchmarks.

## Introduction and Theoretical Foundation

### Background and Motivation

Large Language Model (LLM) agents handle increasingly complex, long-horizon tasks with trajectories spanning hundreds of rounds of reasoning, tool use, and environment interactions. As trajectories grow, KV cache I/O and storage become the dominant cost—KV cache reads account for **82.4% of total API cost** (Figure 13). Additionally, trajectories can exceed the LLM's maximum context length, degrading accuracy.

Prior work relies on **context engineering**—trimming or summarizing history—but this creates a dilemma:
- Modifying the trajectory prefix invalidates subsequent cached states, triggering costly re-prefilling of retained context
- Aggressive reductions risk losing useful information from intermediate rounds

### Key Theoretical Insight

The paper identifies that cache states are **history-dependent**: identical token sequences can yield different cache states under different cache operation sequences. This makes standard prefix matching insufficient for correct cache reuse, since the token prefix alone no longer determines the cached states.

The authors propose **cache engineering** as an alternative: instead of modifying the text trajectory, agents directly manipulate the active KV cache, keeping the logical token sequence unchanged while controlling which KV entries participate in future attention.

## Methodology

### Cache Operations

Three basic cache operations are formalized:

**1. Append** – Standard cache update:
$$\operatorname{append} (x _ {t + 1}): \qquad \mathcal {C} \leftarrow [ \mathcal {C}; C _ {t + 1} ]$$

**2. Drop** – Removes selected KV states from attention:
$$\operatorname{drop} (P): \qquad \mathcal {C} \leftarrow \mathcal {C} \setminus \{C _ {i}: i \in P \}$$

**3. Repos** – Adjusts position encodings of retained keys via RoPE rotation:
$$K \leftarrow R (q) R (p) ^ {- 1} K$$

### Cache Operation Sequence

A formal description of cache operations:
$$o _ {1: m} = \left(o _ {1}, \dots , o _ {m}\right), \quad o _ {j} \in \{\text { append } (x), \text { drop } (P), \text { repos } \}$$

This sequence is necessary because cache states are history-dependent—the same retained tokens can have different KV states depending on when drops and appends occurred.

### AKV Serving System

AKV follows a **stateless interface, stateful serving** principle:

1. **Extended radix tree**: Adds drop nodes (Δ) to track cache operation history alongside token prefixes, enabling correct cache reuse across different operation histories.

2. **Drop-aware eviction**: Cache entries become evictable only when no ongoing computation or restoration needs them, prioritizing eviction of dropped cache over cache from completed requests.

3. **Segment-batch attention**: Restores missing cache states in a single prefill pass by splitting queries at drop trigger points, with each segment attending causally to its own tokens and active cache from earlier segments. FFN processes the full sequence as one batch to avoid repeated weight reads.

4. **Reposition-aware management**: Copies and rotates keys for repos operations, evicts positional versions independently, and extends segment-batch attention to handle position encoding variations.

## Empirical Validation / Results

### Benchmark Performance

**Table 1: Benchmark scores (pass rate, %) and costs (cents) by model and management method**

| Method | BCP Score | BCP Cost | GAIA-Text Score | GAIA-Text Cost | SWE-Verified Score | SWE-Verified Cost | xbench-DS Score | xbench-DS Cost | Avg Score | Avg Cost |
|--------|-----------|----------|-----------------|----------------|-------------------|------------------|-----------------|-----------------|-----------|----------|
| **MiniMax-M2.7** | | | | | | | | | | |
| Standard | 58.9 | 5.2 | 35.0 | 2.4 | 75.2 | 7.0 | 40.5 | 5.5 | 52.4 | 5.0 |
| Text Engineering | 62.3 | 5.3 | 36.9 | 2.7 | 72.0 | 9.4 | 40.0 | 5.2 | 52.8 | 5.7 |
| **AKV** | **58.9** | **4.8** | **40.8** | **2.6** | **74.4** | **5.5** | **40.5** | **4.7** | **53.7** | **4.4** |
| **GPT-OSS-120B** | | | | | | | | | | |
| Standard | 57.5 | 13.1 | 68.0 | 5.4 | 40.0 | 20.0 | 38.5 | 6.2 | 51.0 | 11.2 |
| Text Engineering | 52.7 | 12.6 | 65.1 | 6.8 | 36.8 | 15.8 | 38.5 | 5.4 | 48.3 | 10.2 |
| **AKV** | **61.9** | **11.4** | **69.9** | **4.1** | **43.2** | **9.7** | **37.5** | **3.8** | **53.1** | **7.3** |
| **MiroThinker-1.7-Mini** | | | | | | | | | | |
| Standard | 60.0 | 25.2 | 68.0 | 9.2 | 7.8 | 33.6 | 40.5 | 13.5 | 44.1 | 20.4 |
| Text Engineering | 45.7 | 17.6 | 66.0 | 11.0 | 4.2 | 28.4 | 32.5 | 15.3 | 37.1 | 18.1 |
| **AKV** | **63.5** | **15.3** | **68.9** | **9.5** | **7.4** | **26.3** | **43.5** | **8.1** | **45.8** | **14.8** |

AKV achieves the highest average success rates across all three models and reduces inference cost by **27.7%** relative to Standard and **21.9%** relative to Text Engineering.

### Serving Efficiency

**Table 2: Serving performance of GPT-OSS-120B on Top-80 SWE-Verified at concurrency C=14**

| Metric | Standard | Text Engineering | AKV |
|--------|----------|------------------|-----|
| Mean latency (s) | 30.16 | 19.59 | **9.09** |
| P95 latency (s) | 108.86 | 60.67 | **28.49** |
| Input throughput (tok/s) | 12,767 | 19,992 | **51,837** |
| Output throughput (tok/s) | 146 | 267 | **369** |

AKV achieves **1.36×** and **1.51×** speedups in mean and P95 request latency over Standard, with **1.53×** input and **1.18×** output throughput improvements.

### Ablations and Policy Analysis

**Table 3: Drop-target and repos ablations (GPT-OSS-120B, first 100 tasks)**

| Strategy | Repos | BCP Score | BCP Cost | SWE Score | SWE Cost |
|----------|-------|-----------|----------|-----------|----------|
| Standard | ✘ | 56.0 | 13.5 | 41.0 | 19.1 |
| Tool drop | ✘ | 55.0 | 7.1 | 41.0 | 9.4 |
| Reasoning drop | ✓ | 56.5 | 8.2 | 36.5 | 14.2 |
| Both drop | ✓ | 56.5 | 8.7 | 34.5 | 7.8 |
| **Tool drop** | **✓** | **62.0** | **10.9** | **41.0** | **9.5** |

Adding repos raises BCP's pass rate from 55% to 62%, and tool-response drop with repos achieves the highest pass rate on both benchmarks.

### Recursive Self-Improvement (RSI)

RSI finds lower-cost cache policies than expert design: policies from iterations 3 and 6 reduce per-task cost by **15.6%** and **17.8%** relative to Standard, respectively, while maintaining accuracy on held-out tasks.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Formalization of cache engineering**: The paper provides a rigorous mathematical framework for cache operations and their trajectories, establishing that cache states are history-dependent and require both token and operation histories for correct reuse.

2. **Decoupling position encodings**: The repos operation demonstrates that position encoding indices can be managed independently of logical token indices, enabling agents to work beyond the model's position limit without re-prefilling.

3. **Information preservation**: Retained KV states can preserve information from dropped history used in their computation, providing a theoretical advantage over context engineering's information loss.

### Practical Implications

1. **Serving system design**: AKV demonstrates that cache engineering can be supported efficiently in production serving systems through:
   - Stateless interfaces compatible with existing APIs
   - Extended radix trees for history-dependent cache reuse
   - Segment-batch attention for single-pass cache restoration

2. **Agent performance**: Cache engineering improves both accuracy and cost simultaneously, breaking the traditional trade-off between re-prefill cost and information retention.

3. **Programmable cache management**: Agents can now define, search, and refine cache policies through RSI or by exposing cache management as a tool, enabling adaptive strategies beyond fixed heuristics.

## Conclusion

This paper introduces **cache engineering** as a new paradigm for managing LLM agent context, enabling agents to directly control active KV states while reusing retained caches without recomputation. The co-designed **AKV serving system** supports this through a stateless interface, extended radix tree, and segment-batch attention, achieving:
- Improved task accuracy (average +1.7 points)
- Reduced inference costs (average -27.7%)
- Higher serving throughput (1.18× output) and lower latency (1.51× P95 speedup)

**Future directions** include:
- Further exploration of cache engineering strategies
- Training agents under this scheme
- Extending serving support to linear attention models

---

_Markdown view of https://picx.dev/p/cX5DVA, served by PicX — AI-generated visual whiteboard summaries of research papers._