# Context Has a Price: Adding and Removing Context for LLM Coding Agents under Prefix Caching

> Adding or removing context in LLM coding agents rarely cuts cost: retrieved code raises bills 33-42% despite less searching, and masking savings flip to losses depending on cache pricing.

- **Source:** [arXiv](https://arxiv.org/abs/2611.22238)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/ffDUci
- **Whiteboard:** https://picx.dev/p/ffDUci/image

## Summary

## Summary (Overview)

- This paper investigates whether adding or removing context in LLM coding agents actually reduces the total cost of solving a task, accounting for prefix caching effects and subsequent agent behavior.
- Through more than 20,000 runs across SWE-bench Verified, SWE-bench, and Loc-Bench with seven open models, the authors find that retrieved code reduces search commands by 18–34% but increases cost by 33–42% even with cheap cached tokens, without measurable resolution gains.
- Removing old observations (masking) can either save money (19–25% without cache discount) or increase cost dramatically (38–116% with a tenth-price cache) depending on the price of cached tokens.
- The paper introduces a price-vector accounting framework that separates uncached input, cached input, and output tokens, and identifies a critical bias: concurrent experiments sharing cached prefixes can distort cost comparisons.
- A closed-form break-even rule (Equation 2) for context removal is derived and validated, but no tested method is consistently cheaper than doing nothing across the current API price range.

## Introduction and Theoretical Foundation

Coding agents solve repository-level tasks by searching for code, editing files, and running checks. Each action adds information to the conversation history, which the model repeatedly reads. Context management presents a fundamental question: should we give the agent more information to guide its work, or remove old information to reduce cost?

The key theoretical insight is that **token counts alone do not determine the bill**. Serving systems reuse cached prompt prefixes, and cached input is often priced below fresh input. The paper defines cost as:

$$\text{cost}(c, p) = U + cC + pO$$

where:
- $U$ = uncached input tokens summed over all requests
- $C$ = cached input tokens
- $O$ = output tokens
- $c$ = price of a cached input token relative to an uncached input token
- $p$ = price of an output token relative to an uncached input token

Two examples motivate the framework:
1. **A token added to the prompt is paid again at every later turn** — adding context has compounding costs.
2. **Changing an earlier part of the prompt invalidates the cache from that point on** — the rest of the prompt is paid at the uncached price once more.

The paper examines two interventions:
- **Adding context**: Retrieved code (candidate edit locations, related definitions) meant to make agents search less.
- **Removing context**: Masking or evicting old observations to shorten prompts.

## Methodology

### Pricing an Agent Run

For a run with requests $t = 1, \ldots, T$, let $P_t$ be prompt tokens, $H_t \le P_t$ those served from prefix cache, and $O_t$ output tokens. Then:

$$U = \sum_t (P_t - H_t), \quad C = \sum_t H_t, \quad O = \sum_t O_t$$

**Own-cache accounting**: For isolated-run comparison, cache reuse is attributed only to prefixes computed by the run's own earlier requests:

$$H_t = \min(P_t, \text{LCP}_t)$$

where $\text{LCP}_t$ is the longest common token prefix of request $t$ and request $t-1$ of the same run.

**Break-even condition for removal**: Removing $B$ tokens before request $t$ saves cached price $cB$ on each of the $R$ remaining requests, but the $S$ tokens after the change point lose their cache hit once:

$$R \cdot cB > (1-c)S$$

### Experimental Setup

- **Scaffolds**: A minimal bash agent (one command per turn, max 50 turns or 64K tokens) and mini-SWE-agent 2.4.6, plus SWE-agent 1.1.0 for validation.
- **Sandboxes**: Dependency-free sandbox (separate Linux namespaces, no network) and executable sandbox (read-only Python environment for task tests).
- **Benchmarks**: 80 SWE-bench test tasks, SWE-bench Verified, and 224 Loc-Bench tasks.
- **Models**: Qwen3.6-27B (main), Qwen3-Coder-30B-A3B, Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B, Gemma-4-26B-A4B, GLM-4.7-Flash.
- **Prices**: $c \in \{0.1, 1\}$ (90% cache discount and none), $p = 4$.
- **Protocol**: Preregistered hypotheses, bootstrap confidence intervals (2,000 resamples), Holm-corrected one-sided tests.

## Empirical Validation / Results

### Adding Context: Less Search, Higher Cost

The evidence package (top 3 functions in full + 7 locations + related code, ~14K tokens) was tested:

| Comparison | Read+search change | Cost at c=0.1 | Cost at c=1 | Quality change |
|---|---|---|---|---|
| Evidence vs. none (Verified, 80 tasks) | −32% | +36% | +25% | +3.8 resolved (official) |
| Evidence vs. none, executable sandbox | −9% | +21% | – | −0.3 resolved |
| Retrieval appended to localization | – | ~0% | ~0% | +14 to +20 function Acc@10 |

Key findings:
- **Exploration falls**: 18–34% reduction in read/search commands on every benchmark and Qwen model. Random code of the same length leaves exploration unchanged (+0.68), confirming relevance matters, not length.
- **Cost rises**: Runs with the package cost 33–42% more at $c=0.1$ and 25–33% more at $c=1$. Keeping it to the end costs +80% to +127%.
- **No resolution gain established**: The package changes resolved rate by −1.9 points on SWE-bench test and +3.8 on Verified (not statistically significant). The preregistered confirmation on 217 fresh tasks shows equivalence within ±5 points (+0.0).
- **Retrieval helps as a complement**: Appending retrieval ranking to the agent's localization raises function Acc@10 from 0.386 to 0.538 (+15.2 points above agent alone).

### Removing Context: The Sign Depends on the Price

**Preregistered test** on 201 fresh Verified tasks with rolling masking (keep latest 10 observations), batched masking, and break-even masking:

| Method | Qwen3.6-27B at c=0.1 | Qwen3.6-27B at c=1 | Qwen3-Coder at c=0.1 | Qwen3-Coder at c=1 |
|---|---|---|---|---|
| Rolling masking | +116% | −19% | +87% | −25% |
| Batched masking | +5% | −22% | +12% | −4% |
| Break-even masking | +3% | −26% | +2% | −21% |

**The sign flips**: Rolling masking raises cost at $c=0.1$ by 116% (Qwen3.6-27B) and 87% (Qwen3-Coder), but lowers cost at $c=1$ by 19% and 25%. The crossover lies near $c=0.5$.

**Break-even masking** costs 52% and 46% less than rolling masking at $c=0.1$ and stays close to no masking (+2.8%, +1.8%), but fails the preregistered non-inferiority margin of 5%.

**The flip holds across scaffolds and models**:
- SWE-agent: +101% at $c=0.1$, −29% at $c=1$
- GLM-4.7-Flash: +38% at $c=0.1$, −36% at $c=1$, crossover at 0.21
- Gemma-4: +56% at $c=0.1$, saving at $c=1$ not significant

**Cost decomposition via replay**: With actions held fixed, most of rolling masking's premium is direct (cache-breaking) in mini-SWE-agent (+90 of +116%), and about half in SWE-agent (+52 of +101%). For batched and break-even masking, the premium falls in the behavioral term — agents react to missing information by running longer.

### Representative Methods under a Price Vector

Six methods were compared (3,618 runs):

| Method | Qwen3.6-27B c=0.1 | Qwen3.6-27B c=1 | Qwen3-Coder c=0.1 | Qwen3-Coder c=1 | Qwen3.8-27B c=0.1 | Qwen3.8-27B c=1 |
|---|---|---|---|---|---|---|
| Rolling masking | +116% | −19% | +87% | −25% | +76% | −24% |
| Batched masking | +5% | −22% | +12% | −4% | −14% | −33% |
| Break-even | +3% | −26% | +2% | −21% | +7% | −21% |
| Summarization | +22% | −14% | +28% | +3% | −19% | −53% |
| Reflection | +36% | +4% | +31% | +7% | +29% | +2% |

**Rankings change with price**: Kendall's τ between cost rankings at $c=0.1$ and $c=1$ is +0.20, +0.07, and +0.47 for the three models. No method is within 7% of no management at $c=0.1$ AND at least 21% cheaper at $c=1$ for all three models (break-even masking comes closest).

### Concurrent Experiments Share Caches

In ablations whose conditions share prompt prefixes and run side by side, server-reported cache hits include prefixes warmed by sibling conditions. This changed the measured cost premium of the evidence package on Verified from +8% to +36% once removed.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Price-vector accounting**: Reporting the three token streams (uncached, cached, output) separately with their prices exposes cache sharing between concurrent conditions as a source of bias.

2. **Break-even rule**: Equation (2) provides a closed-form condition for when context removal pays:
$$RcB > (1-c)S$$

3. **Fixed-action replay**: A method for splitting a method's cost into the price of breaking the cache versus the agent's behavioral reaction.

### Practical Implications

- **Token savings ≠ cost savings**: Methods that reduce token counts can raise cost depending on cache prices. Evaluations must report uncached input, cached input, and output separately with their prices.

- **Context management changes how agents work**: Removing observations may cause agents to compensate by taking more steps. In SWE-agent, batched and break-even masking would save 14–22% with actions held fixed, but the agent's response eliminates these savings.

- **The appropriate choice depends on task and price**: None of the fixed configurations tested was consistently cheaper than retaining context across the evaluated API price range. Break-even masking is the only method within 7% of no management at $c=0.1$ and at least 21% cheaper at $c=1$ for all three models.

- **Cache sharing bias**: Cost comparisons should use own-cache hits, report the uncached price alongside, and avoid running conditions that share prefixes side by side.

## Conclusion

The paper demonstrates that context-management methods for LLM coding agents must be evaluated using both token prices and the agent's subsequent behavior. The same method can save money at one cache price and increase cost at another — reporting total tokens alone hides this difference.

Key takeaways:
1. **Retrieved code reduces search but increases cost** without measurable resolution gains; it helps as a complement (appended to the agent's localization) rather than a replacement for agent search.
2. **Removing context has price-dependent outcomes**: rolling masking saves 19–25% without a cache discount but costs 38–116% more with a tenth-price cache.
3. **Shorter prompts do not guarantee cheaper runs** because agents may respond by taking more steps.
4. **Concurrent experiments can share cached prefixes** and distort cost comparisons.

**Future directions** include:
- Interaction between stopping criteria and context removal (both affect the expensive tail of a run).
- Evaluation of frontier API agents and longer reasoning budgets.
- More comprehensive patch definitions (the paper notes staged changes were missed in 1–4% of runs).
- The break-even rule as an accounting baseline, not a universally beneficial policy — it must be combined with quality constraints.

---

_Markdown view of https://picx.dev/p/ffDUci, served by PicX — AI-generated visual whiteboard summaries of research papers._
