# The Physical Cost of Context: Quality-Constrained Resource Accounting for Coding Agents

> Quality-gated evaluation of coding agents reveals that editing reusable history can increase uncached prefill by 608% despite cutting prompt tokens, so physical work must be measured, not just token counts.

- **Source:** [arXiv](https://arxiv.org/abs/2610.34763)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/qh04Pb
- **Whiteboard:** https://picx.dev/p/qh04Pb/image

## Summary

## Summary (Overview)

- This paper introduces a **quality-gated physical-accounting protocol** for evaluating coding agents, measuring not just prompt tokens but also uncached prefill, KV cache residency, wall time, and GPU energy.
- The authors decompose executed prefill into **logical prompt tokens minus reused prefix tokens** (Equation 1), showing that token savings can paradoxically increase physical work when prefix reuse is lost.
- On 450 localization cases from SWE-rebench V2, both **EAGER** (preloading ten retrieved blocks) and **COMPACT** (anchored history compaction) pass a five-point non-inferiority quality gate.
- EAGER reduces model calls by **35.7%** and GPU-energy proxy by **33.2%** versus on-demand search (GREP), while COMPACT adds no further savings.
- **MASK-5** (keeping only five observations) cuts prompt volume by 33.3% but **increases uncached prefill by 608.5%** and energy by 125.6%, failing the quality gate—demonstrating that editing reusable history can multiply physical work.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Repository localization maps an issue report to files, classes, and functions. Modern coding agents use iterative retrieval, structure-aware search, and graph navigation within ReAct-style loops. These agents repeatedly search, read, and reassess retrieved source code, creating long trajectories of tool outputs and intermediate hypotheses.

The core problem: **token counts do not determine physical cost**. Serving runtimes like vLLM and SGLang maintain KV caches; when a new request begins with a cached token prefix, the runtime reuses those entries and prefills only the remaining suffix. A shorter prompt can therefore cost *more* if it rewrites cached history, forcing recomputation of all subsequent tokens.

### Theoretical Foundation

The paper formalizes the relationship between logical and executed work. Let $n_p$ be the number of model requests under policy $p$, $\ell_{p,t}$ the logical prompt tokens in request $t$, and $h_{p,t}$ the tokens served from a reusable prefix. Then:

$$T_{\text{prompt}}(p) = \sum_{t=1}^{n_p} \ell_{p,t}, \quad T_{\text{uncached}}(p) = T_{\text{prompt}}(p) - \underbrace{\sum_{t=1}^{n_p} h_{p,t}}_{T_{\text{hit}}(p)}. \tag{1}$$

For a treatment–control contrast:

$$\Delta T_{\text{uncached}} = \Delta T_{\text{prompt}} - \Delta T_{\text{hit}}.$$

**Key insight**: Prompt savings increase executed prefill when a policy loses more reusable tokens than it removes. Because $n_p$ also depends on the policy, fewer calls can reduce total work even when each call is no cheaper.

### Three Research Challenges

- **C1 (Trajectory attribution)**: Evidence supply, delivery time, and history retention can each change what the agent does next.
- **C2 (Logical-to-physical translation)**: A policy changes both request count and prefix reuse; logical tokens can order policies differently from uncached prefill, KV occupancy, latency, or energy.
- **C3 (Quality-constrained comparison)**: Resource comparisons must first bound task-quality loss.

---

## Methodology

### Experimental Setup

The study uses an **NVIDIA DGX Spark** with an NVIDIA GB10 SoC and 128 GB coherent unified memory. The localization stack uses Qwen3.6-35B-A3B-FP8 (35B total, 3B active) with vLLM, temperature zero, and 65,536-token context limit.

### Policy Algebra

Let $\sigma \in \{\text{loc}, \text{patch}\}$ identify the agent scaffold, $S_\sigma$ its system prompt, and $H_{\sigma,t} = (a_1, o_1, \ldots, a_t, o_t)$ its canonical action–observation history. An experimental arm is $p = (\sigma, e, r)$ where $e \in \{\emptyset, R_{10}\}$ is the evidence supplied before the first turn and $r$ controls visible serialization. The visible state and serialized model input are:

$$V^{(e)}_{\sigma,t} = q \oplus e \oplus H_{\sigma,t}, \quad P^{(e,r)}_{\sigma,t} = S_\sigma \oplus r\left(V^{(e)}_{\sigma,t}\right), \tag{2}$$

where $\oplus$ denotes serialization.

Three deterministic context-serialization operators:

$$\mathcal{I}\left(V^{(e)}_{\sigma,t}\right) = V^{(e)}_{\sigma,t},$$

$$\mathcal{A}_\sigma\left(V^{(R_{10})}_{\sigma,t}\right) = \begin{cases} V^{(R_{10})}_{\sigma,t}, & t < \tau, \\ D_{\tau,\sigma} \oplus H_{\sigma,\tau+1:t}, & t \geq \tau, \end{cases}$$

$$\mathcal{L}_5\left(V^{(R_{10})}_{\text{patch},t}\right) = q \oplus R_{10} \oplus H_{\text{patch},t}[o_i \leftarrow \mu(|o_i|) \text{ for } i \leq t-5]. \tag{3}$$

Here $\tau$ is the first successful source read, $D_{\tau,\sigma}$ is a scaffold-specific anchor containing the task query, the successful read, and a bounded pre-rewrite assessment. $\mathcal{L}_5$ follows SWE-agent's LastNObservations: it keeps actions and tool IDs but replaces older outputs with line-count markers $\mu(|o_i|)$.

The arm definitions:

$$G = (\text{loc}, \emptyset, \mathcal{I}), \quad N = (\text{patch}, \emptyset, \mathcal{I}),$$
$$E_\sigma = (\sigma, R_{10}, \mathcal{I}), \quad C_\sigma = (\sigma, R_{10}, \mathcal{A}_\sigma), \quad M_5 = (\text{patch}, R_{10}, \mathcal{L}_5). \tag{4}$$

### Quality-Constrained Resource Objective

The resource vector collected for each policy:

$$\mathcal{R}(p) = (T_{\text{prompt}}, T_{\text{output}}, N_{\text{req}}, T_{\text{uncached}}, L_{\text{TTFT}}, L_{\text{wall}}, K_R, K_{\text{max}}, U_{\text{max}}, E_{\text{gpu}}) \tag{5}$$

where $K_R$ is the KV time-area (integrated KV bytes over wall time, in GiB·s), $K_{\text{max}}$ is peak KV occupancy, $U_{\text{max}}$ is peak unified-memory pressure, and $E_{\text{gpu}}$ is sampled GPU energy.

A policy is eligible only if:

$$\text{LCB}_{.95}[Q(p) - Q(c(p))] > -\epsilon, \quad \epsilon = 0.05, \tag{6}$$

where LCB$_{.95}$ is the lower endpoint of a paired 95% interval obtained by resampling whole repositories, and $Q(p)$ is the task-quality endpoint.

### Datasets

| Cohort | Source | Size | Purpose |
|--------|--------|------|---------|
| Resource cohort | codenib-synthesis test split | 100 tasks × 3 reps = 900 cells | Repeated device measurement |
| Quality cohort | SWE-rebench V2 | 450 Python issues | Quality gate (powered) |
| Cache mechanism study | Same 100 tasks | 600 cells | Prefix-cache intervention |
| External patch study | SWE-bench Verified | 100 instances | Cross-scaffold validation |

---

## Empirical Validation / Results

### RQ1: Quality-Eligible Policies and Shorter Trajectories

Across 450 localization cases, the frozen retriever reaches .431 Recall@5 and .548 Recall@10. Agent Recall@5 is .470, .478, and .494 for GREP, EAGER, and COMPACT respectively.

- **EAGER**: paired difference +0.8 points; lower 95% endpoint −2.6 (clears the −5-point gate)
- **COMPACT**: paired difference +2.4 points; lower endpoint −0.9 (meets the criterion)

**Table 1: Localization-only profile for $P_{\text{loc}} = \{G, E, C\}$**

| Metric | GREP | EAGER | COMPACT |
|--------|------|-------|---------|
| **Task quality** | | | |
| Recall@5 | .470 | .478 | .494 |
| Full@5 | .307 | .318 | .329 |
| Format success | .904 | .976 | .987 |
| **Resource per attempted task** | | | |
| Prompt ktok | 136.8 | 90.5 (−33.9%) | 98.2 (−28.2%) |
| Successful server calls | 13.02 | 8.38 (−35.7%) | 9.49 (−27.1%) |
| Wall seconds | 68.32 | 48.22 (−29.4%) | 52.20 (−23.6%) |
| KV GiB·s | 25.88 | 16.67 (−35.6%) | 18.21 (−29.6%) |
| GPU-proxy kJ | 2.007 | 1.340 (−33.2%) | 1.466 (−27.0%) |
| **Resource per full-recall** | | | |
| Prompt ktok/full | 204.2 | 133.7 (−34.5%) | 151.9 (−25.6%) |
| GPU-proxy kJ/full | 3.00 | 1.98 (−34.0%) | 2.27 (−24.3%) |

**Key finding**: COMPACT uses 8.5% more prompt tokens, 13.3% more calls, 8.2% more wall time, and 9.4% more energy than EAGER. The anchor drops remaining candidates, and the agent may search again for evidence that EAGER still shows.

### RQ2: Physical Savings Follow Fewer Calls

For all four primary cost metrics, intervals for both policies exclude zero. Mean TTFT varies by less than 2%, and latency per successful call rises from 5.218 to 5.721 seconds (EAGER) and 5.468 (COMPACT), so **task-time savings come from fewer calls, not faster calls**.

**Cache intervention results** (600 cells, toggling vLLM prefix-cache flag):

- EAGER's logical prompt changes by **+8.3%** (95% CI [−5.7, +24.7]) but uncached prefill by **−67.7%** ([−71.0, −63.8])
- The planned $C_{\text{loc}} - E_{\text{loc}}$ cache interaction is −7.1k uncached tokens per task; 95% interval [−17.3k, +2.3k] (includes zero)

### RQ3: Token Savings Reverse under External Retention

**Table 2: Patch-generation-only profile for $P_{\text{patch}}$: mini-SWE-agent on 100 SWE-bench Verified instances**

| Metric | MINI | EAGER | COMPACT | MASK-5 |
|--------|------|-------|---------|--------|
| **Task outcome (of 100)** | | | | |
| Resolved | 30 | 29 | 30 | 26 |
| Submitted / call-capped | 45/55 | 44/56 | 45/55 | 40/60 |
| **Generation resource per attempt** | | | | |
| Prompt ktok | 220.7 | 225.6 (+2.2%) | 210.8 (−6.5%) | 150.5 (−33.3%) |
| Model calls | 26.13 | 25.66 (−1.8%) | 25.84 (+0.7%) | 26.33 (+2.6%) |
| Uncached ktok | 9.8 | 10.0 (+1.4%) | 12.2 (+22.0%) | 70.8 (+608.5%) |
| KV frac.·s | 2.94 | 3.25 (+10.6%) | 2.69 (−17.2%) | 3.55 (+9.4%) |
| Wall seconds | 120.2 | 122.7 (+2.0%) | 116.6 (−5.0%) | 167.9 (+36.9%) |
| GPU-proxy kJ | 1.833 | 1.956 (+6.7%) | 1.917 (−2.0%) | 4.412 (+125.6%) |

**Critical finding**: MASK-5 removes 75.1k logical prompt tokens per attempt but loses 135.9k device-hit tokens, adding 60.8k uncached tokens. Cached prefixes serve about 96% of EAGER's prompt but only 53% of MASK-5's. The paired quality difference is −3.0 points with lower 95% endpoint −8.6, **crossing the −5-point gate** and thus failing non-inferiority.

---

## Theoretical and Practical Implications

### Break-Even Analysis for History Deletion

The paper derives a first-order estimate: if a runtime served a fraction $\rho$ of the hits observed here in both arms, masking would change uncached prefill by:

$$-75.1\text{k} + 135.9\text{k} \cdot \rho \text{ tokens per attempt}$$

This turns positive above $\rho \approx 0.55$. At $\rho = 0$ (no prefix cache), masking would save prefill; batching or cache pressure that evicts prefixes lowers $\rho$ and with it the masking penalty.

### Three Key Recommendations

1. **Separate evidence delivery from retention (C1)**: Eager uses the fewest resources among eligible localization arms; the anchor remains eligible but adds no saving because shorter requests don't offset additional calls.

2. **Measure prefix reuse (C2)**: Reports should expose request count, prefix reuse, and uncached prefill alongside prompt volume.

3. **Constrain quality before comparing resources (C3)**: Apply Equation (6) before interpreting resource differences. Token count alone would favor the policy that the quality gate excludes.

---

## Conclusion

A context policy changes both what an agent does and how much of its prompt the runtime can reuse. The paper demonstrates that:

- **Early evidence (EAGER) saved work by avoiding calls**, reducing model calls by 35.7% and GPU energy by 33.2%.
- **Repeated observation masking (MASK-5)** turned a 33.3% prompt saving into a 608.5% rise in uncached prefill and a 125.6% energy increase, while failing the quality gate.
- **Resource comparisons should measure executed work** (uncached prefill, KV time-area, energy) and admit only policies that pass a quality gate.

### Limitations and Future Directions

- Results cover two stacks on one DGX Spark serving one request at a time; batching, cache pressure, and cloud GPUs remain untested.
- The GPU-energy proxy omits CPU, host-memory, and whole-system energy.
- Only four fixed policies were tested; window sweeps, LLM summarization, learned pruning, and external memory remain unmeasured, though Equation (1) applies to each.
- The 32-call patch cap measures fixed-budget performance, not unrestricted SWE-bench ability.

---

_Markdown view of https://picx.dev/p/qh04Pb, served by PicX — AI-generated visual whiteboard summaries of research papers._