TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents

Summary (Overview)

• Key finding: Tool observations in coding-agent trajectories are surprisingly small (median 118–286 tokens), while the large build logs that motivate compression techniques occur only near the 99th percentile.
• Cache-aware pricing: 98.6% of input tokens are served from prefix cache, meaning flat input pricing overestimates loop cost by ~7.1× on average under near-ideal caching conditions.
• Break-even condition: The paper derives a closed-form condition for when model delegation is profitable, showing that the relevant comparison is against the strong model's cached-read price, not its list price—making the constraint ~10× tighter than flat-price calculations suggest.
• Cost decomposition: Turn count explains 84% of variance in log trajectory cost, while mean observation size explains only 4%—yet all existing optimization techniques target the observation axis.
• Tool release: The authors release TurnCost, an open-source tool for re-scoring trajectory corpora under alternative caching, pricing, compression, and routing policies.

---

Introduction and Theoretical Foundation

Background

AI coding agents (e.g., SWE-agent, OpenHands) operate in iterative ReAct loops: propose an edit → apply it → compile → run tests → inspect tool output (the observation) → decide next step. The design of this iterative control loop—including context management, model selection, compaction, and termination—is termed loop engineering by the authors.

The Anomaly

The paper is motivated by an empirical puzzle from AgentDiet (a context-reduction system):

• A learned reducer achieved 0.652 of baseline cost at 60% pass rate
• A trivial Delete baseline (removing observations with no summarization) achieved 0.655 of baseline cost at 58% pass rate
• The learned reducer invoked an additional language model and was selected from five candidates, yet gained only 0.3% cost saving over simple deletion

Why does a sophisticated learned reducer provide so little cost advantage over simply deleting the observation?

Research Questions

• RQ1: How large are tool observations in real agentic coding trajectories?
• RQ2: By how much does flat input pricing overstate loop cost under cache-aware pricing?
• RQ3: Does delegating observation handling to a cheaper model outperform deterministic truncation?
• RQ4: Does turn count or observation size contribute more to trajectory cost variation?

---

Methodology

Corpus

• Source: nvidia/Open-SWE-Traces (207,489 published trajectories, CC BY 4.0)
• Design: 2 scaffolds (OpenHands, SWE-agent) × 2 models (Qwen3.5-122B-A10B, Minimax-M2.5) = 4 cells
• Sample: 150 trajectories per cell, 600 total, deterministically sampled (first 150 rows in file order)

Reconstruction

TurnCost reconstructs per-turn context from message sequences, tokenizes with o200k_base, and simulates prefix caching exactly using the append-only structure:

$$h_t = C_t - r_{t-1}$$

where $h_t$ is cache-served tokens at turn $t$, $C_t$ is total context, and $r_{t-1}$ is the observation appended after the previous turn.

Conditions Evaluated

Pricing

Prices normalized with p_S^{in} = \1$per million tokens. Defaults:$p_S^{cache} = 0.1 \cdot p_S^{in}$,$p_W^{in} = 0.05 \cdot p_S^{in}$, compression ratio$\sigma = 0.05$.

---

Empirical Validation / Results

RQ1: Observations Are Small

Table 1: Corpus shape and observation-size distribution

Median observation size: 118–286 tokens across cells. Large logs are rare (99th percentile: 4,274–6,208 tokens).

RQ2: Flat Pricing Overstates Cost by 7.1×

• 98.1–99.0% of input tokens served from cache
• Flat pricing overstates cost by 6.5× to 7.8× (mean 7.14×)
• Inflation grows with loop length: 5.33× (<40 turns) → 7.99× (>130 turns)

RQ3: Truncation and Delegation Are Cost-Indistinguishable

• B3 (truncation) matches or beats R1 (delegation) across all 20 cell-threshold combinations
• Maximum gap: 0.17 percentage points at default prices
• At aggressive thresholds: both save 11–19%
• At conservative thresholds: savings decline toward zero and turn negative in 2 of 4 cells

RQ4: Turn Count Drives Cost

• Turn count alone: $R^2 = 0.844$ with exponent 1.23 (superlinear, confirming re-read effect)
• Mean observation size alone: $R^2 = 0.040$ with exponent −0.31 (slightly negative!)
• Together: $R^2 = 0.945$

The most expensive trajectories have the most turns AND the smallest observations.

Key Distributional Finding

Table 2: Turn types pooled across all cells (56,164 turns)

The largest observations come from edit operations (editor tools echo updated file context, ~787 tokens mean), not build logs. The motivating example in the literature is misplaced twice over.

Oracle Ceiling

• Observations contribute 66–74% of transcript growth
• Oracle compression (all observations to 5% size, free, no quality loss): saves 47.8–60.3% maximum
• This is the theoretical ceiling for observation-focused methods

---

Theoretical and Practical Implications

The Cache-Aware Cost Model

The trajectory cost under prefix caching:

$$\text{Cost} = \sum_{t=1}^{T} \left[ p_S^{\text{cache}} h_t + p_S^{\text{in}} (C_t - h_t) + p_S^{\text{out}} o_t \right]$$

The Break-Even Condition (Eq. 3)

For out-of-band delegation at turn $j$, with compression ratio $\sigma$ and prompt overhead fraction $e$:

$$T - j > \frac{p_W^{\text{in}} (1 + e) + p_W^{\text{out}} \sigma}{(1 - \sigma) \, p_S^{\text{cache}}} \xrightarrow[e \to 0]{\sigma \to 0} \frac{p_W^{\text{in}}}{p_S^{\text{cache}}}$$

Key insights from Eq. (3):

1. Observation size is absent—it governs how much delegation saves, not whether it's worth doing
2. The strong model's list price is absent—only its cached-read price matters (typically 10% of list)
3. Poor compression penalizes superlinearly (appears in both numerator and denominator)

Worked Example

With p_S^{in} = \3.00$,$p_S^{cache} = $0.30$,$p_W^{in} = $0.15$:

• Correct break-even horizon: $H = 0.15/0.30 = 0.5$ (one remaining turn suffices)
• Using list price instead: $0.15/3.00 = 0.05$ (understates threshold by 10×)

Three Recommendations for Loop Engineering

1. Price the cache before optimizing anything—98–99% of input tokens are cache hits
2. Use Eq. (3) as a deployment rule—delegate only when remaining turns exceed the break-even horizon
3. Always compare against a truncation baseline—a method that doesn't beat deletion is buying quality, not cost

Why the Opportunity Is Small

The median observation is small by design: SWE-agent's agent–computer interface (ACI) returns concise, structured output rather than raw shell dumps. The compression opportunity was largely removed by interface design before context reduction or routing was applied.

---

Conclusion

Main Takeaways

1. Context reduction saturation has a measurable cause: tool observations are small (118–286 tokens median)
2. Flat pricing creates illusory savings: routing results measured against flat-priced baselines cannot be read directly
3. The positive finding: turn count (84% of cost variance) is the unexamined opportunity—every existing technique targets observation size (4% of variance)

Key Caveats

• The turn-count association is partly mechanical (cost is a sum over turns), but the exponent >1 and negative observation-size correlation are not definitional
• Compression quality was not measured—the parity result concerns cost only
• The corpus excludes trajectories that hit iteration limits, potentially truncating the heavy tail of observation sizes
• Results use open-weight models; frontier proprietary agents may differ

Future Directions

The open question: Why does an agent need a hundred turns to fix one bug, and can redundant turns be removed without costing task success? Removing a turn eliminates not only that turn's cost but every subsequent re-read of its content—making turn reduction potentially far more impactful than observation compression.

"The most useful thing a loop engineer can know before choosing where to spend effort is that the compression opportunity had largely been removed by interface design before either method was applied."