An Empirical Cost Attribution of Context-Compression Gateways in Multi-Turn Coding Agents

Why Compression Rate Is Not the End-to-End Saving

Authors: Luzhuo Chen, Jiayu Shi, Paritok

Summary (Overview)

• Decomposition of token costs: The paper instruments a production compression gateway (Paritok) to decompose the token bill of real multi-turn coding agent sessions (Claude Code, Codex) into three independent levers: tool-schema filtering, content compression, and history summarization.
• Tool filtering dominates per-turn: Tool-schema filtering removes a fixed block of ~21K–57K tokens per turn (linear in turn count $N$) and is the only lever that is unambiguously and reproducibly positive, with savings ratios of 20:1 to 33:1 over content compression.
• Content compression is quadratic but small per-turn: Content compression saves only ~2% of the cache-priced prefix per turn, but its cumulative saving grows quadratically ($\approx 3{,}350 N^2$ tokens), overtaking tool filtering around turn 6 before the context window caps it.
• The recall trap: Non-destructive gateways allow agents to recall original bytes, but each recall costs exactly one compressed segment—bounded and deterministic, never a multiplicative blowup—eroding content savings only in proportion to recall frequency.
• Benchmark caveat: Single-shot compression quality (86.5% SWE-bench retained at 25.7% compression rate) is orthogonal to multi-turn agent cost and must not be cited as a cost-saving argument.

Introduction and Theoretical Foundation

The dominant cost of an autonomous coding agent is not the model's output—it is the input re-sent on every turn. Modern agents (Claude Code, Cursor, Codex, OpenHands) build each request from three re-sent parts:

1. A block of tool/function JSON schemas
2. An ever-growing message history
3. Large tool-result / file-read blocks

Because agents solve tasks over many turns of grep, read, edit, and test, this prefix is transmitted dozens of times per task. Context compression—shrinking file reads and tool outputs with a small model before they reach the expensive frontier LLM—is a natural response, but the paper demonstrates that a well-trained compressor achieving a quarter-size segment with preserved single-shot quality does not proportionally cut the bill.

The Cost Model

The bill is a sum over turns with cache pricing dominating:

$$\mathrm{Cost} = \sum_{t = 1}^{N} \Big(c_{\mathrm{cr}} P_{t}^{\mathrm{cached}} + c_{\mathrm{in}} P_{t}^{\mathrm{new}} + c_{\mathrm{out}} O_{t}\Big), \tag{1}$$

with (Sonnet) c_{\mathrm{in}} = \3$,$c_{\mathrm{cr}} = $0.30$,$c_{\mathrm{out}} = $15$ per million tokens.

Two consequences drive the entire paper:

• Turn count is the first-order variable: The prefix $P_t$ grows with the conversation and is paid every turn, so $\mathrm{Cost} \approx \sum_t P_t$ is dominated by how many turns the agent takes—an emergent property of agent behavior largely orthogonal to compression. The same task swings from 7 to 18 turns, dwarfing a 2% per-turn effect.
• Not all saved tokens cost the same: A token removed from the re-sent prefix is a cache-read token worth $c_{\mathrm{cr}}$, not $c_{\mathrm{in}}$.

Methodology

Experimental Harness

Paritok is a proxy speaking the Anthropic Messages, OpenAI Chat, and OpenAI Responses protocols. On each request the gateway may:

• (a) Filter the tool-schema block down to semantically relevant tools via a local CPU embedding model, stubbing the rest behind a recall tool
• (b) Compress each file read/tool result with a 4B code-native compression model (a LoRA adapter over Qwen3-4B) to ~26% of its size, tagged as [REF:id]
• (c) Summarize stale history once the window fills

Every compression is non-destructive: the agent can call a recall tool (read_original, gateway_search_tools) to retrieve exact bytes on demand.

Isolation strategy: Each lever can be independently disabled:

• Content compression: token floor min_tokens = 512 ("compress") vs. 49999 ("no-op")
• Tool filtering: toggle while holding content path fixed
• MCP jitter: --strict-mcp-config flag

Two agent families were tested—Claude Code (Claude Sonnet) and Codex (GPT-5)—on real repositories (textual, jinja2, werkzeug, rich, httpx, tox). All dollar figures use provider-reported usage, priced per tier.

Empirical Validation / Results

Lever 1 — Tool-Schema Filtering (Dominant)

Coding agents advertise their whole toolbox in full JSON schema on every request. Claude Code sends ~31 tools (~29K tokens) without MCP servers, and 60–90 tools (44–65K tokens) with MCP attached.

Table 1: Isolated per-session savings: tool filtering vs. content compression (MCP attached)

A subtle "proxy tax": base_url=api.anthropic.com yields ~40K tokens/turn of tools, while base_url=127.0.0.1 (any proxy) yields ~57K/turn—a 17K/turn penalty. Tool filtering cancels this tax, allowing a filtered proxy to beat a direct connection outright ($0.043 vs.$0.057 on a Sonnet edit task).

Caveat: Codex exposes only ~9 tools and lives or dies by shell_command; an embedding filter that intermittently drops it below top-k paralyzes the agent. A whitelist of core execution tools (shell, exec, apply_patch, etc.) is required. With the whitelist, Codex runs commands every turn; without it, filtering is unsafe.

Lever 2 — Content Compression (Marginal Per-Turn Effect)

Table 2: Turn-aligned prefix, content compression only (Sonnet/Claude Code)

The per-turn saving is only ~1.5–2K tokens—exactly the compressed file slice, entirely in the cheap cache-read tier. Whole-task comparisons come out time-varying in sign due to turn-count variance: one block cost $0.638 (18 turns) compressed vs.$0.230 (7 turns) uncompressed on the local backend, and the reverse on the GPU backend.

Scaling Laws: Linear vs. Quadratic

Table 3: Five consecutive read-only turns (A/B isolates content compression)

Content compression is quadratic: The cumulative saving fits $c N^2$ with $c \approx 3{,}350$ (per-$N^2$ ratios lie in $3.35–3.53 \times 10^3$). The mechanism is compounding: each turn reads a file whose compressed form saves tokens that turn, and because the compressed block stays in history and is re-sent on every later turn, the per-turn saving grows roughly linearly (~7K/turn) while the running sum grows quadratically.

Tool filtering is linear: ~21K tokens removed every turn, independent of conversation length; cumulative ≈ $21{,}000 N$.

Crossover and cap: Setting $3{,}350 N^2 = 21{,}000 N$ gives $N \approx 6$: tool filtering dominates only for the first ~6 turns; past that, content compression overtakes it. But the quadratic does not run forever—an unassisted session saturates the ~200K window around turn 12–15, at which point client-side compaction truncates history and accumulation flattens.

The Recall Trap

Each recall wastes a fixed, bounded amount—one compressed segment—since compressing that segment then bought nothing. The recalled original is ephemeral: the gateway resolves it only for the turn that asks and re-stubs it to its [REF] on the next turn, so it never persists to re-inflate the prefix.

Recall economics:

• Changes the summary already supports (e.g., set max_retries=-1): recalls = 0, compression stays marginally positive (~6.5% fewer total input tokens)
• Changes needing untouched bytes (e.g., precise multi-line replacement): stable 2 recalls, each costing one compressed segment—a small, bounded loss
• Deleting the recall tool entirely is far worse: the agent reconstructs exact bytes via grep/cat/re-reads, reaching 46 turns / $1.85, 3.2× the cost of keeping recall

Benchmark Caveat

Table 4: SWE-bench Lite, single-shot compression quality

The benchmark harness makes one API call with no tools, no turns, no re-reads, no exact-match Edit. It measures single-shot understanding under compression—but a real agent's cost gap is in multi-turn behavior and the tool block, both omitted entirely by the benchmark.

Theoretical and Practical Implications

Recommendations for Practitioners

1. Filter the tool schema first: Largest per-turn lever, the only reliably-positive one; keep selection frozen per conversation for cache stability; never stub core execution tools
2. Treat content compression as a session-length bet: Pays off super-linearly on long sessions where the agent seldom recalls (auditing/Q&A over large codebases); on short sessions or recall-heavy turns it costs at most one compressed segment—bounded, never multiplicative
3. Keep compression non-destructive but budget for recall: Subtract one compressed segment's saving on any turn where the agent recalls
4. Report end-to-end dollars at the correct cache tier: Never cite a single-shot benchmark as a multi-turn cost result
5. Next frontier: A symbol map / directed retrieval (be the agent's grep) rather than indiscriminate body deletion, which agents route around by slice-reading and which harms editing

Methodology for Reproducible Measurement

• Report distributions, not single runs: The same task swings 2× in turn count and cost; run ≥3 trials per arm
• Use provider-reported usage, priced per tier: Cache-read is ~0.1× the fresh-input price; a saving's tier matters as much as its size
• Control the tool block: MCP servers load asynchronously (40 vs. 90 tools across two concurrent runs—a 40K first-turn swing)
• Deduplicate streamed usage: Aggregate by message id; server-side recall must be counted via the gateway's own logs
• Isolate one lever at a time via the min_tokens floor, filter toggle, and strict-MCP

Conclusion

"Compress the file reads" is the intuitive way to cut a coding agent's token bill, and it is mostly the wrong lever. The key findings:

1. Turn count, not compression rate, sets the bill
2. Tool-schema filtering is the dominant, reproducibly-positive lever (linear savings)
3. Content compression is a ~2% per-turn effect that accumulates quadratically over long sessions, overtaking tool filtering near turn 6 before the context window caps it
4. Each recall costs a bounded, fixed amount—one compressed segment—eroding content savings only in proportion to the minority recall count
5. Single-shot compression benchmarks are orthogonal to multi-turn cost

The practical takeaway is a hierarchy: filter tools always, compress content when the session is long and the agent rarely recalls, and measure in end-to-end dollars at the right cache tier. Decomposition, not a headline compression rate, is what tells you where the tokens actually go.

Reproducibility: The gateway, the 4B compression model, and the evaluation harness are open source (Apache 2.0) at https://github.com/Paritok-official/paritok-4b-v1; model weights are on the Hugging Face Hub.