# Prompt-Induced Waste in Coding Agents: Reasoning, Effort, Harness Design, and End-to-End Cost

> Prompt, effort, and harness interact to determine coding-agent cost per successful task, not token counts; harness design can shift intervention effects by 4–15×.

- **Source:** [arXiv](https://arxiv.org/abs/2608.01347)
- **Published:** 2026-09-05
- **Permalink:** https://picx.dev/p/SUP9G2
- **Whiteboard:** https://picx.dev/p/SUP9G2/image

## Summary

# Summary of "Prompt-Induced Waste in Coding Agents: Reasoning, Effort, Harness Design, and End-to-End Cost"

## Summary (Overview)

- **Core finding**: Coding-agent efficiency cannot be reduced to token count or model price; end-to-end cost per successful task depends jointly on prompt semantics, inference effort, harness policy, model, and task difficulty, which are **interacting** factors rather than independent controls.
- **Prompt effects isolated**: Instructions requesting multiple approaches, deep thinking, or high certainty increased reasoning tokens by **1.3–7.4×** across models with **no measurable success benefit** (e.g., 96.5% baseline vs. 95.2% branching success; discordant blocks favored baseline 26:13).
- **Waste mechanisms differ**: Discarded branches are primarily **token-borne** (\[1.9×\] median cost, unchanged tool calls); redundant verification is **tool-borne** (up to **18.25×** median cost at severity level 3+, with 2.5× tool calls, no success gradient).
- **Harness effects dominate**: Between-harness cost differences reach **5–30×** per success (Claude Code vs. PI.DEV); harness design changes the value of efficiency interventions (same effort control cuts cost 19% under Claude Code vs. 75% under DeepSeek Harness).
- **Proposed remedy**: Introduce **cost per successful task (CPS)** as central metric and design an adaptive, state-conditioned controller that minimizes expected cost subject to a success non-inferiority floor.

## Introduction and Theoretical Foundation

Coding-agent cost is typically analyzed through billable inputs/units (input tokens, output tokens, context size, cache traffic, model price). The authors argue these matter for billing but don't reveal *work performed*: the same task can produce different reasoning steps, model turns, tool calls, repository reads, edits, tests, and retries. The central question: not just token price, but what determines the **trajectory** that produces tokens and tool actions.

Three controllable variables are examined:

- **Prompt semantics**: instructions can change search, branching, verification, and stopping behavior even with fixed task/model/harness/effort
- **Inference effort**: more reasoning can help on hard tasks but can also add compute without improvement
- **Agent harness**: system prompt, tool schemas, reasoning controls, context management, and tool/subagent behavior, which can change the effect of both prompt and effort

All factors are treated as **interacting** rather than independent, with task (\[T\]), model (\[M\]), prompt (\[P\]), effort (\[E\]), harness (\[H\]), environment (\[X\]), and pricing (\[\Pi\]) jointly determining the behavioral trajectory.

The trajectory yields token, cache, latency, and cost metrics; the terminal metric is cost per successful task. This framework yields a concrete training problem: an adaptive controller with small action space conditioned on agent state.

## Methodology

- **Models**: Six open-weight reasoning models (≥500B-class) plus Claude Sonnet 5, run under two production harnesses: PI.DEV directly and Claude Code via a pinned protocol-translation gateway (native Claude Code arm for the closed model).
- **Tasks**: 24 deterministic coding tasks with frozen fixtures, visible tests, and hidden deterministic tests (unavailable to the agent).
- **Prompt variants**: 18 frozen prompt variants with 9 primary variants preserving objective, acceptance criteria, and test command verbatim; Table 4 reports exact manipulated clauses.
- **Effort experiments**: Separate SWE-bench campaign (150 paired model-task-effort cells).
- **Harness extension**: Post-registration change to DeepSeek Harness (dsh) 0.1.0rc7 with Sonnet 5 and 5 tasks, analyzing 75 run pairs.

**Trace annotation**: 2,801 condition-blind trace annotations using a frozen rubric; the judge was not shown the prompt variant and provided evidence quotes for each mechanism count.

**Statistical rigor**: Preregistered hypotheses, frozen 8-task holdout for screening, paired comparisons within task-model-harness blocks, task-clustered bootstrap confidence intervals, and Fisher/McNemar/Holm procedures. Prompt effects were tested under matched conditions.

## Empirical Validation / Results

### Prompt Effects

- **Multiple approaches** instruction increased reasoning tokens **2.4–7.4×** across all six open models at comparable success, reproduced on the frozen held-out set. No success benefit: 96.5% (baseline) vs. 95.2% (branching); discordant blocks favored baseline 26:13.
- **Deep thinking**: increased reasoning text **1.6–2.2×** across three phases with zero new functional units observed.
- **Certainty-oriented**: increased redundant re-verification (on all six models), +1.75 calls post-success, and the most extreme loop re-ran an already-green suite six times.
- **Bounded-efficiency** template: had no measurable cost penalty and reduced reasoning for one model; on an ambiguous specification, it reduced evidence seeking—producing \(3/5\) hidden-test failures vs. \(0/5\) baseline.
- **Verbatim restatement** had approximately unit cost, showing prompt content (not length) drives effects.

**Table 1: Cost by judge-annotated redundant-verification level** (relative to level 0 median)

| Level | n | Median | Mean | Winsorized | 95% CI | Tool calls |
|---|---|---|---|---|---|---|
| 0 | 1,585 | 1.00× | 1.00× | 1.00× | [0.96, 1.03] | 6 |
| 1 | 835 | 1.48× | 1.77× | 1.70× | [1.44, 1.58] | 8 |
| 2 | 168 | 2.36× | 4.34× | 4.28× | [2.01, 2.67] | 9 |
| 3+ | 213 | 18.25× | 17.74× | 18.38× | [15.13, 21.06] | 15 |

No success gradient across levels; level-3+ lower quartile is 10× the level-0 median. The gradient survives winsorization at 95th percentile (18.4× on winsorized means).

**Table 2: Integrated mechanism table** (median paired deltas per model)

| Condition | Observed reasoning | Tool-layer effect | Cost carrier | Success | Evidence |
|---|---|---|---|---|---|
| multiple_approaches | +3 discarded branches; +1 implemented | weak/secondary (p≈.06); edits +0 | token-borne | none | strong semantic; directional tool |
| deep_thinking | 2.2× text, zero new functional units | none (p=.26) | token-borne | none | strong |
| max_certainty | +1 redundant re-verification (6/6) | +1.75 post-success calls; repeat tests | tool-borne + induced tokens | none | strong |
| misleading_hints | +1 unsupported assumption; 4.2× pre-edit deliberation | directional (+1.75 calls, +0.5 failed) | reasoning-borne | none/harm (ρ=-0.19) | mixed |
| bounded_efficiency | no loss of diagnosis/validation | null (p=1.0) | — | preserved | strong null |

### Harness Effects

- **Claude Code** transmitted a **12–15× larger fixed prefix** than PI.DEV and used **2–7× more turns**, producing **5–30× higher cost per success** at comparable success; test executions dominated (including 50–60%+ of calls vs. 22% under PI.DEV).
- Prompt effects are **not invariant across harnesses** (goal-only prompts reduced reasoning under Claude Code but reduced verifiability under PI.DEV).

### DeepSeek Harness Extension Results

**Table 3: DeepSeek Harness extension (Sonnet 5; n=15 paired runs per arm)**

| Arm | Mean billed cost/run | Change vs. baseline | Success |
|---|---|---|---|
| baseline | $0.2753 | — | 15/15 |
| policy | $0.2114 | -23.2% | 15/15 |
| rewrite | $0.2109 | -23.4% | 15/15 |
| effort | $0.0696 | -74.7% | 15/15 |
| full | $0.0490 | -82.2% | 15/15 |

The **effort arm accounts for ~91%** of the baseline-to-full reduction — the **harness × effort interaction** is the dominant factor. In contrast, the same controller's effort arm under Claude Code reduced cost by only ~19%.

**Key numbers**: 4,644 valid runs; 2,801 annotated traces; 878 branching runs (225 paired blocks); 150 paired model-task-effort cells in SWE-bench campaign; 75 valid runs in dsh extension.

## Theoretical and Practical Implications

The paper explicitly refutes naive scaling assumptions: **efficiency ≠ fewer tokens or fewer tool calls**. Agent efficiency must consider the full **trajectory**, treating token and cache counts as sequences of a behavioral process, not the final optimization target.

For **quantitative research**: the paper's proposed CPS metric (cost per successful task) and the interaction-expansion model provide a formal evaluation framework for benchmarks. The hard experiments show that prompt–effort–harness interactions are not separable; benchmarking should preserve these variables to avoid misleading conclusions.

For **practitioners**: the clearest practical recommendations are:

1. **Prompt design**: Only request multiple approaches when comparison is genuinely needed; require explicit verification after final edits; avoid unverified architectural hints; use bounded-efficiency language but **always** include repository inspection when the specification is ambiguous.
2. **Harness choice is crucial**: static-prefix size, tool-schema surface, reasoning defaults, context, and verification policy can dominate model choice; gateway/protocol translators can invalidate cross-platform comparisons.
3. **Efficiency intervention**: Conditional on the harness, effort control can yield ~75% cost reduction under DeepSeek Harness but ~19% transfer to Claude Code.

The concrete **controller architecture** (Section 8) provides a practical roadmap: define state-based safety rules (e.g., block repeated identical tests after a green test), fit calibrated cost and success models, select actions under a non-inferiority floor, and validate with A/B tests rather than pure reward maximization.
## Conclusion

Main takeaways:

- **Prompt, effort, and harness are interacting factors**; efficiency is best measured as **cost per successful task** (non-CPS), not token counts or cache traffic.
- **The trajectory is the unit of analysis**: token volume, cache traffic, latency, and billing follow from the sequence of model requests, tool calls, context updates, retries, and stopping decisions — an intervention that reduces one pollution can still increase total cost.
- **Harness design can shift the value of any single intervention** (the dsh experiment yields 4–15× effect differences for the same controller logic depending on the default).
- **Final extension** proposed: test hard SWE-bench tasks under dsh with both Sonnet 5 and Opus 5 to cross the harness–effort–task space, then train a conservative adaptive controller.

**Future direction**: the adaptive controller is currently a design implication, not an evaluated result, but provides a natural next step for field deployment.

---

_Markdown view of https://picx.dev/p/SUP9G2, served by PicX — AI-generated visual whiteboard summaries of research papers._