Summary of "Prompt-Induced Waste in Coding Agents: Reasoning, Effort, Harness Design, and End-to-End Cost"

Summary (Overview)

Core finding: Coding-agent efficiency cannot be reduced to token count or model price; end-to-end cost per successful task depends jointly on prompt semantics, inference effort, harness policy, model, and task difficulty, which are interacting factors rather than independent controls.
Prompt effects isolated: Instructions requesting multiple approaches, deep thinking, or high certainty increased reasoning tokens by 1.3–7.4× across models with no measurable success benefit (e.g., 96.5% baseline vs. 95.2% branching success; discordant blocks favored baseline 26:13).
Waste mechanisms differ: Discarded branches are primarily token-borne ([1.9×] median cost, unchanged tool calls); redundant verification is tool-borne (up to 18.25× median cost at severity level 3+, with 2.5× tool calls, no success gradient).
Harness effects dominate: Between-harness cost differences reach 5–30× per success (Claude Code vs. PI.DEV); harness design changes the value of efficiency interventions (same effort control cuts cost 19% under Claude Code vs. 75% under DeepSeek Harness).
Proposed remedy: Introduce cost per successful task (CPS) as central metric and design an adaptive, state-conditioned controller that minimizes expected cost subject to a success non-inferiority floor.

Introduction and Theoretical Foundation

Coding-agent cost is typically analyzed through billable inputs/units (input tokens, output tokens, context size, cache traffic, model price). The authors argue these matter for billing but don't reveal work performed: the same task can produce different reasoning steps, model turns, tool calls, repository reads, edits, tests, and retries. The central question: not just token price, but what determines the trajectory that produces tokens and tool actions.

Three controllable variables are examined:

Prompt semantics: instructions can change search, branching, verification, and stopping behavior even with fixed task/model/harness/effort
Inference effort: more reasoning can help on hard tasks but can also add compute without improvement
Agent harness: system prompt, tool schemas, reasoning controls, context management, and tool/subagent behavior, which can change the effect of both prompt and effort

All factors are treated as interacting rather than independent, with task ([T]), model ([M]), prompt ([P]), effort ([E]), harness ([H]), environment ([X]), and pricing ([\Pi]) jointly determining the behavioral trajectory.

The trajectory yields token, cache, latency, and cost metrics; the terminal metric is cost per successful task. This framework yields a concrete training problem: an adaptive controller with small action space conditioned on agent state.

Methodology

Models: Six open-weight reasoning models (≥500B-class) plus Claude Sonnet 5, run under two production harnesses: PI.DEV directly and Claude Code via a pinned protocol-translation gateway (native Claude Code arm for the closed model).
Tasks: 24 deterministic coding tasks with frozen fixtures, visible tests, and hidden deterministic tests (unavailable to the agent).
Prompt variants: 18 frozen prompt variants with 9 primary variants preserving objective, acceptance criteria, and test command verbatim; Table 4 reports exact manipulated clauses.
Effort experiments: Separate SWE-bench campaign (150 paired model-task-effort cells).
Harness extension: Post-registration change to DeepSeek Harness (dsh) 0.1.0rc7 with Sonnet 5 and 5 tasks, analyzing 75 run pairs.

Trace annotation: 2,801 condition-blind trace annotations using a frozen rubric; the judge was not shown the prompt variant and provided evidence quotes for each mechanism count.

Statistical rigor: Preregistered hypotheses, frozen 8-task holdout for screening, paired comparisons within task-model-harness blocks, task-clustered bootstrap confidence intervals, and Fisher/McNemar/Holm procedures. Prompt effects were tested under matched conditions.

Empirical Validation / Results

Prompt Effects

Multiple approaches instruction increased reasoning tokens 2.4–7.4× across all six open models at comparable success, reproduced on the frozen held-out set. No success benefit: 96.5% (baseline) vs. 95.2% (branching); discordant blocks favored baseline 26:13.
Deep thinking: increased reasoning text 1.6–2.2× across three phases with zero new functional units observed.
Certainty-oriented: increased redundant re-verification (on all six models), +1.75 calls post-success, and the most extreme loop re-ran an already-green suite six times.
Bounded-efficiency template: had no measurable cost penalty and reduced reasoning for one model; on an ambiguous specification, it reduced evidence seeking—producing (3/5) hidden-test failures vs. (0/5) baseline.
Verbatim restatement had approximately unit cost, showing prompt content (not length) drives effects.

Table 1: Cost by judge-annotated redundant-verification level (relative to level 0 median)

Level	n	Median	Mean	Winsorized	95% CI	Tool calls
0	1,585	1.00×	1.00×	1.00×	[0.96, 1.03]	6
1	835	1.48×	1.77×	1.70×	[1.44, 1.58]	8
2	168	2.36×	4.34×	4.28×	[2.01, 2.67]	9
3+	213	18.25×	17.74×	18.38×	[15.13, 21.06]	15

No success gradient across levels; level-3+ lower quartile is 10× the level-0 median. The gradient survives winsorization at 95th percentile (18.4× on winsorized means).

Table 2: Integrated mechanism table (median paired deltas per model)

Condition	Observed reasoning	Tool-layer effect	Cost carrier	Success	Evidence
multiple_approaches	+3 discarded branches; +1 implemented	weak/secondary (p≈.06); edits +0	token-borne	none	strong semantic; directional tool
deep_thinking	2.2× text, zero new functional units	none (p=.26)	token-borne	none	strong
max_certainty	+1 redundant re-verification (6/6)	+1.75 post-success calls; repeat tests	tool-borne + induced tokens	none	strong
misleading_hints	+1 unsupported assumption; 4.2× pre-edit deliberation	directional (+1.75 calls, +0.5 failed)	reasoning-borne	none/harm (ρ=-0.19)	mixed
bounded_efficiency	no loss of diagnosis/validation	null (p=1.0)	—	preserved	strong null

Harness Effects

Claude Code transmitted a 12–15× larger fixed prefix than PI.DEV and used 2–7× more turns, producing 5–30× higher cost per success at comparable success; test executions dominated (including 50–60%+ of calls vs. 22% under PI.DEV).
Prompt effects are not invariant across harnesses (goal-only prompts reduced reasoning under Claude Code but reduced verifiability under PI.DEV).

DeepSeek Harness Extension Results

Table 3: DeepSeek Harness extension (Sonnet 5; n=15 paired runs per arm)

Arm	Mean billed cost/run	Change vs. baseline	Success
baseline	$0.2753	—	15/15
policy	$0.2114	-23.2%	15/15
rewrite	$0.2109	-23.4%	15/15
effort	$0.0696	-74.7%	15/15
full	$0.0490	-82.2%	15/15

The effort arm accounts for ~91% of the baseline-to-full reduction — the harness × effort interaction is the dominant factor. In contrast, the same controller's effort arm under Claude Code reduced cost by only ~19%.

Key numbers: 4,644 valid runs; 2,801 annotated traces; 878 branching runs (225 paired blocks); 150 paired model-task-effort cells in SWE-bench campaign; 75 valid runs in dsh extension.

Theoretical and Practical Implications

The paper explicitly refutes naive scaling assumptions: efficiency ≠ fewer tokens or fewer tool calls. Agent efficiency must consider the full trajectory, treating token and cache counts as sequences of a behavioral process, not the final optimization target.

For quantitative research: the paper's proposed CPS metric (cost per successful task) and the interaction-expansion model provide a formal evaluation framework for benchmarks. The hard experiments show that prompt–effort–harness interactions are not separable; benchmarking should preserve these variables to avoid misleading conclusions.

For practitioners: the clearest practical recommendations are:

Prompt design: Only request multiple approaches when comparison is genuinely needed; require explicit verification after final edits; avoid unverified architectural hints; use bounded-efficiency language but always include repository inspection when the specification is ambiguous.
Harness choice is crucial: static-prefix size, tool-schema surface, reasoning defaults, context, and verification policy can dominate model choice; gateway/protocol translators can invalidate cross-platform comparisons.
Efficiency intervention: Conditional on the harness, effort control can yield ~75% cost reduction under DeepSeek Harness but ~19% transfer to Claude Code.

The concrete controller architecture (Section 8) provides a practical roadmap: define state-based safety rules (e.g., block repeated identical tests after a green test), fit calibrated cost and success models, select actions under a non-inferiority floor, and validate with A/B tests rather than pure reward maximization.

Conclusion

Main takeaways:

Prompt, effort, and harness are interacting factors; efficiency is best measured as cost per successful task (non-CPS), not token counts or cache traffic.
The trajectory is the unit of analysis: token volume, cache traffic, latency, and billing follow from the sequence of model requests, tool calls, context updates, retries, and stopping decisions — an intervention that reduces one pollution can still increase total cost.
Harness design can shift the value of any single intervention (the dsh experiment yields 4–15× effect differences for the same controller logic depending on the default).
Final extension proposed: test hard SWE-bench tasks under dsh with both Sonnet 5 and Opus 5 to cross the harness–effort–task space, then train a conservative adaptive controller.

Future direction: the adaptive controller is currently a design implication, not an evaluated result, but provides a natural next step for field deployment.