Summary (Overview)

• This paper systematically investigates the impact of harness (scaffold) choice on coding-agent evaluation, holding models fixed while varying harnesses across 300 trials (3 harnesses × 2 models × 50 tasks).
• Key finding: harness choice induces up to a 40× difference in tokens per solved task, while paired pass-rate differences remain only 0–8 percentage points (95% paired-task bootstrap CIs include zero except for the largest gap).
• Failure-mode fingerprints are harness-specific and replicate across models: Goose shows REASON-dominated failures, OpenHands-SDK shows VERIFY/MAX TURNS, and OpenCode shows time-out/idle-loop failures.
• No-action turns (reasoning-only turns with no file edits or commands) serve as a mechanistic proxy for the cost gap—OpenCode averages ~2.0–2.16 per task versus 0.2–0.3 for Goose, a 10× ratio.
• The authors argue the unit of comparison should be the harness–model pair, not the model alone, and recommend reporting token usage, latency, and harness specifications alongside any model comparison.

---

Introduction and Theoretical Foundation

The paper addresses a critical gap in coding-agent evaluation: public leaderboards typically rank systems by model name and pass rate while treating the surrounding harness as either undisclosed, varied opportunistically, or implicitly constant. From a human-centered perspective, developers need to know cost per resolved task, time-to-completion, and oversight burden—none of which are captured by model-only scores.

The motivation builds on prior observations, notably the Terminal-Bench 2.0 paper reporting Claude Opus 4.5 at 52.1% with one harness and 57.8% with another, consuming 256.9M vs 3.9M input tokens respectively—a 65× token difference for a 5.7-point accuracy gain.

Key theoretical foundations include:

• Terminal-Bench (Merrill et al., 2026) and Terminal-Bench Pro (Wang et al., 2025a; Alibaba, 2026) for realistic command-line coding tasks
• SWE-bench literature noting scaffold choice as a confound to control rather than a phenomenon to study
• Wang et al. (2024) arguing agents are systems rather than models, without quantifying efficiency impacts
• DL4C call for "Interaction-Aware Benchmarks" asking for metrics beyond task completion that capture interaction quality, oversight burden, and verifiability

---

Methodology

Task Selection

• 50-task stratified random sample from Terminal-Bench Pro's 200-task public set
• 8 domain categories: BUG (8), BUILD (6), DATA (7), IMPL (8), ML (8), PUZZLE (5), SEC (6), SYS (2)
• All tasks have deterministic evaluation via pytest suites

Models

• Qwen 3.6 Plus (Alibaba Cloud): hybrid linear-attention MoE with always-on chain-of-thought; reported 61.6 on Terminal-Bench 2.0
• MiniMax M2.5: 228.7B-parameter MoE, 10B active; reported 80.2% on SWE-bench Verified

Harnesses

• Goose (Agentic AI Foundation): heavyweight IDE agent with eager file-tree preinjection
• OpenCode (Open-Code Contributors): persistent tool-loop coding agent; no automatic context preloading
• OpenHands-SDK (Wang et al., 2025b): micro-agent architecture with sub-agent delegation, internal retry, and explicit verification steps

Standardization Protocol

Held constant: verbatim instructions, native test suites, Daytona sandbox per task, 900-second wall-time cap, OpenRouter as the model gateway, and a largely identical system prompt template (same operating principles, non-negotiables, and a shared four-skill operational playbook: build-and-env, deep-debug, terminal-investigation, test-driven-solve).

Maximum turns: 40 for Goose and OpenHands-SDK; OpenCode is bounded only by the wall-time cap (no turn-budget flag exposed through Harbor).

Metrics

• Solved (boolean), turns used, tokens total, average no-action turns (turns with no file modification and no new shell command), hit-turn-budget count, wall seconds
• A 6-class failure taxonomy: REASON, VERIFY, TIME, MAX TURNS, HANG, ERROR
• Bootstrap uncertainty via 95% paired-task bootstrap intervals (B = 10,000)

Token accounting: Tokens per solved task is amortized across the cell:

\text{Tokens per solved task} = \frac{\sum_{t \in \text{cell}} \text{tokens_total}_t}{|\{t : \text{solved}_t\}|}

Failed and infrastructure-error trials are included in the numerator—this reflects the cost a deployer absorbs per successful task.

---

Empirical Validation / Results

Pass Rates

With n = 50 per cell, most pass-rate differences are not statistically distinguishable from zero:

Within a model, the harness range is 2–8 pp; across models within a harness, 4–10 pp.

Token Cost per Solved Task

The ordering is identical for both models: Goose ≪ OpenHands-SDK < OpenCode. The gap is not driven by turn counts (average turns are 21–27 for OpenCode vs 18–25 for Goose, ~1.2×), but rather by per-turn context growth, tool serialization, and harness-specific token accounting.

Bootstrap robustness: Goose's CI upper bound (40–61K) sits well below OpenCode's CI lower bound (733K–1.01M), confirming the order-of-magnitude gap.

Pareto Frontier

Goose lies on the Pareto frontier for both models (best cost at equal-or-better pass rate); OpenCode is Pareto-dominated by both Goose and OpenHands-SDK.

No-Action Turns

• OpenCode: 2.0–2.16 no-action turns per task
• Goose: 0.2–0.3 per task (10× ratio, replicating across both models)
• Each no-action turn carries a full context window of input tokens due to message-history accumulation

Failure-Mode Fingerprints

Three distinct fingerprints replicate across both models:

• Goose: REASON-dominated (20/15 failures, zero VERIFY)—stops cleanly when stuck
• OpenHands-SDK: VERIFY + MAX TURNS—persists toward closure, accepting plausible-but-incorrect solutions
• OpenCode: TIME + idle spinning, zero VERIFY—commits only to solutions that pass tests but exhausts wall-time

Pass Rate by Task Category

OpenHands-SDK substantially outperforms on IMPL (fresh implementation): 69% vs 38%/38% for Goose/OpenCode. All harnesses struggle on SEC (8–17%) and fail completely on SYS (0%).

Harness vs. Model Effects Summary

---

Theoretical and Practical Implications

Human-Center Consequences of the Cost Gap

• Dollar cost: ~40× more tokens means ~40× more API spend, dominating the 1–2× price spread between the strongest current models
• Wall-clock time: No-action turns are round-trip API calls the user waits through—a per-task "wait tax"
• Oversight burden: Users must read/skim idle turns to confirm they were not destructive; "re-reading the same file three times" is cheap for the model but expensive for the human reviewer

Failure Fingerprint Implications

Each harness implies a different oversight discipline:

• Goose gives an honest "I cannot do this" rather than a plausible-but-wrong patch
• OpenHands-SDK requires users to verify outputs more carefully (VERIFY failures)
• OpenCode trusts positive verdicts but costs more in time and tokens

Benchmark Reporting Recommendations

The authors recommend that leaderboards adopt harness–model pairs as the unit of evaluation with three first-class metrics alongside pass rate:

1. Tokens per solved task
2. Average no-action turns per task
3. Failure-category vector

---

Conclusion

The paper provides controlled evidence that harness choice introduces a 40× token-cost difference and harness-specific failure fingerprints in coding-agent evaluation, while shifting paired pass rate by at most 8 percentage points. Both findings replicate independently across two recent models.

Key takeaways:

• The unit of evaluation should be the harness–model pair, not the model alone
• Cost, idle-turn, and failure-mode signals should be reported alongside pass rate
• Model-only leaderboards conflate two independent sources of variance and discard deployment-relevant signals

Limitations: n = 50 tasks (pass-rate effects within bootstrap noise); token accounting asymmetry for Goose (no input/output breakdown); turn-budget asymmetry (OpenCode lacks a turn-budget flag); three harnesses and two models may not generalize to all scaffold architectures; provider defaults may evolve.

Future directions implicitly include: expanding to more harnesses/models, standardizing token accounting across harnesses, and developing richer interaction-aware benchmark metrics.

Data and Code Availability: Anonymized configs, raw Harbor trial logs, aggregated snapshots, and analysis pipeline available at https://anonymous.4open.science/r/scaffold-effects-dl4c-supp/