# The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation

> Harness choice drives up to a 40x token cost difference per solved coding task while shifting pass rates by only 0-8 percentage points, making harness-model pairs the correct evaluation unit.

- **Source:** [arXiv](https://arxiv.org/abs/2607.22585)
- **Published:** 2026-08-22
- **Permalink:** https://picx.dev/p/R06uw3
- **Whiteboard:** https://picx.dev/p/R06uw3/image

## Summary

## Summary (Overview)

- This paper systematically investigates the impact of harness (scaffold) choice on coding-agent evaluation, holding models fixed while varying harnesses across 300 trials (3 harnesses × 2 models × 50 tasks).
- **Key finding: harness choice induces up to a 40× difference in tokens per solved task**, while paired pass-rate differences remain only 0–8 percentage points (95% paired-task bootstrap CIs include zero except for the largest gap).
- **Failure-mode fingerprints are harness-specific and replicate across models**: Goose shows REASON-dominated failures, OpenHands-SDK shows VERIFY/MAX TURNS, and OpenCode shows time-out/idle-loop failures.
- **No-action turns** (reasoning-only turns with no file edits or commands) serve as a mechanistic proxy for the cost gap—OpenCode averages ~2.0–2.16 per task versus 0.2–0.3 for Goose, a 10× ratio.
- The authors argue the **unit of comparison should be the harness–model pair**, not the model alone, and recommend reporting token usage, latency, and harness specifications alongside any model comparison.

---

## Introduction and Theoretical Foundation

The paper addresses a critical gap in coding-agent evaluation: public leaderboards typically rank systems by model name and pass rate while treating the surrounding harness as either undisclosed, varied opportunistically, or implicitly constant. From a human-centered perspective, developers need to know **cost per resolved task, time-to-completion, and oversight burden**—none of which are captured by model-only scores.

The motivation builds on prior observations, notably the Terminal-Bench 2.0 paper reporting Claude Opus 4.5 at 52.1% with one harness and 57.8% with another, consuming 256.9M vs 3.9M input tokens respectively—a **65× token difference for a 5.7-point accuracy gain**.

Key theoretical foundations include:
- **Terminal-Bench** (Merrill et al., 2026) and **Terminal-Bench Pro** (Wang et al., 2025a; Alibaba, 2026) for realistic command-line coding tasks
- **SWE-bench** literature noting scaffold choice as a confound to control rather than a phenomenon to study
- **Wang et al. (2024)** arguing agents are systems rather than models, without quantifying efficiency impacts
- **DL4C call for "Interaction-Aware Benchmarks"** asking for metrics beyond task completion that capture interaction quality, oversight burden, and verifiability

---

## Methodology

### Task Selection
- 50-task stratified random sample from Terminal-Bench Pro's 200-task public set
- 8 domain categories: BUG (8), BUILD (6), DATA (7), IMPL (8), ML (8), PUZZLE (5), SEC (6), SYS (2)
- All tasks have deterministic evaluation via pytest suites

### Models
- **Qwen 3.6 Plus** (Alibaba Cloud): hybrid linear-attention MoE with always-on chain-of-thought; reported 61.6 on Terminal-Bench 2.0
- **MiniMax M2.5**: 228.7B-parameter MoE, 10B active; reported 80.2% on SWE-bench Verified

### Harnesses
- **Goose** (Agentic AI Foundation): heavyweight IDE agent with eager file-tree preinjection
- **OpenCode** (Open-Code Contributors): persistent tool-loop coding agent; no automatic context preloading
- **OpenHands-SDK** (Wang et al., 2025b): micro-agent architecture with sub-agent delegation, internal retry, and explicit verification steps

### Standardization Protocol
Held constant: verbatim instructions, native test suites, Daytona sandbox per task, 900-second wall-time cap, OpenRouter as the model gateway, and a largely identical system prompt template (same operating principles, non-negotiables, and a shared four-skill operational playbook: build-and-env, deep-debug, terminal-investigation, test-driven-solve).

Maximum turns: 40 for Goose and OpenHands-SDK; OpenCode is bounded only by the wall-time cap (no turn-budget flag exposed through Harbor).

### Metrics
- Solved (boolean), turns used, tokens total, average no-action turns (turns with no file modification and no new shell command), hit-turn-budget count, wall seconds
- A 6-class failure taxonomy: **REASON, VERIFY, TIME, MAX TURNS, HANG, ERROR**
- Bootstrap uncertainty via 95% paired-task bootstrap intervals (B = 10,000)

**Token accounting**: Tokens per solved task is amortized across the cell:

$$\text{Tokens per solved task} = \frac{\sum_{t \in \text{cell}} \text{tokens_total}_t}{|\{t : \text{solved}_t\}|}$$

Failed and infrastructure-error trials are included in the numerator—this reflects the cost a deployer absorbs per successful task.

---

## Empirical Validation / Results

### Pass Rates
With n = 50 per cell, most pass-rate differences are not statistically distinguishable from zero:

| Harness | Qwen 3.6 Plus | MiniMax M2.5 |
|---------|---------------|--------------|
| Goose | 48.0% [34.0, 62.0] | 38.0% [24.0, 52.0] |
| OpenCode | 50.0% [36.0, 64.0] | 46.0% [32.0, 60.0] |
| OpenHands-SDK | 50.0% [36.0, 64.0] | 46.0% [32.0, 60.0] |

Within a model, the harness range is 2–8 pp; across models within a harness, 4–10 pp.

### Token Cost per Solved Task

| Harness | Model | Tokens/Solve | ×Goose | Avg. Turns |
|---------|-------|--------------|--------|------------|
| **Goose** | Qwen | 28,142 | 1.0 | 17.96 |
| **Goose** | MiniMax | 36,950 | 1.0 | 25.25 |
| **OpenHands** | Qwen | 841,201 | 29.9 | 25.95 |
| **OpenHands** | MiniMax | 843,286 | 22.8 | 24.19 |
| **OpenCode** | Qwen | 1,147,740 | 40.8 | 21.71 |
| **OpenCode** | MiniMax | 1,546,977 | 41.9 | 27.46 |

The ordering is identical for both models: **Goose ≪ OpenHands-SDK < OpenCode**. The gap is not driven by turn counts (average turns are 21–27 for OpenCode vs 18–25 for Goose, ~1.2×), but rather by per-turn context growth, tool serialization, and harness-specific token accounting.

**Bootstrap robustness**: Goose's CI upper bound (40–61K) sits well below OpenCode's CI lower bound (733K–1.01M), confirming the order-of-magnitude gap.

### Pareto Frontier
Goose lies on the Pareto frontier for both models (best cost at equal-or-better pass rate); OpenCode is Pareto-dominated by both Goose and OpenHands-SDK.

### No-Action Turns
- OpenCode: 2.0–2.16 no-action turns per task
- Goose: 0.2–0.3 per task (10× ratio, replicating across both models)
- Each no-action turn carries a full context window of input tokens due to message-history accumulation

### Failure-Mode Fingerprints

| Harness | Model | Solved | REASON | VERIFY | TIME | MAX TURNS | HANG | ERROR |
|---------|-------|--------|--------|--------|------|-----------|------|-------|
| **Goose** | Qwen | 23 | 20 | 0 | 3 | 1 | 2 | 1 |
| **Goose** | MiniMax | 17 | 15 | 0 | 7 | 8 | 2 | 1 |
| **OpenCode** | Qwen | 25 | 15 | 0 | 5 | 0 | 4 | 1 |
| **OpenCode** | MiniMax | 22 | 16 | 0 | 10 | 0 | 1 | 1 |
| **OpenHands** | Qwen | 25 | 3 | 6 | 5 | 6 | 4 | 1 |
| **OpenHands** | MiniMax | 21 | 1 | 8 | 6 | 6 | 7 | 1 |

Three distinct fingerprints replicate across both models:
- **Goose**: REASON-dominated (20/15 failures, zero VERIFY)—stops cleanly when stuck
- **OpenHands-SDK**: VERIFY + MAX TURNS—persists toward closure, accepting plausible-but-incorrect solutions
- **OpenCode**: TIME + idle spinning, zero VERIFY—commits only to solutions that pass tests but exhausts wall-time

### Pass Rate by Task Category
OpenHands-SDK substantially outperforms on IMPL (fresh implementation): 69% vs 38%/38% for Goose/OpenCode. All harnesses struggle on SEC (8–17%) and fail completely on SYS (0%).

### Harness vs. Model Effects Summary

| Variable | Harness | Model Upgrade |
|----------|---------|---------------|
| Pass rate (paired) | 0–8 pp | 4–10 pp |
| Tokens/solved task | 40× | 1.0–1.3× |
| Failure profile | fingerprint | consistent |
| No-action turns | 10× | <1.1× |

---

## Theoretical and Practical Implications

### Human-Center Consequences of the Cost Gap
- **Dollar cost**: ~40× more tokens means ~40× more API spend, dominating the 1–2× price spread between the strongest current models
- **Wall-clock time**: No-action turns are round-trip API calls the user waits through—a per-task "wait tax"
- **Oversight burden**: Users must read/skim idle turns to confirm they were not destructive; "re-reading the same file three times" is cheap for the model but expensive for the human reviewer

### Failure Fingerprint Implications
Each harness implies a different oversight discipline:
- **Goose** gives an honest "I cannot do this" rather than a plausible-but-wrong patch
- **OpenHands-SDK** requires users to verify outputs more carefully (VERIFY failures)
- **OpenCode** trusts positive verdicts but costs more in time and tokens

### Benchmark Reporting Recommendations
The authors recommend that leaderboards adopt harness–model pairs as the unit of evaluation with three first-class metrics alongside pass rate:
1. Tokens per solved task
2. Average no-action turns per task
3. Failure-category vector

---

## Conclusion

The paper provides controlled evidence that harness choice introduces a **40× token-cost difference** and harness-specific failure fingerprints in coding-agent evaluation, while shifting paired pass rate by at most 8 percentage points. Both findings replicate independently across two recent models.

**Key takeaways:**
- The unit of evaluation should be the harness–model pair, not the model alone
- Cost, idle-turn, and failure-mode signals should be reported alongside pass rate
- Model-only leaderboards conflate two independent sources of variance and discard deployment-relevant signals

**Limitations:** n = 50 tasks (pass-rate effects within bootstrap noise); token accounting asymmetry for Goose (no input/output breakdown); turn-budget asymmetry (OpenCode lacks a turn-budget flag); three harnesses and two models may not generalize to all scaffold architectures; provider defaults may evolve.

**Future directions** implicitly include: expanding to more harnesses/models, standardizing token accounting across harnesses, and developing richer interaction-aware benchmark metrics.

**Data and Code Availability:** Anonymized configs, raw Harbor trial logs, aggregated snapshots, and analysis pipeline available at https://anonymous.4open.science/r/scaffold-effects-dl4c-supp/

---

_Markdown view of https://picx.dev/p/R06uw3, served by PicX — AI-generated visual whiteboard summaries of research papers._