Summary of "Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite"

Summary (Overview)

• This study directly measures whether vendor-native agent harnesses outperform neutral multi-provider harnesses when paired with the same language model on a private, contamination-controlled suite of 256 coding tasks.
• Using paired same-model contrasts (claude-opus-4-8 and gpt-5.5 each run under both their vendor-native SDK and the neutral deepagents harness), the study finds no resolvable average solve-rate advantage for either harness type: Opus difference = −1.25 pp (95% CI [−10.0, +7.5]), GPT-5.5 difference = +1.25 pp (95% CI [−4.4, +6.9]).
• A post-hoc workload interaction emerges on Opus 4.8: the native harness trails by 9.0 pp on repository tasks but leads by 23.7 pp on contest tasks (permutation p = 0.003), though this partition was chosen after seeing the data.
• The neutral harness costs 1.2–1.6× more per solved task on observed usage, takes longer, and hits the wall-clock ceiling far more often—22 of 81 ceilinged runs had already produced passing patches.
• A telemetry usage-semantics defect was discovered post-study: cache tokens were double-counted for the neutral harness, inflating its apparent cost by 4.8–7.7× in the original ledger; all cost figures were re-derived from raw per-turn events.

Introduction and Theoretical Foundation

An agentic coding system couples a language model to a harness: the tools, prompts, truncation policies, and control flow that convert a chat model into an autonomous software engineer. Vendors ship harnesses tuned to their own models, and practitioners commonly assume the vendor-native pairing solves more tasks. However, existing benchmarks either vary the model while holding the harness fixed, or report vendor harness and model as one opaque system—neither design answers the question of which harness software to use when the model is already chosen.

The study was planned in June 2026 and executed in August 2026 around two research questions fixed in the program plan before any scored run:

• RQ1: Does a vendor-native harness have a capability premium over a neutral harness on the same model? (Hypotheses H1–H3 cover existence, vendor dependence, and concentration by task type.)
• RQ2: Is the capability winner also the cost-per-solved-task winner? (Hypothesis H4)

Two methodological problems shape the design:

1. Contamination: Public benchmarks leak into training corpora, so the suite is private—179 repository tasks mined from four production codebases plus 77 contest tasks published after a mechanically derived eligibility date (2026-03-02 = newest binding cutoff among study models, Opus 4.8 at 2026-01-31, plus a 30-day buffer). A machine-readable cutoff registry was frozen before data collection, and a runtime drift gate verifies served model identity before every scored phase.

2. Cost measurement: Agent evaluations increasingly report dollar costs derived from harness-reported token usage. The study's own telemetry normalizer applied one SDK's convention for cache tokens to all three harnesses, producing a defect (§5.1) that inflated cost estimates for the neutral harness by 4.8–7.7×. This revision re-derives every cost figure from raw per-turn events.

Methodology

Experimental Design

Six cells cross three harnesses with four models:

Key design features:

• Pool selection: A screening pass ran all 256 tasks under C1 and C3 at k=1. All 24 discordant tasks formed the "band"; remaining slots filled alternately from both-pass and both-fail pools to a target of 80 tasks (28+28), locked by hash.
• Repeats: k=2 for core cells C1–C4, k=1 for side cells C5–C6, giving 800 main-run keys.
• Execution substrate: One KVM microVM per run, identical repository seeds and prompts, a shared wire protocol, and an append-only canonical event ledger. Hypervisor isolation is the only sandbox.
• Wall-clock ceiling: 1,200 seconds; the driver cancels the agent and the oracle grades whatever patch is present at cancellation.
• Grading: Offline, Docker-isolated oracle with manifest-driven service sidecars (PostgreSQL, MSSQL, mock-OIDC), validated by polarity (gold passes, base fails) and determinism invariants.

Statistical Methods

• Primary uncertainty: Task-level bootstrap intervals (10,000 resamples of tasks)
• H1 test: Sign-flip permutation on per-task differences
• H2 test: GEE logistic model (pass ~ nativeness × vendor + workload, exchangeable within-task correlation, robust standard errors)
• H3 test: Workload label permutation (interaction null) and stratified sign-flip test
• Multiplicity: Benjamini–Hochberg FDR over primary tests

Empirical Validation / Results

Capability Results

Paired contrasts (n = 80 tasks each):

Cell solve rates (frozen pool of 80 tasks):

Workload split (exploratory, post-hoc):

The interaction test gives p = 0.003 for the workload-label permutation and p = 0.014 for the stratified sign-flip test. With Benjamini–Hochberg control over the four primary tests, the interaction has q = 0.013.

Correctness vs. autonomous completion: 22 of 81 graded runs that hit the wall-clock ceiling had produced passing patches (C2: 11 of 32, C4: 4 of 7, C5: 2 of 12, C6: 5 of 29). The neutral harness hit the ceiling far more often (C2: 32 of 160 final attempts vs. C1: 1; C4: 7 vs. C3: 0) and made about twice the tool invocations per session (medians C1 44.5 vs. C2 90.5; C3 51.5 vs. C4 75).

Cost Results

The telemetry defect: The host normalizer added cache-read and cache-write tokens to tokens_in when they appeared in usage records. This is correct for Anthropic's API (which reports input exclusive of cache) but double-counts for LangChain's usage_metadata (which reports input inclusive of cache). The result: the neutral harness's cache-read share appeared to be ~48% instead of the true 87–98%.

Corrected cost per cell (USD, list prices):

Observed-cost ratios per solved task:

Account-level reconciliation: On the OpenAI account (only 6 rows without usage records), the corrected estimator reproduces billed spend within 2–4%. On the Anthropic account, a residual of 25–29% remains (58 rows without usage records). Allocating the unrecorded spend entirely to either cell moves the Opus ratio between 0.7 and 2.3, so the billed ordering on Anthropic is unresolved.

Theoretical and Practical Implications

1. Workload composition changes the aggregate comparison: The Opus contrast of −1.25 pp is 61/80 of a −9.0 pp repository effect plus 19/80 of a +23.7 pp contest effect. A single aggregate solve rate is not the quantity a deployment decision needs; teams should measure the harness effect under their own workload mix.

2. Correct output and autonomous completion are different endpoints: A benchmark that scores only the patch will call harnesses equivalent when a user waiting for the agent to finish would not. Both endpoints—solve rate and time-to-completion—should be reported together.

3. Measured token cost and billed operational cost need different evidence: Raw per-turn events identify what every recorded turn consumed, but billed spend includes turns that never produced a usage record. Researchers must pin usage semantics before applying prices, then reconcile against money that left the account.

4. A cap experiment needs demonstrated exposure to the cap: The truncation probe was inconclusive because the archive did not retain raw tool-output lengths, making the manipulation check impossible.

5. Methodological contribution: The study documents a usage-semantics defect where input-token fields are cache-inclusive on some SDKs and cache-exclusive on others—a normalizer assuming one convention doubles the other. This is a reproducible source-to-ledger reconciliation and account-level check against billed spend.

Conclusion

Neither same-model contrast resolved an average harness advantage on the private 80-task pool. The Opus 4.8 difference is −1.25 pp with a task-bootstrap interval of [−10.0, +7.5], and the GPT-5.5 difference is +1.25 pp with an interval of [−4.4, +6.9]. On Opus, the average combines a repository stratum favoring the neutral harness with a contest stratum favoring the native one—a post-hoc pattern with permutation p = 0.003 that a designed replication should test. The neutral harness reached the ceiling more often, made about twice the tool calls, and on observed usage at list prices cost 1.2–1.6× as much per solved task, with the billed ordering on Anthropic unresolved.

The study's central methodological takeaway: pin every SDK's usage semantics before pricing a token, compare raw per-turn events with the ledger, and compare the ledger with billed spend. The tasks remain private to preserve the suite's contamination-controlled status; the orchestrator, grading oracle, reanalysis code, and derived aggregates are released for replication.

Future directions: A designed replication should pre-specify the workload strata, power the contest stratum (19 tasks here), and use three repeats. Trajectory-level measurement is needed to determine whether correct patches exist earlier in ceilinged trajectories and whether stopping can be optimized safely.