Summary (Overview)

• HARNESSOPT-BENCH is a new benchmark for evaluating how well frontier LLMs perform at harness optimization—the iterative, evaluation-guided improvement of an agent's prompts, tools, control flow, memory, and orchestration code.
• The benchmark introduces a trusted, reproducible evaluation protocol that enforces a fixed target-evaluation budget, keeps a held-out test partition inaccessible during search, and versions every candidate for audit.
• Experiments with 5 frontier LLMs across 4 downstream tasks and 111 scored runs show that optimizer model choice matters more than coding harness choice (model contrast ~1.8× larger than harness contrast).
• Native harnesses provide no consistent advantage over a shared coding harness (11 vs. 9 wins across 20 model–task pairs).
• Key capability gaps remain: broader search breadth correlates with higher gain, while detailed trace reading is rarely used and not positively associated with gain.

Introduction and Theoretical Foundation

As LLMs are deployed within agentic systems, their capabilities depend not only on model weights but on the harness — the program specifying prompts, tools, control flow, memory, and orchestration code. The same model can exhibit substantially different capabilities under different harnesses, making harness optimization (iterative improvement of a harness on a measured outcome under a fixed budget) an increasingly important part of building capable AI systems.

The paper motivates harness optimization as a distinct capability from coding ability:

• Expensive and noisy evaluation: Unlike test suites that report cheaply whether a code change is correct, harness changes must be estimated by running stochastic agents over many cases at substantial cost. Optimizers must diagnose failures from incomplete evidence, spend a limited budget, separate real improvement from noise, and decide what to deploy.
• Long-horizon, tool-diverse reasoning: The task requires reasoning-driven interpretation of a stochastic system in pursuit of a measured metric, over a diverse tool ecosystem.
• Generalization: The held-out test partition ensures improvement reflects generalization rather than fitting to a visible score.

Prior work differs in the role assigned to the LLM: some use it as a mutation operator (e.g., ShinkaEvolve, GEPA), while others (e.g., VERO, MetaHarness) use coding agents as end-to-end optimizers that edit the harness as a codebase. HARNESSOPT-BENCH is complementary: it fixes the optimization problem and evaluation protocol so that optimizer models, harnesses, and search algorithms can be compared on common ground.

Methodology

Optimization Problem Formalization

The benchmark formalizes harness optimization as a constrained, stochastic program-optimization problem (Algorithm 1). Key elements:

• Candidates and invariants: A candidate harness $H$ is an executable codebase, editable subject to a fixed execution interface and immutable paths. The task fixes invariants $\theta = (\mathcal{M}, E, V)$: models available, environment per case, and verifier mapping a trajectory to a score in $[0,1]$.

• Evaluation and disclosure: Cases are partitioned into disjoint development, validation, and test sets. Expected score on partition $D$ is defined as:

$$\mathcal{E}_{\theta}(H; \mathcal{D}) = \mathbb{E}_{x \sim \mathcal{D}} \mathbb{E}_{\tau \sim \mathrm{Rollout}(H, \theta, x)}[V(\tau, x)]$$

• Budget: Each evaluation request incurs a cost vector $c_j$, and search must satisfy $\sum_j c_j \leq B$ componentwise. Primary caps: 100 evaluation calls per partition, four full case passes per partition, and a cap on total expendable target-model tokens.

• Objective: Maximize expected improvement over the pinned seed on the held-out partition, with normalized gain defined as:

$$g = \frac{\mathcal{E}_{\theta}(H^{+}) - \mathcal{E}_{\theta}(H_{0})}{1 - \mathcal{E}_{\theta}(H_{0})}$$

Experimental Design

• Task grid: 5 optimizer models (claude-opus-5, claude-sonnet-5, gpt-5.6-sol, gpt-5.6-terra, kimi-k3) × 2 harnesses (shared opencode; native claude-code, codex, or kimi-cli) = 10 configurations, across 4 tasks (OfficeQA, BrowseComp-Plus, Terminal-Bench, GAIA).
• Scoring: Each held-out evaluation is the mean of three attempts per test case; each configuration run twice.
• Analysis protocol:
  • Outcome: normalized gain $g$ (fraction of headroom above pinned baseline).
  • Resolution band: task-specific threshold from scoring the same candidate twice; differences smaller than the band are treated as unresolved.
  • Composite model score (LSS-λ): task-adjusted mean performance relative to the grand mean, expressed in normalized-gain units:

$$\widehat{\lambda}_m = \frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \bar{g}_{mt} - \frac{1}{|\mathcal{M}| |\mathcal{T}|} \sum_{m' \in \mathcal{M}} \sum_{t \in \mathcal{T}} \bar{g}_{m't}$$

Empirical Validation / Results

Key Findings

1. Model choice dominates harness choice.

Table 1: Normalized gain for every optimizer on every task. The best in each column is bolded. GAIA's seed is a non-functional stub with a measured-zero baseline, so gain there is the raw held-out score.

• Holding task and harness fixed, changing the optimizer model moves gain by 0.142 on average; holding task and model fixed, changing the harness moves it by 0.079 — model contrast ~1.8× larger.

• The strongest configuration captures ~two-thirds of OfficeQA headroom, the weakest is unresolved from zero on two tasks.

2. Model progress is measurable. Across 5 GPT releases on OfficeQA, gain rises monotonically from +0.03 to +0.49. Across 5 Claude Opus releases, gain ranges +0.37 to +0.59, non-monotonic but with first-to-last spread exceeding the resolution band.

3. Where current optimizers fall short:

• Broader search is associated with greater gain: the fraction of 8 pre-specified harness levers touched during search positively correlates with gain on every task (Spearman ρ from +0.34 to +0.88).
• Trace reading is not associated with higher gain: the share of actions spent reading evaluation output is negatively associated with gain (−0.31 to −0.64). Detailed trace spans were requested only 16 times by 7 of 111 cells.
• Case passes, not evaluation calls, bind: median optimizer uses 8 calls (4% of the cap) but 82% of its case allowance; 55 of 100 cells exhaust at least one partition's case budget.
• Visible validation scores are optimistic: most submitted candidates' test scores are lower than the best validation score observed during search.

4. Native harnesses provide no consistent advantage. Across 20 model–task pairs, shared harness wins 11, native wins 9, 0 ties. On GAIA (only task with >2 harness levels), the best harness depends on the model: both GPT models are ~4–5 resolution bands better under codex, while the two Claudes and Kimi sit within a band or two of zero either way.

Theoretical and Practical Implications

• Harness optimization is a measurable, discriminative capability: The benchmark resolves variation among frontier models and successive releases, establishing that this capability can be empirically evaluated.
• Model capability, not scaffold, is the primary bottleneck: The finding that model differences dominate harness differences suggests that improving the optimizer model itself (rather than its coding interface) is the more promising avenue.
• The evaluation protocol is crucial: The trusted execution boundary, held-out test partition, and budget enforcement are necessary conditions for isolating optimizer capability, and are applicable beyond harness optimization to other forms of automated research and self-improvement.
• Search process matters: Broader exploration is associated with better outcomes, while deep trace inspection is not — offering actionable guidance for optimizer design.

Conclusion

HARNESSOPT-BENCH turns harness engineering from an optimizer-specific demonstration into a reproducible evaluation target. Current frontier models can improve agents, but unevenly: the strongest search broadly, yet gains remain task-dependent and often too close for fine-grained ranking. The benchmark is designed to be hack-resistant, not hackproof — repeated development and validation feedback may reward strategies specific to a fixed evaluation, and future versions should introduce per-run jitter. The seed harness itself is a task-specific prior; a controlled ladder of harness completeness would reveal how optimizer performance changes. The next frontier is not merely better agents, but models that reliably make agents better.