# HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

> HARNESSOPT-BENCH shows optimizer model choice matters 1.8x more than coding harness choice for agent improvement, with broader search driving gains and trace reading providing no benefit.

- **Source:** [arXiv](https://arxiv.org/abs/2608.06301)
- **Published:** 2026-08-22
- **Permalink:** https://picx.dev/p/L2OmH2
- **Whiteboard:** https://picx.dev/p/L2OmH2/image

## Summary

## Summary (Overview)

- **HARNESSOPT-BENCH** is a new benchmark for evaluating how well frontier LLMs perform at **harness optimization**—the iterative, evaluation-guided improvement of an agent's prompts, tools, control flow, memory, and orchestration code.
- The benchmark introduces a **trusted, reproducible evaluation protocol** that enforces a fixed target-evaluation budget, keeps a held-out test partition inaccessible during search, and versions every candidate for audit.
- Experiments with **5 frontier LLMs** across **4 downstream tasks** and **111 scored runs** show that **optimizer model choice matters more than coding harness choice** (model contrast ~1.8× larger than harness contrast).
- **Native harnesses provide no consistent advantage** over a shared coding harness (11 vs. 9 wins across 20 model–task pairs).
- Key capability gaps remain: broader search breadth correlates with higher gain, while detailed trace reading is rarely used and not positively associated with gain.

## Introduction and Theoretical Foundation

As LLMs are deployed within **agentic systems**, their capabilities depend not only on model weights but on the *harness* — the program specifying prompts, tools, control flow, memory, and orchestration code. The same model can exhibit substantially different capabilities under different harnesses, making **harness optimization** (iterative improvement of a harness on a measured outcome under a fixed budget) an increasingly important part of building capable AI systems.

The paper motivates harness optimization as a *distinct capability* from coding ability:

- **Expensive and noisy evaluation**: Unlike test suites that report cheaply whether a code change is correct, harness changes must be estimated by running stochastic agents over many cases at substantial cost. Optimizers must diagnose failures from incomplete evidence, spend a limited budget, separate real improvement from noise, and decide what to deploy.
- **Long-horizon, tool-diverse reasoning**: The task requires reasoning-driven interpretation of a stochastic system in pursuit of a measured metric, over a diverse tool ecosystem.
- **Generalization**: The held-out test partition ensures improvement reflects generalization rather than fitting to a visible score.

Prior work differs in the role assigned to the LLM: some use it as a mutation operator (e.g., ShinkaEvolve, GEPA), while others (e.g., VERO, MetaHarness) use coding agents as end-to-end optimizers that edit the harness as a codebase. HARNESSOPT-BENCH is complementary: it fixes the optimization problem and evaluation protocol so that optimizer models, harnesses, and search algorithms can be compared on common ground.

## Methodology

### Optimization Problem Formalization

The benchmark formalizes harness optimization as a constrained, stochastic program-optimization problem (Algorithm 1). Key elements:

- **Candidates and invariants**: A candidate harness $H$ is an executable codebase, editable subject to a fixed execution interface and immutable paths. The task fixes invariants $\theta = (\mathcal{M}, E, V)$: models available, environment per case, and verifier mapping a trajectory to a score in $[0,1]$.

- **Evaluation and disclosure**: Cases are partitioned into disjoint development, validation, and test sets. Expected score on partition $D$ is defined as:

$$
\mathcal{E}_{\theta}(H; \mathcal{D}) = \mathbb{E}_{x \sim \mathcal{D}} \mathbb{E}_{\tau \sim \mathrm{Rollout}(H, \theta, x)}[V(\tau, x)]
$$

- **Budget**: Each evaluation request incurs a cost vector $c_j$, and search must satisfy $\sum_j c_j \leq B$ componentwise. Primary caps: 100 evaluation calls per partition, four full case passes per partition, and a cap on total expendable target-model tokens.

- **Objective**: Maximize expected improvement over the pinned seed on the held-out partition, with **normalized gain** defined as:

$$
g = \frac{\mathcal{E}_{\theta}(H^{+}) - \mathcal{E}_{\theta}(H_{0})}{1 - \mathcal{E}_{\theta}(H_{0})}
$$

### Experimental Design

- **Task grid**: 5 optimizer models (claude-opus-5, claude-sonnet-5, gpt-5.6-sol, gpt-5.6-terra, kimi-k3) × 2 harnesses (shared *opencode*; native *claude-code*, *codex*, or *kimi-cli*) = 10 configurations, across 4 tasks (OfficeQA, BrowseComp-Plus, Terminal-Bench, GAIA).
- **Scoring**: Each held-out evaluation is the mean of three attempts per test case; each configuration run twice.
- **Analysis protocol**: 
  - **Outcome**: normalized gain $g$ (fraction of headroom above pinned baseline).
  - **Resolution band**: task-specific threshold from scoring the same candidate twice; differences smaller than the band are treated as unresolved.
  - **Composite model score (LSS-λ)**: task-adjusted mean performance relative to the grand mean, expressed in normalized-gain units:

$$
\widehat{\lambda}_m = \frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \bar{g}_{mt} - \frac{1}{|\mathcal{M}| |\mathcal{T}|} \sum_{m' \in \mathcal{M}} \sum_{t \in \mathcal{T}} \bar{g}_{m't}
$$

## Empirical Validation / Results

### Key Findings

**1. Model choice dominates harness choice.**

| Model | Optimizer harness | OfficeQA ±0.045 | BrowseComp-Plus ±0.066 | Terminal-Bench ±0.054 | GAIA ±0.035 |
|-------|-------------------|-----------------|-----------------------|----------------------|--------------|
| claude-opus-5 | claude-code | 0.59 | 0.41 | 0.18 | 0.42 |
| claude-opus-5 | opencode | 0.63 | 0.48 | 0.29 | 0.47 |
| claude-sonnet-5 | claude-code | 0.53 | 0.07 | 0.10 | 0.33 |
| claude-sonnet-5 | opencode | 0.51 | 0.15 | 0.15 | 0.25 |
| gpt-5.6-sol | codex | 0.49 | 0.03 | 0.12 | 0.49 |
| gpt-5.6-sol | opencode | 0.29 | 0.09 | 0.13 | 0.31 |
| gpt-5.6-terra | codex | 0.07 | -0.03 | 0.01 | 0.30 |
| gpt-5.6-terra | opencode | 0.14 | 0.02 | 0.04 | 0.17 |
| kimi-k3 | kimi-cli | 0.59 | 0.23 | 0.16 | 0.31 |
| kimi-k3 | opencode | 0.41 | 0.16 | 0.12 | 0.28 |

*Table 1: Normalized gain for every optimizer on every task. The best in each column is bolded. GAIA's seed is a non-functional stub with a measured-zero baseline, so gain there is the raw held-out score.*

- Holding task and harness fixed, changing the optimizer model moves gain by **0.142** on average; holding task and model fixed, changing the harness moves it by **0.079** — model contrast ~1.8× larger.

- The strongest configuration captures ~two-thirds of OfficeQA headroom, the weakest is unresolved from zero on two tasks.

**2. Model progress is measurable.** Across 5 GPT releases on OfficeQA, gain rises monotonically from +0.03 to +0.49. Across 5 Claude Opus releases, gain ranges +0.37 to +0.59, non-monotonic but with first-to-last spread exceeding the resolution band.

**3. Where current optimizers fall short:**
- **Broader search is associated with greater gain**: the fraction of 8 pre-specified harness levers touched during search positively correlates with gain on every task (Spearman ρ from +0.34 to +0.88).
- **Trace reading is not associated with higher gain**: the share of actions spent reading evaluation output is negatively associated with gain (−0.31 to −0.64). Detailed trace spans were requested only 16 times by 7 of 111 cells.
- **Case passes, not evaluation calls, bind**: median optimizer uses 8 calls (4% of the cap) but 82% of its case allowance; 55 of 100 cells exhaust at least one partition's case budget.
- **Visible validation scores are optimistic**: most submitted candidates' test scores are lower than the best validation score observed during search.

**4. Native harnesses provide no consistent advantage.** Across 20 model–task pairs, shared harness wins 11, native wins 9, 0 ties. On GAIA (only task with >2 harness levels), the best harness depends on the model: both GPT models are ~4–5 resolution bands better under codex, while the two Claudes and Kimi sit within a band or two of zero either way.

## Theoretical and Practical Implications

- **Harness optimization is a measurable, discriminative capability**: The benchmark resolves variation among frontier models and successive releases, establishing that this capability can be empirically evaluated.
- **Model capability, not scaffold, is the primary bottleneck**: The finding that model differences dominate harness differences suggests that improving the optimizer model itself (rather than its coding interface) is the more promising avenue.
- **The evaluation protocol is crucial**: The trusted execution boundary, held-out test partition, and budget enforcement are necessary conditions for isolating optimizer capability, and are applicable beyond harness optimization to other forms of automated research and self-improvement.
- **Search process matters**: Broader exploration is associated with better outcomes, while deep trace inspection is not — offering actionable guidance for optimizer design.

## Conclusion

HARNESSOPT-BENCH turns harness engineering from an optimizer-specific demonstration into a reproducible evaluation target. Current frontier models can improve agents, but unevenly: the strongest search broadly, yet gains remain task-dependent and often too close for fine-grained ranking. The benchmark is designed to be **hack-resistant, not hackproof** — repeated development and validation feedback may reward strategies specific to a fixed evaluation, and future versions should introduce per-run jitter. The seed harness itself is a task-specific prior; a controlled ladder of harness completeness would reveal how optimizer performance changes. The next frontier is not merely better agents, but **models that reliably make agents better**.

---

_Markdown view of https://picx.dev/p/L2OmH2, served by PicX — AI-generated visual whiteboard summaries of research papers._