# Rethinking the Evaluation of Harness Evolution for Agents

> Rethinking harness evolution evaluations shows reported gains conflate search effects, as simple test-time scaling (parallel sampling, sequential refinement) consistently outperforms automatic harness evolution.

- **Source:** [arXiv](https://arxiv.org/abs/2607.12227)
- **Published:** 2026-08-15
- **Permalink:** https://picx.dev/p/nNsgCt
- **Whiteboard:** https://picx.dev/p/nNsgCt/image

## Summary

# Rethinking the Evaluation of Harness Evolution for Agents

## Summary (Overview)

- This paper critically re-examines how automatic harness evolution for LLM agents is evaluated, arguing that current evaluation protocols conflate genuine harness design improvements with simple test-time search effects.
- The authors formalize a unified budget framework comparing four methods—parallel sampling, sequential refinement, harness evolution, and harness scaling—under matched feedback and inference budgets.
- Experiments on Terminal-Bench 2.1 with Claude Opus 4.6, GPT-5.4, and GPT-5.4 mini show that automatic harness evolution **does not consistently outperform** simple test-time scaling baselines.
- When unit test cases are unavailable, harness evolution underperforms direct sampling on average (67.4 vs. 68.2 baseline); with unit tests, it still lags behind parallel sampling and sequential refinement on both pass@1 and pass@5.
- Evolved harnesses show **poor generalization** to held-out tasks (average gain of only +0.6 points), suggesting overfitting to the search set rather than learning transferable harness design principles.

---

## Introduction and Theoretical Foundation

### Background and Motivation

LLM agents increasingly depend on external **harnesses**—the prompts, tools, memory, verification routines, and control logic that mediate how a model observes tasks and acts. Prior work shows harness engineering substantially affects agent performance even with a fixed model, yet it remains largely manual. This has motivated **automatic harness evolution**, where agents iteratively improve their own harnesses through a search loop: analyze trajectories and failures → propose modifications → evaluate on benchmark tasks → repeat.

### Two Fundamental Concerns with Current Evaluation

The authors identify two critical flaws in existing evaluation protocols:

1. **Conflation with test-time scaling**: Harness evolution is an iterative search procedure that repeatedly evaluates and revises candidate harnesses using task feedback. It should therefore be compared against simple task-level search baselines (e.g., parallel sampling, sequential refinement) under *matched feedback and inference budgets* to determine whether gains arise from improved harness design or merely from additional search.

2. **Overfitting risk**: Because search and final evaluation share the same benchmark, reported gains may reflect adaptation to task-specific patterns rather than generalizable harness improvements.

### Key Definitions

Let $\pi_{\theta}$ denote the agent policy, $x$ a task from distribution $\mathcal{X}$, and $h$ the harness. Executing the agent yields a trajectory $y \sim \pi_{\theta}(\cdot | x; h)$. When a unit test case $g$ is available, an outcome function $R(y, g) \in \{0, 1\}$ returns 1 iff the trajectory solves the task. A summarization map $\Phi$ (realized by the same underlying model) summarizes an experience store into consumable form for downstream refinement or editing.

---

## Methodology

The authors formalize four methods under a unified compute budget $K$, specifying what each method updates and what feedback it observes:

### 1. Parallel Sampling (Test-Time Scaling)
Draws $K$ independent trajectories with a fixed harness $h$:

$$y_{1}, \dots, y_{K} \sim \pi_{\theta}(\cdot | x; h)$$

- **Without unit tests**: selection via self-judge $J$: $\hat{y} = \arg\max_{k \leq K} J(y_k)$
- **With unit tests**: $\hat{y} \in \{y_k : R(y_k, g) = 1\}$

### 2. Sequential Refinement (Test-Time Scaling)
Allocates budget to depth, conditioning each trajectory on the previous one:

$$y_{k} \sim \pi_{\theta}(\cdot | x, \Phi(y_{k-1}); h) \quad \text{(without unit tests)}$$

$$y_{k} \sim \pi_{\theta}\big(\cdot \mid x, \Phi(y_{k-1}, R(y_{k-1}, g)); h\big) \quad \text{(with unit tests)}$$

Returns $\hat{y} = y_K$ (or any accepted trajectory when tests are available).

### 3. Harness Evolution
Optimizes the harness across a task distribution rather than fitting to one task. Draws a batch $\{x^{(i)}\}_{i=1}^{n} \sim \mathcal{X}$, initializes experience store $\mathcal{C}_0$, and at each round $k$:

$$y_{k}^{(i,j)} \sim \pi_{\theta}\big(\cdot \mid x^{(i)}; h_k\big), \quad i = 1,\dots,n, \quad j = 1,\dots,m$$

Each round contributes evidence $e_k = \big(h_k, \{y_k^{(i,j)}\}_{i,j}\big)$ (plus outcomes when available). A meta agent $\mathcal{M}$ emits the next harness:

$$h_k = \mathcal{M}\big(\Phi(\mathcal{C}_{k-1})\big), \quad k = 2, \dots, K$$

The search optimizes: $h^{\star} = \arg\max_h \mathbb{E}_{x \sim \mathcal{X}}[R(y, g)]$

The authors instantiate harness evolution with **AHE** (Agentic Harness Engineering), with its explore agent disabled.

### 4. Harness Scaling (Newly Introduced)
A harness-level analogue of test-time scaling that adapts the harness for a *single evaluation instance*:

$$h_k = \mathcal{M}\big(x, \Phi(h_{k-1}, y_{k-1})\big), \quad y_k \sim \pi_{\theta}(\cdot | x; h_k)$$

(With unit tests, the update also conditions on $R(y_{k-1}, g)$.)

### Experimental Setup
- **Benchmark**: Terminal-Bench 2.1 (89 terminal tasks)
- **Models**: Claude Opus 4.6, GPT-5.4, GPT-5.4 mini
- **Budget**: $K = 5$, $m = 1$ rollout per task per harness
- **Max generation**: 128k tokens, high reasoning effort
- All results averaged over two independent runs

---

## Empirical Validation / Results

### Setting 1: Without Unit Test Cases

**Table 1: Results without unit test feedback (pass@1)**

| Method | Claude Opus 4.6 | GPT-5.4 | GPT-5.4 mini | Average |
|---|---|---|---|---|
| Direct sampling (initial harness) | 69.9 | 75.3 | 59.4 | 68.2 |
| **Test-Time Scaling** | | | | |
| Parallel Sampling | 74.7 | **79.2** | **62.9** | **72.3** |
| Sequential Refinement | 73.0 | 73.0 | 61.8 | 69.3 |
| **Automatic Harness Evolution** | | | | |
| Harness Evolution | 71.4 | 69.7 | 61.3 | 67.4 |
| Harness Scaling | **76.0** | 78.1 | 61.2 | 71.8 |

**Key findings:**
- Harness Evolution **fails to outperform** the direct sampling baseline on average (67.4 vs. 68.2)
- Most pronounced degradation on GPT-5.4: drops from 75.3 to 69.7
- Parallel Sampling is the most consistent approach (+4.1 average improvement)
- Self-generated feedback is noisy; sequential revision risks compounding early mistakes

### Setting 2: With Unit Test Cases

**Table 2: Results with unit test feedback**

| Method | Claude Opus 4.6 (pass@1 / pass@5) | GPT-5.4 (pass@1 / pass@5) | Average (pass@1 / pass@5) |
|---|---|---|---|
| Direct sampling | 69.9 / - | 75.9 / - | 72.9 / - |
| **Test-Time Scaling** | | | |
| Parallel Sampling | 84.8 / 84.8 | 87.1 / 87.1 | **86.0** / 86.0 |
| Sequential Refinement | 83.1 / **90.4** | 85.4 / **93.3** | 84.3 / **91.8** |
| **Automatic Harness Evolution** | | | |
| Harness Evolution | 73.0 / 83.2 | 78.6 / 89.3 | 75.8 / 86.2 |
| Harness Scaling | 83.1 / 89.9 | 82.0 / 88.8 | 82.6 / 89.3 |

**Key findings:**
- Neither Harness Evolution nor Harness Scaling outperforms simpler baselines on either metric
- On pass@1, Parallel Sampling achieves the best average (86.0); Harness Evolution barely improves over direct sampling
- On pass@5, Sequential Refinement is strongest (91.8 average)
- Gains from harness evolution only materialize when selecting among multiple trajectories—suggesting gains stem from multiple attempts, not improved harness design

### Setting 3: Generalization to Held-Out Tasks

**Table 3: Disjoint search and evaluation tasks (pass@1 on test set)**

| Method | Claude Opus 4.6 | GPT-5.4 | Average |
|---|---|---|---|
| Direct sampling (initial harness) | 63.3 | 72.1 | 67.7 |
| Harness Evolution | 64.5 (+1.2) | 72.1 (+0.0) | 68.3 (+0.6) |

**Key findings:**
- Evolved harness yields only marginal gains on held-out tasks (+0.6 average)
- Sharp contrast with improvements seen when evaluated on the same tasks used for optimization
- Revisions encode **task-specific shortcuts** rather than generalizable harness design principles

---

## Theoretical and Practical Implications

### Why Harness Evolution Underperforms

The authors analyze the meta agent's edits and identify several structural issues:

1. **Rational but marginal edits**: The meta agent makes well-motivated changes across prompt, middleware, and tool layers (e.g., adding behavioral rules, turn budget trackers, truncating oversized outputs, finalization gates). However, a "stable core of hard tasks" remains unaffected.

2. **Memorization rather than distillation**: Most edits "memorize fixes" (known bugs, file paths, command sequences) rather than distilling transferable strategies. This information is often rediscoverable by a competent agent within a single rollout—saving time on already-solvable tasks but rarely converting failures into successes.

3. **Context bloat**: Growing volumes of persistent prompt text introduce context bloat that can offset remaining gains.

4. **Task difficulty and harness sensitivity**: Terminal-Bench may not be very sensitive to harness design—a minimal setup (shell tool + basic prompt) suffices for most solvable tasks. Performance is bottlenecked by model reasoning rather than scaffolding.

### Recommendations for Future Benchmarks

The authors suggest harness evolution should be studied on benchmarks satisfying two conditions:
1. **Sufficient difficulty**: Tasks leave substantial headroom for improvement beyond current agent capabilities
2. **Harness sensitivity**: Performance depends heavily on specialized tools, skills, or workflows

---

## Conclusion

The paper's central message is that **current evidence for automatic harness evolution should be interpreted with caution**. The authors demonstrate that:

1. Harness evolution does not consistently outperform simple test-time scaling baselines under matched feedback and inference budgets
2. Its gains generalize poorly beyond the training distribution, indicating overfitting
3. The evaluation protocol itself (using the same benchmark for search and evaluation) conflates genuine harness improvements with test-time discovery effects

**Future directions** include:
- Developing fairer evaluation protocols that separate optimization feedback from final measurement
- Identifying benchmarks where harness design genuinely matters (difficult, harness-sensitive tasks)
- Exploring whether stronger priors or better meta-agents could make harness revision more effective

The code is available at https://github.com/rethinking-harness-evolution.

---

_Markdown view of https://picx.dev/p/nNsgCt, served by PicX — AI-generated visual whiteboard summaries of research papers._