# EVO-BENCH: CAN LANGUAGE MODELS IMPROVE AGENT HARNESS?

> Evo-Bench, the first benchmark for agent harness evolution, shows top LLMs gain up to 16.6 points by autonomously refining their own executable code.

- **Source:** [arXiv](https://arxiv.org/abs/2608.09096)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/4gfsEG
- **Whiteboard:** https://picx.dev/p/4gfsEG/image

## Summary

# EVO-BENCH: Can Language Models Improve Agent Harness?

## Summary (Overview)

- **Evo-Bench** is the first benchmark designed to evaluate large language models' (LLMs) intrinsic **harness-evolving capability**—the ability to autonomously optimize their own executable agent harnesses through long-horizon, code-centric iterative refinement.
- The benchmark spans **three domains** (Search, Office, and General agent tasks) built from five established benchmarks (BrowseComp, HLE, GDPval, APEX-Agents, Claw-Eval), comprising a 160-task validation suite and a disjoint 448-task evaluation suite.
- A novel **harness-guided construction framework** is proposed, combining auxiliary-task evolution to generate diverse harnesses, then sensitivity-aware stratified splitting to ensure tasks genuinely respond to harness improvements while maintaining cross-suite generalization.
- Across 9 frontier and open-weight models, top evolvers (GPT-5.6 Sol, Claude Opus 4.8) achieve **massive absolute gains of 16.6 and 16.1 points** over the seed harness, closely approaching the human-engineered baseline of 47.5.
- Key findings reveal **early saturation** in evolutionary behavior, **domain-dependent gains** (strong in Search, weak in Office), and **cross-policy transferability** of evolved harnesses across diverse policy models.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Recent breakthroughs in LLMs have elevated agentic systems to execute complex, long-horizon tasks at system-level scale. These gains arise not only from improved base models but also from **carefully engineered agent harnesses** that structure, coordinate, and constrain agent behavior (e.g., Claude Code, Codex).

The paper identifies **harness evolution** as "generally considered the first step toward achieving self-improvement." The pivotal question: *Can language models truly improve agent harnesses, and how can we systematically benchmark this evolutionary capability?*

### Three Key Challenges

1. **Harness Sensitivity**: Benchmark task performance must be responsive to harness improvements rather than dominated by base model strength.
2. **Cross-Split Generalization**: Validation and evaluation splits must exhibit aligned responsiveness to prevent task-specific overfitting.
3. **Long-Horizon Evolution**: Models must sustain multi-round iterative refinement—diagnosing failures, formulating hypotheses, and progressively updating code.

### Theoretical Framework

The task formulation formalizes the evolution process. At iteration $t$, the policy agent is $A_t^{\text{task}} = (\pi, H_t)$, where $\pi$ is the fixed policy model and $H_t$ is the editable policy harness. The evolver operates through a separate, fixed evolve harness $\mathcal{H}_{\text{evo}}$, constrained by a resource budget $\mathbf{b} = (b^{\text{iter}}, b^{\text{time}}, b^{\text{steps}})$.

---

## Methodology

### Task Formulation

Given cumulative validation-side evidence $\mathcal{E}_t^{\text{val}} = ((H_i, j_i^{\text{val}}, O_i^{\text{val}}))_{i<t}$ (where $j_i^{\text{val}}$ is the aggregate validation score and $O_i^{\text{val}}$ contains task-level outcomes, trajectories, and diagnostics), the evolver:

1. Inspects accumulated validation evidence
2. Edits the current harness $H_t$
3. Requests formal evaluation on the visible validation suite

The final revision $H_T$ is frozen and evaluated on a **disjoint held-out evaluation suite**.

### Two-Stage Harness-Guided Benchmark Construction

#### Stage 1: Auxiliary Harness Generation

- Collect 320 auxiliary tasks (128 search, 128 office, 64 general) from sources disjoint from the five main benchmarks
- Run four independent evolution experiments with frontier models (GLM-5.2, Claude-Opus-4.8, Claude-Sonnet-5, GPT-5.6-Sol)
- Collect 73 harness variants, select 12 representative harnesses $\mathcal{H}_{\text{aux}} = \{h_1, \ldots, h_K\}$ via diversity-aware selection

#### Stage 2: Harness-Guided Task Selection

Evaluate 2,329 candidate tasks under the 12 selected harnesses. Define two key metrics:

$$\operatorname{Sens}(x) = \operatorname{corr}\left(\{m_h(x)\}_{h \in \mathcal{H}_{\text{aux}}}, \{Q_h^{(-x)}\}_{h \in \mathcal{H}_{\text{aux}}}\right)$$

$$\operatorname{Perf}(x) = \frac{1}{|\mathcal{H}_{\text{aux}}|} \sum_{h \in \mathcal{H}_{\text{aux}}} m_h(x)$$

where $Q_h^{(-x)}$ is the leave-one-task-out average performance of harness $h$.

**Selection procedure**:
- Remove tasks with $\operatorname{Sens}(x) \leq 0$
- Partition remaining tasks into difficulty strata by $1 - \operatorname{Perf}(x)$
- Select highest $\operatorname{Sens}(x)$ tasks within each stratum
- Randomly split within strata into validation (160 tasks) and evaluation (448 tasks) suites

### Evaluation Metrics

**Overall Score** (final generalization performance):

$$\operatorname{Overall}(E) := \mathcal{S}(\pi, H_T; \mathcal{D}_{\text{eval}})$$

**Anytime Validation Score** (evolutionary progress):

$$S_t^* = \max_{i \leq t} \mathcal{S}(\pi, H_i; \mathcal{D}_{\text{val}})$$

$$\mathrm{AnytimeVal}(E) := \frac{1}{b^{\text{iter}}} \sum_{t=1}^{b^{\text{iter}}} S_t^*$$

### Experimental Setup

- **Fixed policy model**: DeepSeek-V4-Flash (max reasoning effort, temperature 1.0, 256K context)
- **Budget**: 20 iterations, 1,000 evolver steps, 48 hours
- **Policy rollout cap**: 300 steps, one hour
- **Baselines**: CodeAct seed harness (starting point), Artificial harness (composite of human-engineered frameworks: MiroFlow for search, Stirrup for office, Claw-Eval for general)

---

## Empirical Validation / Results

### Main Leaderboard Results

| Rank | Model | Search | Office | General | Overall | AnytimeVal |
|------|-------|--------|--------|---------|---------|------------|
| 1 | GPT-5.6 Sol | 44.5 (+32.8) | 41.6 (+3.2) | 59.4 (+11.0) | **46.3 (+16.6)** | 50.1 |
| 2 | Claude Opus 4.8 | 46.5 (+34.8) | 39.7 (+1.3) | 56.3 (+7.9) | 45.8 (+16.1) | 51.4 |
| 3 | GLM-5.2 | 45.4 (+33.7) | 39.2 (+0.8) | 48.4 (±0.0) | 43.5 (+13.8) | 51.0 |
| 4 | Qwen3.7-Max | 36.3 (+24.6) | 37.8 (−0.6) | 59.4 (+11.0) | 41.5 (+11.8) | 49.3 |
| 5 | Minimax-M3 | 33.6 (+21.9) | 41.7 (+3.3) | 56.3 (+7.9) | 41.4 (+11.7) | 49.0 |
| 6 | Qwen3.6-27b | 34.8 (+23.1) | 38.8 (+0.4) | 50.0 (+1.6) | 39.4 (+9.7) | 46.9 |
| 7 | Deepseek V4 Pro | 34.4 (+22.7) | 39.1 (+0.7) | 48.4 (±0.0) | 39.1 (+9.4) | 45.4 |
| 8 | Kimi K2.7 Code | 34.5 (+22.8) | 38.1 (−0.3) | 48.4 (±0.0) | 38.7 (+9.0) | 43.4 |
| 9 | Gemma-4-31B | 24.2 (+12.5) | 40.4 (+2.0) | 50.0 (+1.6) | 35.9 (+6.2) | 36.2 |
| — | CodeAct (baseline) | 11.7 | 38.4 | 48.4 | 29.7 | — |
| — | Artificial Harness | 46.7 | 43.9 | 56.3 | 47.5 | — |

### Key Findings

1. **Widespread gains**: All models achieve positive overall improvements, with top models reaching +16.6 points over the seed harness
2. **Domain-dependent gains**: 
   - **Search**: Massive gains (+34.8 for Claude Opus 4.8), nearly matching the Artificial harness
   - **Office**: Marginal improvements or slight regressions, failing to match the Artificial baseline
   - **General**: Top evolvers (GPT-5.6-Sol, Qwen3.7-Max) **strictly surpass** the Artificial harness
3. **Early saturation**: Models rapidly discover high-quality structures then introduce detrimental modifications
4. **Budget behavior**: Two distinct patterns—exhaustive exploration (GPT-5.6-Sol, Kimi-K2.7-Code) vs. premature termination (Qwen3.7-Max, DeepSeek-V4-Pro)

### Cost Analysis

The Pareto frontier reveals a steep logarithmic trade-off between cost and capability:
- **GPT-5.6-Sol**: Highest score but exceeds $500 per run
- **GLM-5.2, Qwen3.7-Max**: Efficient knee of the curve, <$40 for formidable performance
- **DeepSeek-V4-Pro**: <$1 for functional improvements

### Ablation: Policy Model Transferability

| Policy Model | Baseline | Qwen3.7-Max Evolved | GLM-5.2 Evolved |
|-------------|----------|---------------------|-----------------|
| Qwen3.6-35B-A3B | 13.9 | 27.9 | 29.2 |
| DeepSeek-V4-Flash | 29.7 | 41.5 | 43.5 |
| GLM-5.2 | 38.0 | 42.7 | **48.4** |

Evolved harnesses consistently achieve massive improvements across all policy models, demonstrating genuine synthesis of generalizable reasoning structures rather than overfitting to a single model's flaws.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Harness evolution as measurable capability**: Evo-Bench provides the first rigorous framework to isolate and quantify intrinsic harness-evolving capability, disentangling it from base model strength and task-specific overfitting.

2. **Sensitivity-aware construction**: The harness-guided construction framework (using Pearson correlation between task performance and harness quality) offers a principled method for selecting tasks that reliably reflect harness improvements.

3. **Transferable reasoning structures**: Evolved harnesses act as generalizable reasoning structures that boost diverse policy models, suggesting harnesses capture domain-general problem-solving patterns.

### Practical Implications

1. **Cost-efficiency insights**: The steep cost-performance trade-off suggests that mid-tier models (GLM-5.2, Qwen3.7-Max) offer the best cost-effectiveness for harness evolution.

2. **Domain-specific challenges**: Office tasks requiring highly specific processing workflows remain a frontier challenge, while Search tasks are more amenable to autonomous optimization.

3. **Architectural guidance**: Case studies reveal that successful evolvers build hierarchical routers with domain-specific prompts/tools, implement web-search/fetch tools with webpage cleaning, and add failure recovery mechanisms—providing a blueprint for future harness design.

---

## Conclusion

Evo-Bench is the **first benchmark** designed to evaluate language models' intrinsic capacity to autonomously improve agent harnesses. Key contributions:

1. **Novel benchmark**: 160-task validation suite and 448-task evaluation suite across Search, Office, and General agent domains
2. **Harness-guided construction framework**: Two-stage process using auxiliary-task evolution followed by sensitivity-aware stratified splitting
3. **Systematic scientific account**: Evaluation across 9 frontier and open-weight models, revealing substantial gains (+16.6 points), temporal anomalies (early saturation), and robust cross-policy transferability

### Future Work

- Integrate coding tasks and challenging scientific research tasks
- Extend compatibility across diverse agent frameworks
- Evaluate a wider range of models
- Maintain Evo-Bench as a living benchmark for measuring AI self-evolution

The authors hope Evo-Bench serves as a foundational testbed to catalyze research on self-evolving agent architectures and execution-grounded reasoning.

---

_Markdown view of https://picx.dev/p/4gfsEG, served by PicX — AI-generated visual whiteboard summaries of research papers._