EVO-BENCH: Can Language Models Improve Agent Harness?

Summary (Overview)

• Evo-Bench is the first benchmark designed to evaluate large language models' (LLMs) intrinsic harness-evolving capability—the ability to autonomously optimize their own executable agent harnesses through long-horizon, code-centric iterative refinement.
• The benchmark spans three domains (Search, Office, and General agent tasks) built from five established benchmarks (BrowseComp, HLE, GDPval, APEX-Agents, Claw-Eval), comprising a 160-task validation suite and a disjoint 448-task evaluation suite.
• A novel harness-guided construction framework is proposed, combining auxiliary-task evolution to generate diverse harnesses, then sensitivity-aware stratified splitting to ensure tasks genuinely respond to harness improvements while maintaining cross-suite generalization.
• Across 9 frontier and open-weight models, top evolvers (GPT-5.6 Sol, Claude Opus 4.8) achieve massive absolute gains of 16.6 and 16.1 points over the seed harness, closely approaching the human-engineered baseline of 47.5.
• Key findings reveal early saturation in evolutionary behavior, domain-dependent gains (strong in Search, weak in Office), and cross-policy transferability of evolved harnesses across diverse policy models.

---

Introduction and Theoretical Foundation

Background and Motivation

Recent breakthroughs in LLMs have elevated agentic systems to execute complex, long-horizon tasks at system-level scale. These gains arise not only from improved base models but also from carefully engineered agent harnesses that structure, coordinate, and constrain agent behavior (e.g., Claude Code, Codex).

The paper identifies harness evolution as "generally considered the first step toward achieving self-improvement." The pivotal question: Can language models truly improve agent harnesses, and how can we systematically benchmark this evolutionary capability?

Three Key Challenges

1. Harness Sensitivity: Benchmark task performance must be responsive to harness improvements rather than dominated by base model strength.
2. Cross-Split Generalization: Validation and evaluation splits must exhibit aligned responsiveness to prevent task-specific overfitting.
3. Long-Horizon Evolution: Models must sustain multi-round iterative refinement—diagnosing failures, formulating hypotheses, and progressively updating code.

Theoretical Framework

The task formulation formalizes the evolution process. At iteration $t$, the policy agent is $A_t^{\text{task}} = (\pi, H_t)$, where $\pi$ is the fixed policy model and $H_t$ is the editable policy harness. The evolver operates through a separate, fixed evolve harness $\mathcal{H}_{\text{evo}}$, constrained by a resource budget $\mathbf{b} = (b^{\text{iter}}, b^{\text{time}}, b^{\text{steps}})$.

---

Methodology

Task Formulation

Given cumulative validation-side evidence $\mathcal{E}_t^{\text{val}} = ((H_i, j_i^{\text{val}}, O_i^{\text{val}}))_{i<t}$ (where $j_i^{\text{val}}$ is the aggregate validation score and $O_i^{\text{val}}$ contains task-level outcomes, trajectories, and diagnostics), the evolver:

1. Inspects accumulated validation evidence
2. Edits the current harness $H_t$
3. Requests formal evaluation on the visible validation suite

The final revision $H_T$ is frozen and evaluated on a disjoint held-out evaluation suite.

Two-Stage Harness-Guided Benchmark Construction

Stage 1: Auxiliary Harness Generation

• Collect 320 auxiliary tasks (128 search, 128 office, 64 general) from sources disjoint from the five main benchmarks
• Run four independent evolution experiments with frontier models (GLM-5.2, Claude-Opus-4.8, Claude-Sonnet-5, GPT-5.6-Sol)
• Collect 73 harness variants, select 12 representative harnesses $\mathcal{H}_{\text{aux}} = \{h_1, \ldots, h_K\}$ via diversity-aware selection

Stage 2: Harness-Guided Task Selection

Evaluate 2,329 candidate tasks under the 12 selected harnesses. Define two key metrics:

$$\operatorname{Sens}(x) = \operatorname{corr}\left(\{m_h(x)\}_{h \in \mathcal{H}_{\text{aux}}}, \{Q_h^{(-x)}\}_{h \in \mathcal{H}_{\text{aux}}}\right)$$ $$\operatorname{Perf}(x) = \frac{1}{|\mathcal{H}_{\text{aux}}|} \sum_{h \in \mathcal{H}_{\text{aux}}} m_h(x)$$

where $Q_h^{(-x)}$ is the leave-one-task-out average performance of harness $h$.

Selection procedure:

• Remove tasks with $\operatorname{Sens}(x) \leq 0$
• Partition remaining tasks into difficulty strata by $1 - \operatorname{Perf}(x)$
• Select highest $\operatorname{Sens}(x)$ tasks within each stratum
• Randomly split within strata into validation (160 tasks) and evaluation (448 tasks) suites

Evaluation Metrics

Overall Score (final generalization performance):

$$\operatorname{Overall}(E) := \mathcal{S}(\pi, H_T; \mathcal{D}_{\text{eval}})$$

Anytime Validation Score (evolutionary progress):

$$S_t^* = \max_{i \leq t} \mathcal{S}(\pi, H_i; \mathcal{D}_{\text{val}})$$ $$\mathrm{AnytimeVal}(E) := \frac{1}{b^{\text{iter}}} \sum_{t=1}^{b^{\text{iter}}} S_t^*$$

Experimental Setup

• Fixed policy model: DeepSeek-V4-Flash (max reasoning effort, temperature 1.0, 256K context)
• Budget: 20 iterations, 1,000 evolver steps, 48 hours
• Policy rollout cap: 300 steps, one hour
• Baselines: CodeAct seed harness (starting point), Artificial harness (composite of human-engineered frameworks: MiroFlow for search, Stirrup for office, Claw-Eval for general)

---

Empirical Validation / Results

Main Leaderboard Results

Key Findings

1. Widespread gains: All models achieve positive overall improvements, with top models reaching +16.6 points over the seed harness
2. Domain-dependent gains:
   • Search: Massive gains (+34.8 for Claude Opus 4.8), nearly matching the Artificial harness
   • Office: Marginal improvements or slight regressions, failing to match the Artificial baseline
   • General: Top evolvers (GPT-5.6-Sol, Qwen3.7-Max) strictly surpass the Artificial harness
3. Early saturation: Models rapidly discover high-quality structures then introduce detrimental modifications
4. Budget behavior: Two distinct patterns—exhaustive exploration (GPT-5.6-Sol, Kimi-K2.7-Code) vs. premature termination (Qwen3.7-Max, DeepSeek-V4-Pro)

Cost Analysis

The Pareto frontier reveals a steep logarithmic trade-off between cost and capability:

• GPT-5.6-Sol: Highest score but exceeds $500 per run
• GLM-5.2, Qwen3.7-Max: Efficient knee of the curve, <$40 for formidable performance
• DeepSeek-V4-Pro: <$1 for functional improvements

Ablation: Policy Model Transferability

Evolved harnesses consistently achieve massive improvements across all policy models, demonstrating genuine synthesis of generalizable reasoning structures rather than overfitting to a single model's flaws.

---

Theoretical and Practical Implications

Theoretical Implications

1. Harness evolution as measurable capability: Evo-Bench provides the first rigorous framework to isolate and quantify intrinsic harness-evolving capability, disentangling it from base model strength and task-specific overfitting.

2. Sensitivity-aware construction: The harness-guided construction framework (using Pearson correlation between task performance and harness quality) offers a principled method for selecting tasks that reliably reflect harness improvements.

3. Transferable reasoning structures: Evolved harnesses act as generalizable reasoning structures that boost diverse policy models, suggesting harnesses capture domain-general problem-solving patterns.

Practical Implications

1. Cost-efficiency insights: The steep cost-performance trade-off suggests that mid-tier models (GLM-5.2, Qwen3.7-Max) offer the best cost-effectiveness for harness evolution.

2. Domain-specific challenges: Office tasks requiring highly specific processing workflows remain a frontier challenge, while Search tasks are more amenable to autonomous optimization.

3. Architectural guidance: Case studies reveal that successful evolvers build hierarchical routers with domain-specific prompts/tools, implement web-search/fetch tools with webpage cleaning, and add failure recovery mechanisms—providing a blueprint for future harness design.

---

Conclusion

Evo-Bench is the first benchmark designed to evaluate language models' intrinsic capacity to autonomously improve agent harnesses. Key contributions:

1. Novel benchmark: 160-task validation suite and 448-task evaluation suite across Search, Office, and General agent domains
2. Harness-guided construction framework: Two-stage process using auxiliary-task evolution followed by sensitivity-aware stratified splitting
3. Systematic scientific account: Evaluation across 9 frontier and open-weight models, revealing substantial gains (+16.6 points), temporal anomalies (early saturation), and robust cross-policy transferability

Future Work

• Integrate coding tasks and challenging scientific research tasks
• Extend compatibility across diverse agent frameworks
• Evaluate a wider range of models
• Maintain Evo-Bench as a living benchmark for measuring AI self-evolution

The authors hope Evo-Bench serves as a foundational testbed to catalyze research on self-evolving agent architectures and execution-grounded reasoning.