# Beneath the Surface: What We Overlook When We Talk About Agent Evolution

> Benchmark scores conceal that evolved agents gain newly covered tasks while losing previously covered ones, making aggregate metrics insufficient for evaluating agent evolution.

- **Source:** [arXiv](https://arxiv.org/abs/2611.07624)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/LtEkQK
- **Whiteboard:** https://picx.dev/p/LtEkQK/image

## Summary

# Beneath the Surface: What We Overlook When We Talk About Agent Evolution

## Summary (Overview)

- This paper critically examines **agent harness evolution**—a form of recursive self-improvement (RSI) where model weights stay frozen while the surrounding system (prompts, skills, tools, orchestration) evolves—and asks whether higher benchmark scores actually translate into **reliable, efficient, and transferable** improvement.
- The authors evaluate four evolution methods (AHE, HyperAgents, A-Evolve, GEPA) across two harnesses (MiniSWE, Codex) and two task domains (general tasks via $\tau^3$-Banking, coding tasks via Terminal-Bench 2), finding that **all sixteen evolved agents gain previously uncovered tasks while losing previously covered ones**—a coverage redistribution pattern that aggregate scores conceal.
- Key findings reveal that: (1) reliability gains often come with higher deployment costs, (2) the highest-scoring agent does **not** yield the largest mean improvement in any of the four parent pools, (3) stricter admission rules can discard useful alternatives, and (4) transfer selectively preserves gains depending on the target benchmark and harness type.
- As a constructive follow-up, the authors introduce **EvoAudit**, a diagnostic workflow that translates empirical findings into coordinated operator revisions (select, analyze, mutate, gate), yielding **Audit-Guided Evolution (AGE)** which achieves the best performance across all four benchmark–harness settings.

## Introduction and Theoretical Foundation

### Background and Motivation

Recursive self-improvement (RSI) studies agents that iteratively revise their own designs based on experience. **Agent harness evolution** is a concrete instantiation: model weights remain fixed while the surrounding system—the harness—is persistently modified. These changes can target prompts, skills, tool interfaces, orchestration, or source code, going beyond static prompting and memory-based adaptation.

Recent frameworks report benchmark gains, but the authors argue that **a higher overall benchmark score does not imply a more useful agent**. Aggregate scores average over tasks and trials, so newly solved tasks can conceal lost ones, while deployment cost and out-of-distribution performance remain invisible.

### Research Questions

The paper organizes its investigation around three questions:

- **RQ1 (Phenomenology):** Does agent evolution yield reliable and efficient improvements, or does it suffer from task-coverage trade-offs and hidden costs?
- **RQ2 (Generalization):** Do gains achieved during evolution survive deployment in novel environments or tasks?
- **RQ3 (Mechanisms):** How do specific evolutionary operators contribute to the generation and preservation of beneficial modifications?

### Formal Definitions

An agent $A = (M, H)$ consists of a frozen model $M$ and a harness $H$. An evolution algorithm $E$ starts from a Seed $A_0 = (M, H_0)$ and runs for $G$ generations. The final Champion is $A_\star = (M, H_\star)$:

$$
\mathcal{C}_g = E(H_{<g}, \mathcal{Z}_{<g}, \mathcal{D}_{\text{evo}}; B_{\text{evo}}), \quad H_\star = Q\left(\{H_0\} \cup \bigcup_{g=1}^{G} \mathcal{C}_g\right)
$$

where $H_{<g}$ is the history of harness versions, $\mathcal{Z}_{<g}$ is evaluation evidence, $\mathcal{D}_{\text{evo}}$ is the evolution task set, and $B_{\text{evo}}$ is the search budget.

## Methodology

### Evolution Methods and Harnesses

Four evolution methods are evaluated using GPT-5.4:

| Method | Type | Description |
|--------|------|-------------|
| **AHE** | Implementation-centric | Exposes editable harness components, uses distilled execution evidence |
| **HyperAgents** | Implementation-centric | Makes both task agent and modification procedure editable |
| **A-Evolve** | Reflection-centric | Frames deployment-time adaptation as goal-directed optimization |
| **GEPA** | Reflection-centric | Combines trajectory reflection with Pareto selection |

Two harnesses are used:
- **MiniSWE 2.4.5**: Source-editable, permits inner-source changes
- **Codex 0.143.0**: Outer-layer-only, permits changes to prompts, skills, configuration, and external orchestration

### Benchmarks and Metrics

- **General tasks**: $\tau^3$-Banking (97 tasks)
- **Coding tasks**: Terminal-Bench 2 (89 tasks)

Key metrics include **Reliability (Avg@k)** and **Coverage (Pass@k)**:

$$
\text{Avg}@k = \frac{1}{Nk}\sum_{n=1}^{N}\sum_{i=1}^{k} Y_{ni}(A), \quad \text{Pass}@k = \frac{1}{N}\sum_{n=1}^{N} \mathbb{I}\left[\max_{1 \leq i \leq k} Y_{ni}(A) = 1\right]
$$

Cost metrics include evolution cost $C_{\text{evo}}$ and deployment cost:

$$
C_{\text{evo}} = \sum_{q \in \mathcal{L}_{\text{evo}}} c(q), \quad C_{\text{dep}}@k(A; \mathcal{D}) = k \cdot \bar{c}_{\text{dep}}^{(k)}(A; \mathcal{D})
$$

### Experimental Design

The study employs:
- **Repeated evaluations** (8 passes) comparing evolved vs. initial agents
- **Controlled agent expansion** (common-process Parent testing)
- **Component interventions** (isolating specific changes)
- **Admission replays** on frozen candidate pools
- **Split and cross-benchmark comparisons** for generalization assessment

## Empirical Validation / Results

### Phenomenology: Reliability and Coverage

**Table 1** (key results, eight passes):

| Benchmark | Harness | Variant | Avg@8 | ΔAvg | Pass@8 | ΔPass |
|-----------|---------|---------|-------|------|--------|-------|
| τ³-Banking | MiniSWE | Seed | 27.4 | - | 54.6 | - |
| τ³-Banking | MiniSWE | AHE | 32.9 | +5.4 | 66.0 | +11.3 |
| τ³-Banking | MiniSWE | HyperAgents | 32.6 | +5.2 | 57.7 | +3.1 |
| τ³-Banking | Codex | Seed | 30.3 | - | 56.7 | - |
| τ³-Banking | Codex | HyperAgents | 36.9 | +6.6 | 66.0 | +9.3 |
| Terminal-Bench 2 | Codex | AGE | 73.5 | +3.5 | 92.1 | +2.2 |

**Key findings:**
- Reliability gains in 11/16 lineages; coverage gains in 10/16
- **All 16 Champions gain and lose covered tasks** simultaneously
- One general-task case shows reliability rising 5.2 points while losing **ten previously covered tasks**
- 8 of 11 reliability-improving Champions cost more per trial than Seed

### Generalization

Coverage on the general-task target ($\tau^2$-bench) **falls for 7 of 8 Champions**, while coverage on the coding target (DeepSWE) **rises for 7 of 8 Champions**. Transfer changes the balance between reliability and coverage, so persistence must be assessed separately for each outcome and target.

### Mechanisms

**Select:** Original selection rules fall below the 25% random baseline on general tasks (17.5%, 8.8%) but exceed 55% on coding tasks—showing that current scores do not consistently identify improvement potential.

**Analyze & Mutate:**
- **15 of 16 Components outperform generic reminders** in reliability
- 6 of 8 Champions outperform either isolated Component alone
- In one coding case, **only the combination** improves over Seed

**Gate:** Constrained admission reduces best-Candidate retention from 15 to 6 pools, but effects are harness-dependent—improving size-normalized scores for outer-layer-only harness while degrading source-editable ones.

### Audit-Guided Evolution (AGE) Results

AGE improves reliability and coverage over A-Evolve in **all four settings**:

| Benchmark | Harness | ΔAvg (AGE) | ΔPass (AGE) | Tasks Lost |
|-----------|---------|------------|-------------|------------|
| τ³-Banking | MiniSWE | +6.6 | +11.3 | 2 |
| τ³-Banking | Codex | +7.9 | +10.3 | 5 |
| Terminal-Bench 2 | MiniSWE | +5.9 | +3.4 | 2 |
| Terminal-Bench 2 | Codex | +3.5 | +2.2 | 1 |

AGE loses fewer Seed-covered tasks than A-Evolve in three settings and ties in the fourth, while maintaining competitive or lower deployment costs.

## Theoretical and Practical Implications

### Feedback Narrowness

The paper's central theoretical insight is that **feedback can be narrower than a change's effects**. Harness changes affect many behaviors, while evaluation samples finite executions and compresses trade-offs. A prompt, skill, or execution routine can help tasks that supply feedback while disrupting behavior elsewhere. This explains task regressions, context-dependent change interactions, and selective transfer.

### Evidence–Decision Matching

The authors argue that **current performance, improvement potential, change compatibility, and retention are distinct objectives** that require different evidence:

- A high-scoring Parent need not yield the greatest subsequent gain
- A useful Component can interfere with other changes
- A conservative gate can discard useful alternatives

Evaluation should test the **desired consequence of each decision**, rather than reusing the same aggregate score throughout evolution.

### Operator Revisions (EvoAudit)

The paper translates findings into four operator revisions:

1. **Select**: Regression-aware tie-breaking—minimize Seed-relative regressions among highest-scoring Parents
2. **Analyze**: Behavior-grounded diagnosis using chronological tool calls with behavioral scores (0–10) distinguishing observed behavior from inference
3. **Mutate**: Mechanism-linked, dependency-closed changes requiring explicit evidence grounding, dependency closure, and behavior preservation
4. **Gate**: Score-first admission with cost-aware tie-breaking: accept if $s(C) > s(P)$ or ($s(C) = s(P) \wedge Q(C,P) \wedge c(C) < c(P)$)

Expert review broadly endorsed the revisions (mean 6–6.75 out of 8 across measures).

## Conclusion

This study demonstrates that **harness evolution redistributes observed task success**, bringing both gains and regressions across harnesses, task domains, and evolution paradigms. Practical improvement depends on productive Parents, compatible changes, preservation through search, and generalization at affordable cost.

The key takeaway is that **benchmark scores alone are insufficient** for evaluating agent evolution. The authors advocate for:
- **Preservation-aware evaluation** that tracks task-level gains and losses separately
- **Tested decision rules** that validate each operator's specific objective
- **Explicit applicability conditions** for when evolution gains transfer

EvoAudit demonstrates how diagnostic analysis can guide coordinated operator redesign, with AGE achieving the best performance across all settings. Future work should focus on evaluating evolution by the **useful behavior it creates and sustains**, beyond final benchmark scores, and on developing decision-specific feedback mechanisms that better represent the consequences of harness changes.

---

_Markdown view of https://picx.dev/p/LtEkQK, served by PicX — AI-generated visual whiteboard summaries of research papers._