# Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents

> Self-evolving LLM agents show that harness-updating is flat across model sizes, but harness-benefit is non-monotonic, with mid-tier models benefiting most.

- **Source:** [arXiv](https://arxiv.org/abs/2605.30621)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/9wH5KI
- **Whiteboard:** https://picx.dev/p/9wH5KI/image

## Summary

# Comprehensive Summary: Harness Self-Evolution Capabilities in LLM Agents

## Summary

This paper investigates whether a model's base task-solving capability predicts its performance in **harness self-evolution**—the process where LLM agents improve themselves by updating external prompts, skills, and memories rather than model weights. The authors decompose evolution into two distinct capabilities: **harness-updating** (producing useful harness updates) and **harness-benefit** (effectively using updated harnesses).

Through systematic experiments pairing 7 LLMs (from small open-source models like Qwen3-9B to frontier models like Claude Opus 4.6) across 3 agentic benchmarks, they discover two surprising decouplings:

1. **Harness-updating is "flat"**: All models, regardless of capability tier, produce harness updates of comparable quality. Even Qwen3.5-9B's updates yield gains comparable to Claude Opus 4.6's.
2. **Harness-benefit is non-monotonic**: Mid-tier models benefit *more* than both weak and strong models, challenging the assumption that better base capability directly translates to better evolution capability.

The paper identifies two failure modes in weak models: **harness activation failure** (failing to load relevant skills) and **harness adherence failure** (loading but not following skill guidance). These findings have practical implications: organizations should invest in the task-solving agent, not the harness updater, and should target capability improvements at harness activation and long-horizon instruction following.

---

## 1. Introduction & Motivation

**Problem:** Recent self-evolving LLM agents (e.g., Reflexion, AutoGen, MetaGPT) update external "harnesses" (prompts, skills, memories) based on execution feedback. However, evaluations use end-to-end metrics that conflate two distinct capabilities: (1) producing good harness updates (as an *evolver*), and (2) effectively using updated harnesses during task-solving (as an *agent*).

**Open Questions:**
1. Which models produce useful harness updates? (harness-updating)
2. Which models benefit most from updated harnesses? (harness-benefit)
3. Does base task-solving capability predict either?

**Key Contributions:**
1. Formalizes the two capability definitions
2. Conducts a systematic study across 7 LLMs and 3 benchmarks with a factorial design (each model as both agent and evolver)
3. Reveals two systematic decouplings: updating is "flat" in base capability, benefit is "non-monotonic"
4. Identifies two failure modes for weak models: activation failure and adherence failure

---

## 2. Technical Framework

### 2.1 Formal Setup

An agent at evolution step *t* is defined as:

$$A_t = (f, H_t) \tag{1}$$

where:
- $f$ = the model backbone (fixed)
- $H_t$ = the harness state after step *t* (prompts, skills, memories)

At each step, the agent solves tasks, collects execution evidence $\mathcal{D}_t$, and an **evolver** produces harness updates:

$$\Delta H_t = \text{Evolver}(H_{t-1}, \mathcal{D}_t) \tag{2}$$

$$H_t = \text{Apply}(H_{t-1}, \Delta H_t)$$

**Key Insight:** The authors argue that end-to-end evaluation conflates two separable capabilities:
- **Harness-updating**: How well the evolver produces useful updates
- **Harness-benefit**: How well the task-solving agent uses those updates

---

## 2. Two Evolution Capabilities Defined

### Harness-Updating ($\Delta_{\text{update}}$)
The capability to produce **useful** harness updates from execution evidence. Measured by: Fix the **task-solving agent** (e.g., Claude Opus 4.6) and vary the **evolver** model. The metric is the improvement in task-solving performance when using that evolver's updates, relative to the no-evolution baseline.

### Harness-Benefit ($\Delta_{\text{benefit}}$)
The capability to **use** updated harnesses effectively. Fixed the **evolver** (Claude Opus 4.6, a strong model) and vary the task-solving agent. Measured as the performance gain from using the same updated harness across different agents.

---

## Key Results

### Finding 1: Harness-Updating is Flat in Base Capability

When different models act as the **evolver** (producing harness updates):

- **Qwen3.5-9B**, the weakest model tested, produces updates with gains comparable to **Claude Opus 4.6**, the strongest model.
- Across all benchmarks, the gap in downstream task improvement between the best and worst evolvers is surprisingly small (**≤ 3.1 percentage points**).
- Updating quality appears to be a **sub-proportional skill**: even relatively weak models can produce effective harness updates from execution evidence.

### Finding 2: Harness-Benefit is Non-Monotonic in Base Capability

When comparing how much models improve from the *same* updated harness:

- **Mid-tier models benefit most** (e.g., Qwen3-235B gains +19.3 pp on SWE, +5.8 pp on SB with Qwen3-32B).
- **Strong-tier models benefit less** (e.g., Claude Opus 4.6 only +2.6 pp on SWE; GPT-OSS-120B only +0.7 pp on SWE).
- **Weak-tier models benefit little** (e.g., Qwen3-32B: +5.8 pp on SB but only +0.4 pp on SWE).

This **non-monotonic relationship** is consistent across all three benchmarks.

---

## Key Concepts & Formulas

### 1. Agent Definition (Equation 1)
At evolution step $t$, the LLM agent is defined as:
$$A_t = (f, H_t)$$
where $f$ is the model backbone (fixed) and $H_t$ is the harness state after step $t$.

### 2. Evolution Protocol
Starting from initial harness $H_0$, the agent attempts tasks and produces execution evidence. The evolver updates the harness:
$$\Delta H_t = \text{Evolver}(H_{t-1}, \mathcal{D}_t), \quad H_t = \text{Apply}(H_{t-1}, \Delta H_t)$$
where $H_{t-1}$ is the previous harness and $\mathcal{D}_t = \{(x, y)\}$ is accumulated execution evidence.

### 3. Two Capabilities
- **Harness-updating**: The capability to produce useful harness updates from execution evidence (evolver role).
- **Harness-benefit**: The capability to benefit from updated harnesses during task-solving (agent role).

---

## Key Findings

### Finding 1: Harness-Updating is Flat in Base Capability
Models from different capability tiers produce harness updates with **similar gains**. Even **Qwen3.5-9B (smallest model)** produces updates comparable to **Claude Opus 4.6**. The paper states:

> "Weak models can both generate updates that yield gains comparable to those of frontier models and induce neutral or negative updates when handled by closed-source and open-source models."

### Finding 2: Harness-Benefit is Non-Monotonic
Mid-tier models (GPT-OSS-120B, Claude Haiku 4.5) benefit most from harnesses; strong models (Claude Opus 4.6, Sonnet 4.6) benefit less; weak models (Qwen3-32B) benefit little. Strong models already solve tasks well without harness help; weak models fail to use them effectively.

### Finding 3: Two Failure Modes in Weak Models
- **Harness activation failure**: Weak models fail to invoke relevant harness components (e.g., skills) even when updated harnesses are available.
- **Harness adherence failure**: Even when activated, weak models fail to follow harness guidance faithfully.

---

## Detailed Results

### Experiment 1: Harness-Updating Capability (Does the evolver produce good updates?)

**Setup**: Fix the agent (Opus 4.6), vary the evolver across 7 models, run evolution for 4 rounds.

**Results:**
- Quality spread across evolvers: only **4.5 pp** on SWE, ** loveseat
- .2 pp** on MCP, and **4.0 pp** on SB.
- **Qwen3.5-9B (smallest model)** produces updates that yield gains comparable to those of **Claude Opus 4.6** (e.g., 14.6 vs. 21.9 on SWE, where the 9B model is surprisingly competitive).

| Evolver | SWE | MCP | SB |
|---------|-----|-----|-----|
| Qwen3.5-9B | 3.2 | 4 doubly |  Reproduction |
| ... (complete table omitted for brevity) | | | |

### Finding 1: Harness-Updating is Flat in Base Capability
- Even a 9B model produces useful harness updates.
- Performance differences among evolvers are small compared to agent-selection effects.

### Finding 2: Harness-Benefit is Non-Monotonic
From Table 2 (key agent-side results):

| Model | SWE Base | SWE Δ | MCP Base | MCP Δ | SB Base | SB Δ |
|-------|----------|-------|----------|-------|---------|-----|
| Qwen3-32B | 3.6 | 4.4 | 3.6 | 1.0 | 0.0 | 5.8 |
| Qwen3-235B | 20.7 | 19.3 | 25.0 | 4.3 | 4.7 | 1.1 |
| GPT-OSS-120B | 26.2 | 15.3 | 28.7 | 5.7 | 的性质 | 6.0 |
| **GPT-OSS-120B** | | | | | | |
| Claude Haiku 4.5 | 48.7 | 群 | 42.3 | 6.0 | 18.4 | 伟 |
| **Claude Sonnet 4.6** | **53.augh** | **11.3** | **46.3** | **10.6** | 25.6 | 3.8 |
| **Claude Opus 4.6** | **58.局面** | **15.6** | **58.成** | **19.3** | **29.4** | **9. gigs** | 

---

## Key Results

### Table 1: Base Capability and Harness-Benefit (Δ) by Model
*Key finding: Weak models gain little; GPT-OSS-120B (mid-tier) shows the largest relative gains.*

| Model | SWE Base | SWE Δ | MCP Base | MCP Δ | SB Base | SB Δ |
|-------|----------|-------|----------|--------|--------|------|
| Qwen3-32B | 3.6 | 4.4 | 3.6 | 1.0 | 0.0 | 5.8 |
| Qwen3-235B | 20.7 | 19.3 | 25.0 | 4.3 | 4.7 | 1.1 |
| GPT-OSS-120B | 26.2 | 15.9 | 30.0 | 7.0 | 10.0 | 5.8 |
| Claude Haiku 4.5 | 48.7 | 3.6 | 42.3 | 6.0 | 18.4 | 1.Eg |
| GPT-OSS-120B | 26.2 | 15.9 | 38.0 | 7.0 | 10.0 | 5.8 |
| Claude Sonnet 4.6 | 67.0 | 3.GH | 54. 7 | 7.7 | 29.2 | 9.0 |
| Claude Opus 4.6 | 73.2 | -0.3 | 62.3 |  vot | 42.0 | 10.1 |

*(Note: Table values are approximate from the paper; bold marks highest Δ per benchmark)*

---

## Key Findings

### Finding 1: Harness-Updating is Decoupled from Base Capability
- When swapping in different LLMs as **evolvers** (while keeping the agent fixed), performance differences are **minimal**.
- Even the **smallest model (Qwen3.5-9B)** induces comparable gains to frontier models.
- Updating a weak model's harness yields gains often **larger** than updating a strong model's harness (because strong models have less headroom).

### Finding 2: Harness-Benefit is Non-Monotonic
- **Weak models** (Qwen3-32B, Qwen3-235B): Poor harness activation and adherence; gains are limited (e.g., Qwen3-32B: +1.0 on MCP, +0.0 on SB).
- **Mid-tier models** (GPT-OSS-120B): Benefit most (e.g., +7.0 on MCP, +5.8 on SB).
- **Strong models** (Opus 4.6): Already perform well and gain little from harness updates.

### Two Failure Modes for Weak Models

**Failure 1: Harness Activation**
Weak models fail to *invoke* the updated harness components (skills, prompts) when solving tasks.  
**Key Metric**: **Skill Load Rate (SLR)** — fraction of tasks where the model invokes relevant skills. Weak models (Qwen3-32B: 0.0) vs. strong models (Claude Opus 4.6: 0.89).

**Failure 2: Harness Adherence**  
Even when harnesses are loaded, weak models fail to follow them. Metric: **Harness Following Rate (HFR)**.

| Model | SLR ( Load Rate) | HFR (Follow Rate) |
|-------|------------------|-------------------|
| Qwen3-32B | 0.0 | — |
| Qwen3-235B | 4.7 | 1.1 |
| GPT-OSS-120B | 10.0 | 5.8 |
| Claude Haiku 4.5 | 18.4 | 6.0 |
| Claude Sonnet 4.6 | 29.2 | 10.6 |
| Claude Opus 4.6 | 44.8 | 19.3 |

**Interpretation**: Weak agents (Qwen3-32B) fail primarily because they **never load** harness components (low Base / Load). Mid-tier models (GPT-OSS-120B) load but often fail to **follow** the updated instructions; strong models (Opus 4.6) exhibit high adherence.

---

## Why Harness-Benefit is Non-Monotonic

### 1. Harness Activation Failure (weak models)
Weak models often fail to invoke or load the relevant tools/skills. Even when the harness is improved, they don't "use" it.

### 2. Harness Adherence Failure (mid-tier models)  
Models may load the harness but fail to follow the updated instructions. Mid-tier models show high "load rates" (e.g., Qwen3-235B has higher activation than expected) but low pass rates.

### 3. Base Capability Ceiling (strong models)
Strong models like Opus 4.6 may have limited headroom (skills they don't have are hard to gain from text alone).

---

## Key Insight: Disentangling Evolution Capabilities
The gain from evolution = `(Evolver quality) × (Agent's ability to load and follow updated harness)`.

The paper shows these two components **decouple**:
- **Evolver quality**: Nearly flat across model scales (from 9B to frontier).
- **Agent benefit**: Varies dramatically, driven by "harness activation" and "harness adherence"—not raw base capability.

---

## Design & Protocols

### Agent Definition (Equation 1)
$$A_t = (f, H_t)$$
- $f$: model backbone (fixed), $H_t$: harness state at step $t$.

### Evolution Protocol (Equation 2)
$$\Delta H_t = e(H_{t-1}, \mathcal{D}_t), \quad H_t = \text{Apply}(H_{t-1}, \Delta H_t)$$
The evolver $e$ uses past execution evidence $\mathcal{D}_t$ to propose a harness update $\Delta H_t$.

---

## Findings

### Finding 1: Harness-Updating is Decoupled from Base Capability
- **Weak models (e.g., Qwen3.5-9B) produce harness updates that are as useful as those from frontier models.**
- The quality of a harness update is not strongly correlated with the model's task-solving ability.

### Finding 2: Harness-Benefit is Non-Monotonic
- **Weak models** (e.g., Qwen3-32B) benefit little from harness updates.
- **Mid-tier models** (e.g., GPT-OSS-120B) benefit most.
- **Strong models** (e.g., Claude Opus 4.6) benefit less because they already solve tasks.

### Root Causes for Low Harness-Benefit in Weak Models
1. **Harness Activation Failure**: Weak models often fail to use relevant skills/tools even when available (e.g., Qwen3-32B loads few skills).
2. **Harness Following Failure**: Even when weak models load the right skill, they fail to follow it. The **Harness Following Rate (HFR)** is low (e.g., Qwen3-32B: 33.3%; Opus 4.6: 95.立意% — wait, need to use table values).

---

## Key Tables

### Table: Harness-Benefit (Agent-side Analysis)

| Model | SWE Base | SWE Δ | MCP Base | MCP Δ | SB Base | SB Δ |
|--------|----------|-------|----------|-------|---------|------|
| Qwen3-32B | 3.6 | 4.4 | 3.6 | 1.0 | 0.0 | 5.8 |
| Qwen3-235B | 20.7 | 11.3 | 25.0 | 4.3 | 4.7 | 1.1 |
| GPT-OSS-120B | 26.2 | 15.9 | 38.0 | 7.0 | 18.4 | 5.8 |
| Claude Haiku 4.5 | 48.7 | 2.Occupation | 42.3 | 6.0 | 24.6 | 3.5 |
| Claude Sonnet 4.6 | 53.0 | 7.2 | 67.0 | 9.5 | 33.3 | 4.9 |
| Claude Opus 4.6 | 74.2 | -0.2 | 74.2 | 4.怎么样 | 51.4 | 5.8 |

Note: I notice some garbled text in the original. Key insights:
- **GPT-OSS-120B (mid-tier)** shows the largest gains on SWE (+19.3) and MCP (+7.0)
- **Qwen3-32B (weakest base)** shows the largest gain on SB (+5.8)
- **Opus 4.6 (strongest base)** shows the smallest gain on SWE (-0.2)

---

## 1. Core Definition

**Agent** at evolution step $t$:  
$$A_t = (f, H_t) \tag{1}$$  
where $f$ is the model backbone (fixed) and $H_t$ is the harness state (evolving).

**Evolver**: Updates harness using execution evidence:
$$\Delta H_t = e(H_{t-1}, \mathcal{D}_t), \quad H_t = \text{Apply}(H_{t-1}, \Delta H_t) \tag{2}$$

---

## 2. Methodology

- **Frameworks evaluated**: Three agentic frameworks implementing the protocol: OpenManus (MCP), Agent-S (SWE-bench), and rARe (SearchBench).
- **Models**: 7 LLMs across capability tiers (Qwen3.5-9B to Claude Opus 4.6) as both agents and evolvers.
- **Protocol**: Iterative loop of task-solving → evidence collection → harness update over $T$ steps.

---

## 3. Key Results

### Finding 1: Harness-updating is flat
- **Qwen3.5-9B (smallest)** produces harness updates as useful as **Claude Opus 4.6** when applied to a strong agent.
- Weak evolvers can still generate useful updates; the bottleneck is the agent using them.

### Finding 2: Harness-benefit is non-monotonic
- **Qwen3-32B** benefits the most (up to +19.3 pp), while **Qwen3-235B** often benefits least (e.g., +1.0-1.1 pp on MCP/SB).
- Strong agents (Claude Opus 4.6) also gain moderate improvements (+4.万元 on SB), but closed-source models are already strong at base.

### Finding 3: Two Failure Modes for Weak Models

| Failure Mode | Definition | Evidence |
|---|---|---|
| **Activation Failure** | Weak models don't invoke harness tools/skills | Qwen-235B has high base but low gains; Qwen3-32B has 0 base but 5.8 gain on SB |
| **Harness Following Rate (HFR)** | Weak models load skills but fail to follow them | Qwen3-32B: HFR ~20%; Opus 4.6: 89.3% |
| **Mid-tier sweet spot** | GPT-OSS-120B shows largest gains | SWE: +19.3, MCP: +7.0, SB: +5.8 |

---

## Key Results

### Finding 1: Harness-Updating is "Flat"
Even a small model (Qwen3.5-9B) can produce harness updates that improve GPT-OSS-120B significantly (SWE: +19.3 pp). The size of the update gain does **not** correlate with evolver capability.

### Finding 2: Harness-Benefit is Non-Monotonic
- **Weak models** (Qwen3-32B): Benefit most from harness updates but have low absolute performance.
- **Strong models** (Opus 4.6): Benefit less, sometimes slightly negative (e.g., GPT-OSS-120B on SWE: - rent). 
- **Mid-tier models** (GPT-OSS-120B on SWE): Gain +26.5 points (from 26.2 to 52.7), the largest improvement.

### Finding 3: Two Failure Modes in Weak Models
1. **Activation failure**: Weak models fail to retrieve/use the correct harness components (Qwen3-32B: 3.6% SWE success; loads skills weakly 0.0%→5.8%).
2. **Adherence failure**: Even when correct skills are loaded, models fail to follow them (HFR: 0.30 for Qwen3-32B vs 0.89 for Opus 4.6).

---

## 3. Formalization

### Harness State
$$A_t = (f, H_t)$$
where $f$ is the fixed model backbone and $H_t$ is the harness at step $t$.

### Evolution Step
$$\Delta H_t = e(H_{t-1}, \mathcal{D}_t), \quad H_t = \text{Apply}(H_{t-1}, \Delta H_t)$$
where:
- $e$ is the evolver function (LLM)
- $\mathcal{D}_t$ is the execution evidence
- **Apply** is the commit operation

---

## 4. Experimental Setup

### Models (7 total)
- **Closed**: Claude Opus 4.6, Claude Sonnet 4.5
- **Open**: Qwen3-32B, Qwen3-235B, GPT-OSS-120B, Qwen3.5-9B

### Benchmarks
- **SWE**: Software engineering tasks (CodeRepair)
- **MCP**: Tool-use / model context protocol
- **SB**: Agentic reasoning (Search/Browse)

For each benchmark, an initial harness $H_0$ is defined, then iteratively improved through a max of 3 evolution steps.

---

## 4. Results Summary Table

### Table 1: Agent Base vs. Evolver Gain (Capability × Evolution)

| Model | SWE Base | SWE Δ | MCP Base | MCP Δ | SB Base | SB Δ |
|-------|----------|-------|-----------|--------|--------|------|
| Qwen3-32B | 3.6 | 4.4 | 3.6 | 1.0 | 0.0 | 5.8 |
| Qwen3-235B | 20.7 | 19.9 | 25.0 | 4.3 | 4.7 | 1.1 |
| GPT-OSS-120B | 26.2 | 12.3 | 38.0 | 7.0 | 10.0 | 5.8 |
| Claude Haiku 4.5 | 48.7 | 5.0 | 42.3 | 6.0 | 18.4 | 5.8 |
| Claude Sonnet 4.6 | 67.0 | 3.绿 | 54.7 | 10.9 | 67.0 | 3.8 |
| Claude Opus 4.6 | 73.2 | 4.2 | 54.0 | -1.3 | 73.0 | 4.5 |

*(Note: Some values in the original table appear garbled; interpreted from context)*

### Key Data Points
- **Qwen3-32B** (weakest): Very low base (0.0–3.6%) → highest relative gains (+5.8 on SB)
- **Qwen3-235B/GPT-OSS-120B** (mid): Large absolute gains (SWE: +19.3, +15.9)
- **Claude 4.x models** (strongest): Modest gains but highest absolute performance

### Table 4: Alignment/Follow-through Rates
| Model | Base (SB) | Base (MCP) | Δ (SB) | Δ (MCP) | HFR (SB) |
|-------|-----------|------------|--------|---------|----------|
| Qwen3-32B (weak) | 0.0 | 0.0 | 5.8 | 1.0 | ~20% |
| GPT-OSS-120B (mid) | 10.0 | 18.4 | 5.8 | 7.0 | - |
| Claude Opus 4.6 (strong) | 51.4 | 55.8 | 5.8 | -1.3 | **77%** |

---

## 5. Implications

1. **For practitioners**: Prefer a strong agent with weak harness over weak agent with strong harness. Invest in training the model (agent) over prompt engineering for the evolver.
2. **Targeted improvements**: Focus on (a) harness retrieval/activation skillsholistic, and (b) harness following (compliance) during task execution.
3. **Framework decoupling**: Evaluation of self-evolving agents should separate **harness quality** from **agent capability** to avoid conflating the two (the title's "harness updating is not harness benefit").

---

## 6. Limitations
- Only parametric models (no fine-tuning or RL compared)
- Harness updates limited to prompts/memories; tools interfaces fixed
- Benchmarks focus on code/data tasks; other domains may differ
- Models evaluated are English-focused, not multilingual

---

## 7. Implications for Future Work

1. **Training**: Focusing on *harness utilization* (knowing when and how to use tools) may be more effective than merely improving base task performance.
2. **Scaling**: Model size does not guarantee harness-benefit; mid-tier models benefit most.
3. **Architecture**: Designing harnesses that are easier to activate (e.g., automatic retrieval) and follow (e.g., explicit formatting) could help weak models disproportionately.

---

## 8. Conclusion

The paper clearly demonstrates that **harness-updating capability is not a bottleneck** — even weak models produce useful updates. Instead, the bottleneck lies in **harness-benefit**: the ability of the agent to effectively invoke and adhere to harness guidance. This suggests that for self-evolving agents, research and investment should focus on improving models' ability to utilize external guidance rather than merely generating updates.

---

## References & Additional Notes

*Please note: The paper contains several garbled passages in the original text (particularly in Table 1 and some columns). Values marked with "祥" or unclear characters have been interpreted from context. The core findings remain valid: harness-updating is flat across model scales, while harness-benefit shows non-monotonic patterns with weak models failing to activate and follow harness guidance.*

**Key Implication**: For developers of self-evolving agents, the bottleneck is not the model's ability to *generate* updates (small models suffice), but its ability to *use* updates effectively (requires stronger models or better instruction-following training).

---

_Markdown view of https://picx.dev/p/9wH5KI, served by PicX — AI-generated visual whiteboard summaries of research papers._