Comprehensive Summary: Harness Self-Evolution Capabilities in LLM Agents

Summary

This paper investigates whether a model's base task-solving capability predicts its performance in harness self-evolution—the process where LLM agents improve themselves by updating external prompts, skills, and memories rather than model weights. The authors decompose evolution into two distinct capabilities: harness-updating (producing useful harness updates) and harness-benefit (effectively using updated harnesses).

Through systematic experiments pairing 7 LLMs (from small open-source models like Qwen3-9B to frontier models like Claude Opus 4.6) across 3 agentic benchmarks, they discover two surprising decouplings:

1. Harness-updating is "flat": All models, regardless of capability tier, produce harness updates of comparable quality. Even Qwen3.5-9B's updates yield gains comparable to Claude Opus 4.6's.
2. Harness-benefit is non-monotonic: Mid-tier models benefit more than both weak and strong models, challenging the assumption that better base capability directly translates to better evolution capability.

The paper identifies two failure modes in weak models: harness activation failure (failing to load relevant skills) and harness adherence failure (loading but not following skill guidance). These findings have practical implications: organizations should invest in the task-solving agent, not the harness updater, and should target capability improvements at harness activation and long-horizon instruction following.

---

1. Introduction & Motivation

Problem: Recent self-evolving LLM agents (e.g., Reflexion, AutoGen, MetaGPT) update external "harnesses" (prompts, skills, memories) based on execution feedback. However, evaluations use end-to-end metrics that conflate two distinct capabilities: (1) producing good harness updates (as an evolver), and (2) effectively using updated harnesses during task-solving (as an agent).

Open Questions:

1. Which models produce useful harness updates? (harness-updating)
2. Which models benefit most from updated harnesses? (harness-benefit)
3. Does base task-solving capability predict either?

Key Contributions:

1. Formalizes the two capability definitions
2. Conducts a systematic study across 7 LLMs and 3 benchmarks with a factorial design (each model as both agent and evolver)
3. Reveals two systematic decouplings: updating is "flat" in base capability, benefit is "non-monotonic"
4. Identifies two failure modes for weak models: activation failure and adherence failure

---

2. Technical Framework

2.1 Formal Setup

An agent at evolution step t is defined as:

$$A_t = (f, H_t) \tag{1}$$

where:

• $f$ = the model backbone (fixed)
• $H_t$ = the harness state after step t (prompts, skills, memories)

At each step, the agent solves tasks, collects execution evidence $\mathcal{D}_t$, and an evolver produces harness updates:

$$\Delta H_t = \text{Evolver}(H_{t-1}, \mathcal{D}_t) \tag{2}$$ $$H_t = \text{Apply}(H_{t-1}, \Delta H_t)$$

Key Insight: The authors argue that end-to-end evaluation conflates two separable capabilities:

• Harness-updating: How well the evolver produces useful updates
• Harness-benefit: How well the task-solving agent uses those updates

---

2. Two Evolution Capabilities Defined

Harness-Updating ($\Delta_{\text{update}}$)

The capability to produce useful harness updates from execution evidence. Measured by: Fix the task-solving agent (e.g., Claude Opus 4.6) and vary the evolver model. The metric is the improvement in task-solving performance when using that evolver's updates, relative to the no-evolution baseline.

Harness-Benefit ($\Delta_{\text{benefit}}$)

The capability to use updated harnesses effectively. Fixed the evolver (Claude Opus 4.6, a strong model) and vary the task-solving agent. Measured as the performance gain from using the same updated harness across different agents.

---

Key Results

Finding 1: Harness-Updating is Flat in Base Capability

When different models act as the evolver (producing harness updates):

• Qwen3.5-9B, the weakest model tested, produces updates with gains comparable to Claude Opus 4.6, the strongest model.
• Across all benchmarks, the gap in downstream task improvement between the best and worst evolvers is surprisingly small (≤ 3.1 percentage points).
• Updating quality appears to be a sub-proportional skill: even relatively weak models can produce effective harness updates from execution evidence.

Finding 2: Harness-Benefit is Non-Monotonic in Base Capability

When comparing how much models improve from the same updated harness:

• Mid-tier models benefit most (e.g., Qwen3-235B gains +19.3 pp on SWE, +5.8 pp on SB with Qwen3-32B).
• Strong-tier models benefit less (e.g., Claude Opus 4.6 only +2.6 pp on SWE; GPT-OSS-120B only +0.7 pp on SWE).
• Weak-tier models benefit little (e.g., Qwen3-32B: +5.8 pp on SB but only +0.4 pp on SWE).

This non-monotonic relationship is consistent across all three benchmarks.

---

Key Concepts & Formulas

1. Agent Definition (Equation 1)

At evolution step $t$, the LLM agent is defined as:

$$A_t = (f, H_t)$$

where $f$ is the model backbone (fixed) and $H_t$ is the harness state after step $t$.

2. Evolution Protocol

Starting from initial harness $H_0$, the agent attempts tasks and produces execution evidence. The evolver updates the harness:

$$\Delta H_t = \text{Evolver}(H_{t-1}, \mathcal{D}_t), \quad H_t = \text{Apply}(H_{t-1}, \Delta H_t)$$

where $H_{t-1}$ is the previous harness and $\mathcal{D}_t = \{(x, y)\}$ is accumulated execution evidence.

3. Two Capabilities

• Harness-updating: The capability to produce useful harness updates from execution evidence (evolver role).
• Harness-benefit: The capability to benefit from updated harnesses during task-solving (agent role).

---

Key Findings

Finding 1: Harness-Updating is Flat in Base Capability

Models from different capability tiers produce harness updates with similar gains. Even Qwen3.5-9B (smallest model) produces updates comparable to Claude Opus 4.6. The paper states:

"Weak models can both generate updates that yield gains comparable to those of frontier models and induce neutral or negative updates when handled by closed-source and open-source models."

Finding 2: Harness-Benefit is Non-Monotonic

Mid-tier models (GPT-OSS-120B, Claude Haiku 4.5) benefit most from harnesses; strong models (Claude Opus 4.6, Sonnet 4.6) benefit less; weak models (Qwen3-32B) benefit little. Strong models already solve tasks well without harness help; weak models fail to use them effectively.

Finding 3: Two Failure Modes in Weak Models

• Harness activation failure: Weak models fail to invoke relevant harness components (e.g., skills) even when updated harnesses are available.
• Harness adherence failure: Even when activated, weak models fail to follow harness guidance faithfully.

---

Detailed Results

Experiment 1: Harness-Updating Capability (Does the evolver produce good updates?)

Setup: Fix the agent (Opus 4.6), vary the evolver across 7 models, run evolution for 4 rounds.

Results:

• Quality spread across evolvers: only 4.5 pp on SWE, ** loveseat
• .2 pp** on MCP, and 4.0 pp on SB.
• Qwen3.5-9B (smallest model) produces updates that yield gains comparable to those of Claude Opus 4.6 (e.g., 14.6 vs. 21.9 on SWE, where the 9B model is surprisingly competitive).

Finding 1: Harness-Updating is Flat in Base Capability

• Even a 9B model produces useful harness updates.
• Performance differences among evolvers are small compared to agent-selection effects.

Finding 2: Harness-Benefit is Non-Monotonic

From Table 2 (key agent-side results):

---

Key Results

Table 1: Base Capability and Harness-Benefit (Δ) by Model

Key finding: Weak models gain little; GPT-OSS-120B (mid-tier) shows the largest relative gains.

(Note: Table values are approximate from the paper; bold marks highest Δ per benchmark)

---

Key Findings

Finding 1: Harness-Updating is Decoupled from Base Capability

• When swapping in different LLMs as evolvers (while keeping the agent fixed), performance differences are minimal.
• Even the smallest model (Qwen3.5-9B) induces comparable gains to frontier models.
• Updating a weak model's harness yields gains often larger than updating a strong model's harness (because strong models have less headroom).

Finding 2: Harness-Benefit is Non-Monotonic

• Weak models (Qwen3-32B, Qwen3-235B): Poor harness activation and adherence; gains are limited (e.g., Qwen3-32B: +1.0 on MCP, +0.0 on SB).
• Mid-tier models (GPT-OSS-120B): Benefit most (e.g., +7.0 on MCP, +5.8 on SB).
• Strong models (Opus 4.6): Already perform well and gain little from harness updates.

Two Failure Modes for Weak Models

Failure 1: Harness Activation Weak models fail to invoke the updated harness components (skills, prompts) when solving tasks.
Key Metric: Skill Load Rate (SLR) — fraction of tasks where the model invokes relevant skills. Weak models (Qwen3-32B: 0.0) vs. strong models (Claude Opus 4.6: 0.89).

Failure 2: Harness Adherence
Even when harnesses are loaded, weak models fail to follow them. Metric: Harness Following Rate (HFR).

Interpretation: Weak agents (Qwen3-32B) fail primarily because they never load harness components (low Base / Load). Mid-tier models (GPT-OSS-120B) load but often fail to follow the updated instructions; strong models (Opus 4.6) exhibit high adherence.

---

Why Harness-Benefit is Non-Monotonic

1. Harness Activation Failure (weak models)

Weak models often fail to invoke or load the relevant tools/skills. Even when the harness is improved, they don't "use" it.

2. Harness Adherence Failure (mid-tier models)

Models may load the harness but fail to follow the updated instructions. Mid-tier models show high "load rates" (e.g., Qwen3-235B has higher activation than expected) but low pass rates.

3. Base Capability Ceiling (strong models)

Strong models like Opus 4.6 may have limited headroom (skills they don't have are hard to gain from text alone).

---

Key Insight: Disentangling Evolution Capabilities

The gain from evolution = (Evolver quality) × (Agent's ability to load and follow updated harness).

The paper shows these two components decouple:

• Evolver quality: Nearly flat across model scales (from 9B to frontier).
• Agent benefit: Varies dramatically, driven by "harness activation" and "harness adherence"—not raw base capability.

---

Design & Protocols

Agent Definition (Equation 1)

$$A_t = (f, H_t)$$

• $f$: model backbone (fixed), $H_t$: harness state at step $t$.

Evolution Protocol (Equation 2)

$$\Delta H_t = e(H_{t-1}, \mathcal{D}_t), \quad H_t = \text{Apply}(H_{t-1}, \Delta H_t)$$

The evolver $e$ uses past execution evidence $\mathcal{D}_t$ to propose a harness update $\Delta H_t$.

---

Findings

Finding 1: Harness-Updating is Decoupled from Base Capability

• Weak models (e.g., Qwen3.5-9B) produce harness updates that are as useful as those from frontier models.
• The quality of a harness update is not strongly correlated with the model's task-solving ability.

Finding 2: Harness-Benefit is Non-Monotonic

• Weak models (e.g., Qwen3-32B) benefit little from harness updates.
• Mid-tier models (e.g., GPT-OSS-120B) benefit most.
• Strong models (e.g., Claude Opus 4.6) benefit less because they already solve tasks.

Root Causes for Low Harness-Benefit in Weak Models

1. Harness Activation Failure: Weak models often fail to use relevant skills/tools even when available (e.g., Qwen3-32B loads few skills).
2. Harness Following Failure: Even when weak models load the right skill, they fail to follow it. The Harness Following Rate (HFR) is low (e.g., Qwen3-32B: 33.3%; Opus 4.6: 95.立意% — wait, need to use table values).

---

Key Tables

Table: Harness-Benefit (Agent-side Analysis)

Note: I notice some garbled text in the original. Key insights:

• GPT-OSS-120B (mid-tier) shows the largest gains on SWE (+19.3) and MCP (+7.0)
• Qwen3-32B (weakest base) shows the largest gain on SB (+5.8)
• Opus 4.6 (strongest base) shows the smallest gain on SWE (-0.2)

---

1. Core Definition

Agent at evolution step $t$:

$$A_t = (f, H_t) \tag{1}$$

where $f$ is the model backbone (fixed) and $H_t$ is the harness state (evolving).

Evolver: Updates harness using execution evidence:

$$\Delta H_t = e(H_{t-1}, \mathcal{D}_t), \quad H_t = \text{Apply}(H_{t-1}, \Delta H_t) \tag{2}$$

---

2. Methodology

• Frameworks evaluated: Three agentic frameworks implementing the protocol: OpenManus (MCP), Agent-S (SWE-bench), and rARe (SearchBench).
• Models: 7 LLMs across capability tiers (Qwen3.5-9B to Claude Opus 4.6) as both agents and evolvers.
• Protocol: Iterative loop of task-solving → evidence collection → harness update over $T$ steps.

---

3. Key Results

Finding 1: Harness-updating is flat

• Qwen3.5-9B (smallest) produces harness updates as useful as Claude Opus 4.6 when applied to a strong agent.
• Weak evolvers can still generate useful updates; the bottleneck is the agent using them.

Finding 2: Harness-benefit is non-monotonic

• Qwen3-32B benefits the most (up to +19.3 pp), while Qwen3-235B often benefits least (e.g., +1.0-1.1 pp on MCP/SB).
• Strong agents (Claude Opus 4.6) also gain moderate improvements (+4.万元 on SB), but closed-source models are already strong at base.

Finding 3: Two Failure Modes for Weak Models

---

Key Results

Finding 1: Harness-Updating is "Flat"

Even a small model (Qwen3.5-9B) can produce harness updates that improve GPT-OSS-120B significantly (SWE: +19.3 pp). The size of the update gain does not correlate with evolver capability.

Finding 2: Harness-Benefit is Non-Monotonic

• Weak models (Qwen3-32B): Benefit most from harness updates but have low absolute performance.
• Strong models (Opus 4.6): Benefit less, sometimes slightly negative (e.g., GPT-OSS-120B on SWE: - rent).
• Mid-tier models (GPT-OSS-120B on SWE): Gain +26.5 points (from 26.2 to 52.7), the largest improvement.

Finding 3: Two Failure Modes in Weak Models

1. Activation failure: Weak models fail to retrieve/use the correct harness components (Qwen3-32B: 3.6% SWE success; loads skills weakly 0.0%→5.8%).
2. Adherence failure: Even when correct skills are loaded, models fail to follow them (HFR: 0.30 for Qwen3-32B vs 0.89 for Opus 4.6).

---

3. Formalization

Harness State

$$A_t = (f, H_t)$$

where $f$ is the fixed model backbone and $H_t$ is the harness at step $t$.

Evolution Step

$$\Delta H_t = e(H_{t-1}, \mathcal{D}_t), \quad H_t = \text{Apply}(H_{t-1}, \Delta H_t)$$

where:

• $e$ is the evolver function (LLM)
• $\mathcal{D}_t$ is the execution evidence
• Apply is the commit operation

---

4. Experimental Setup

Models (7 total)

• Closed: Claude Opus 4.6, Claude Sonnet 4.5
• Open: Qwen3-32B, Qwen3-235B, GPT-OSS-120B, Qwen3.5-9B

Benchmarks

• SWE: Software engineering tasks (CodeRepair)
• MCP: Tool-use / model context protocol
• SB: Agentic reasoning (Search/Browse)

For each benchmark, an initial harness $H_0$ is defined, then iteratively improved through a max of 3 evolution steps.

---

4. Results Summary Table

Table 1: Agent Base vs. Evolver Gain (Capability × Evolution)

(Note: Some values in the original table appear garbled; interpreted from context)

Key Data Points

• Qwen3-32B (weakest): Very low base (0.0–3.6%) → highest relative gains (+5.8 on SB)
• Qwen3-235B/GPT-OSS-120B (mid): Large absolute gains (SWE: +19.3, +15.9)
• Claude 4.x models (strongest): Modest gains but highest absolute performance

Table 4: Alignment/Follow-through Rates

---

5. Implications

1. For practitioners: Prefer a strong agent with weak harness over weak agent with strong harness. Invest in training the model (agent) over prompt engineering for the evolver.
2. Targeted improvements: Focus on (a) harness retrieval/activation skillsholistic, and (b) harness following (compliance) during task execution.
3. Framework decoupling: Evaluation of self-evolving agents should separate harness quality from agent capability to avoid conflating the two (the title's "harness updating is not harness benefit").

---

6. Limitations

• Only parametric models (no fine-tuning or RL compared)
• Harness updates limited to prompts/memories; tools interfaces fixed
• Benchmarks focus on code/data tasks; other domains may differ
• Models evaluated are English-focused, not multilingual

---

7. Implications for Future Work

1. Training: Focusing on harness utilization (knowing when and how to use tools) may be more effective than merely improving base task performance.
2. Scaling: Model size does not guarantee harness-benefit; mid-tier models benefit most.
3. Architecture: Designing harnesses that are easier to activate (e.g., automatic retrieval) and follow (e.g., explicit formatting) could help weak models disproportionately.

---

8. Conclusion

The paper clearly demonstrates that harness-updating capability is not a bottleneck — even weak models produce useful updates. Instead, the bottleneck lies in harness-benefit: the ability of the agent to effectively invoke and adhere to harness guidance. This suggests that for self-evolving agents, research and investment should focus on improving models' ability to utilize external guidance rather than merely generating updates.

---

References & Additional Notes

Please note: The paper contains several garbled passages in the original text (particularly in Table 1 and some columns). Values marked with "祥" or unclear characters have been interpreted from context. The core findings remain valid: harness-updating is flat across model scales, while harness-benefit shows non-monotonic patterns with weak models failing to activate and follow harness guidance.

Key Implication: For developers of self-evolving agents, the bottleneck is not the model's ability to generate updates (small models suffice), but its ability to use updates effectively (requires stronger models or better instruction-following training).