# Self-Evolving Coding Agents via Harness-Policy Optimization (Self-HPO)

> Self-HPO, a recursive self-improving framework alternating harness and policy optimization, boosts coding agent resolve rates from 57.8% to 64.8%, with joint gains exceeding the sum of individual improvements.

- **Source:** [arXiv](https://arxiv.org/abs/2610.22791)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/OUPrff
- **Whiteboard:** https://picx.dev/p/OUPrff/image

## Summary

## Summary (Overview)

- **Self-HPO** is a recursive self-improving framework for coding agents that **alternates harness optimization (H-step) and policy optimization (P-step)**, using the evolving policy itself as the harness proposer.
- The framework improves the average resolve rate from **57.8% to 64.8%** across three coding benchmarks (SWE-Bench Verified, Multilingual, and Pro), consistently outperforming policy-only and harness-only baselines.
- Gains generalize to **out-of-distribution settings** (non-Python languages and diverse repository-level tasks), and the framework improves both reliability (pass@1) and capability frontier (pass@k).
- Ablations show that **both optimized components retain gains even when paired with their base counterparts**, and their joint gain exceeds the sum of individual gains.
- With a stronger external model (Opus 4.8) as the harness proposer, the same framework yields further performance boosts (up to 71.6% on SWE-Bench Verified).

---

## Introduction and Theoretical Foundation

### Background
Large language model (LLM) agents for coding tasks consist of two components:
- **Policy model**: defines the agent's capability boundary
- **Harness**: the external scaffold that controls what the model observes, how actions are executed, what feedback is received, and when episodes terminate

The same model may succeed or fail on a problem depending on its harness. Two main lines of research exist:
1. **Harness optimization**: holds policy fixed, searches over prompts, workflows, or agent code
2. **Policy optimization**: holds harness fixed, updates model from successful trajectories or verifiable rewards

### Key Insight
Harness and policy are **mutually reinforcing**:
- A harness tuned for one policy checkpoint may not target the failure modes of an updated policy
- Policy learning remains limited to trajectories elicited by a fixed harness

Previous joint optimization approaches rely on **external harness proposers** and face the challenge of a **moving optimization target** as the policy evolves.

### Problem Formulation

Given a software-engineering task $x$, a coding agent combines policy $P$ with harness $H$. The trajectory $\tau \sim \pi_{H,P}(\cdot \mid x)$ is scored by an executable verifier $y(x, \tau) \in \{0, 1\}$. The expected resolve rate is:

$$J(H, P) = \mathbb{E}_{x \sim \mathcal{D}} \mathbb{E}_{\tau \sim \pi_{H,P}(\cdot | x)} [y(x, \tau)]$$

---

## Methodology

### Algorithm Overview (Self-HPO)

Starting from base policy $P_0$, seed harness $H_0$, and random search set $\mathcal{S}_0 \subset \mathcal{D}$:

1. **H-step**: Keep $P_t$ fixed, use it to propose $K$ harness candidates $\{H_t^k\}_{k=1}^K$, retain the best-scoring harness on $\mathcal{S}_t$ as $H_{t+1}$
2. **P-step**: Freeze $H_{t+1}$, update $P_t$ to $P_{t+1}$ on $\mathcal{D}$ using reinforcement learning
3. **Next search set**: Tasks never resolved during the P-step form $\mathcal{S}_{t+1}$
4. Loop stops when the hard-task set stabilizes between rounds

### H-step: Failure-Conditioned Harness Search

- **Self-proposer**: The frozen policy $P_t$ acts as both task solver and harness proposer
- The proposer inspects current failures and produces targeted revisions to prompts or executable runtime logic
- Each of $K$ search iterations revises the current best harness using accumulated experience
- **Evolving search set**: Initialized uniformly from $\mathcal{D}$, subsequent sets consist of tasks never resolved in any rollout during the preceding P-step

### P-step: Policy Optimization under Optimized Harness

- Collects on-policy trajectories over $\mathcal{D}$ as the policy is updated
- Uses **Group Relative Policy Optimization (GRPO)** combined with **Balanced Adaptive Rollout (BAR)** from Orchard
- BAR adaptively draws additional trajectories to construct reward-balanced groups, reducing uninformative zero-variance groups under sparse outcome rewards

---

## Empirical Validation / Results

### Main Results

**Table 1: Main results** (resolve rates with standard deviations over three runs)

| Method | Harness Opt | Policy Opt | SWE-Verified | SWE-Multi | SWE-Pro | Avg. |
|--------|:-----------:|:----------:|:------------:|:---------:|:-------:|:----:|
| Base | ✕ | ✕ | 61.0 ± 0.2 | 54.2 ± 0.3 | 58.3 ± 1.2 | 57.8 ± 0.4 |
| **Setting 1: External-model setup (Opus 4.8)** |
| SFT | ✕ | √ | 65.0 ± 1.1 | 49.6 ± 0.8 | 49.1 ± 0.6 | 54.6 ± 0.5 |
| Meta-Harness | √ | ✕ | 69.1 ± 1.9 | 56.5 ± 0.9 | 64.8 ± 1.4 | 63.5 ± 0.8 |
| **HPO (ours)** | √ | √ | **71.6 ± 0.9** | **61.4 ± 0.7** | **66.0 ± 0.5** | **66.3 ± 0.4** |
| **Setting 2: Self-improving setup** |
| RLVR | ✕ | √ | 67.5 ± 0.8 | 56.3 ± 1.0 | 61.8 ± 1.3 | 61.9 ± 0.6 |
| Self-Harness | √ | ✕ | 65.2 ± 1.5 | 54.3 ± 0.7 | 60.7 ± 0.9 | 60.1 ± 0.6 |
| **Self-HPO (ours)** | √ | √ | **69.5 ± 0.9** | **59.7 ± 0.8** | **65.2 ± 1.4** | **64.8 ± 0.6** |

### Key Findings

1. **Joint optimization wins**: Self-HPO outperforms RLVR by 2.0 and Self-Harness by 4.3 points on SWE-Verified
2. **Out-of-distribution generalization**: Gains transfer to non-Python languages (SWE-Multi) and diverse tasks (SWE-Pro)
3. **Harness optimization > SFT distillation**: Meta-Harness outperforms SFT by 4.1 points on SWE-Verified and 8.9 points on average
4. **Alternating optimization drives continued improvement**: The second H-step improves the updated policy, and the following P-step provides further gains

### Ablation Study (Table 2)

| Configuration | Resolve rate (%) |
|---------------|:----------------:|
| HPO | 71.6 |
| w/ SFT in P-step | 67.2 (-4.4) |
| w/ RFT in P-step | 70.5 (-1.1) |
| w/ Fixed search set in H-step | 70.1 (-1.5) |
| w/ Self-proposer in H-step | 69.5 (-2.1) |

### Cross-Evaluation (Table 3)

**Cross-evaluating Self-HPO policies and harnesses on SWE-Bench Verified**

| | $P_0$ | $P_1$ | $P_2$ |
|---|:-----:|:-----:|:-----:|
| $H_0$ | 61.0 ± 0.2 | 61.7 ± 1.3 | 64.4 ± 1.1 |
| $H_1$ | 64.3 ± 1.1 | 67.8 ± 1.6 | 68.1 ± 1.0 |
| $H_2$ | 64.5 ± 1.5 | 68.6 ± 0.5 | **69.5 ± 0.9** |

The joint gain (8.5 points) **exceeds the sum of individual gains** (3.5 + 3.4 = 6.9 points) by 1.6 points.

### Additional Analysis

- **Pass@k**: Self-HPO remains above Base for every $k$ from 1 to 8; pass@8 increases from 80.0% to 84.8%
- **Pass@1 by difficulty**: Gains of 17.2 and 12.9 points on Medium and Easy tasks, 6.1 points on Hard tasks; 2.2-point decrease on already-Resolved tasks
- **Training signal**: Zero-success rollout groups decline to 6.5% (vs. 11-17% for RLVR), reducing reward sparsity

### Case Study: Harness Evolution

The optimized harness adds targeted controls:
1. **Budget warnings**: Alerts the model as step/time limits approach, prompts best-effort submission
2. **Empty-patch guard**: Asks model to include newly created files via `git add -N`
3. **Interface verification**: Delays first submission to prompt checking API names/signatures
4. **Fallback submission**: Uses sandbox's Git diff when patch fails the apply check

---

## Theoretical and Practical Implications

### Theoretical Contributions
- **Self-evolving agents**: Demonstrates that an agent can use its own policy as a harness proposer without external supervision
- **Co-adaptation dynamics**: Shows that harness and policy improvements are **complementary** and **super-additive** — their joint gain exceeds the sum of individual gains
- **Moving target adaptation**: The evolving search set mechanism addresses the challenge of optimizing against a moving policy

### Practical Implications
- **No external model required**: Self-HPO achieves strong results without relying on a frontier model as proposer
- **Transferable improvements**: Gains generalize beyond the training distribution (Python tasks) to other languages and task types
- **Reliability + capability**: The framework improves both consistency (pass@1) and exploration (pass@k)
- **Training efficiency**: Reduced zero-success groups during RL suggest more informative gradient signals

---

## Conclusion

Self-HPO introduces a recursive self-improving framework for coding agents that alternates harness and policy optimization. The evolving policy proposes its own harness revisions, learns through RL under the selected harness, and supplies hard tasks for the next round of harness search.

**Key takeaways:**
1. Joint harness-policy optimization consistently outperforms single-component baselines
2. The self-improving setup works without external proposers, though external models provide additional gains
3. Both components retain value independently, and their combination is super-additive
4. The framework improves both reliability and capability frontier

**Future directions** implied by this work include:
- Extending the framework to more optimization rounds and larger search budgets
- Exploring alternative policy optimization strategies beyond GRPO
- Investigating the applicability to other agent domains beyond coding (e.g., web navigation, computer use)
- Understanding when harness vs. policy optimization provides the greatest marginal benefit at different stages of training

---

_Markdown view of https://picx.dev/p/OUPrff, served by PicX — AI-generated visual whiteboard summaries of research papers._