Summary (Overview)

• Self-HPO is a recursive self-improving framework for coding agents that alternates harness optimization (H-step) and policy optimization (P-step), using the evolving policy itself as the harness proposer.
• The framework improves the average resolve rate from 57.8% to 64.8% across three coding benchmarks (SWE-Bench Verified, Multilingual, and Pro), consistently outperforming policy-only and harness-only baselines.
• Gains generalize to out-of-distribution settings (non-Python languages and diverse repository-level tasks), and the framework improves both reliability (pass@1) and capability frontier (pass@k).
• Ablations show that both optimized components retain gains even when paired with their base counterparts, and their joint gain exceeds the sum of individual gains.
• With a stronger external model (Opus 4.8) as the harness proposer, the same framework yields further performance boosts (up to 71.6% on SWE-Bench Verified).

---

Introduction and Theoretical Foundation

Background

Large language model (LLM) agents for coding tasks consist of two components:

• Policy model: defines the agent's capability boundary
• Harness: the external scaffold that controls what the model observes, how actions are executed, what feedback is received, and when episodes terminate

The same model may succeed or fail on a problem depending on its harness. Two main lines of research exist:

1. Harness optimization: holds policy fixed, searches over prompts, workflows, or agent code
2. Policy optimization: holds harness fixed, updates model from successful trajectories or verifiable rewards

Key Insight

Harness and policy are mutually reinforcing:

• A harness tuned for one policy checkpoint may not target the failure modes of an updated policy
• Policy learning remains limited to trajectories elicited by a fixed harness

Previous joint optimization approaches rely on external harness proposers and face the challenge of a moving optimization target as the policy evolves.

Problem Formulation

Given a software-engineering task $x$, a coding agent combines policy $P$ with harness $H$. The trajectory $\tau \sim \pi_{H,P}(\cdot \mid x)$ is scored by an executable verifier $y(x, \tau) \in \{0, 1\}$. The expected resolve rate is:

$$J(H, P) = \mathbb{E}_{x \sim \mathcal{D}} \mathbb{E}_{\tau \sim \pi_{H,P}(\cdot | x)} [y(x, \tau)]$$

---

Methodology

Algorithm Overview (Self-HPO)

Starting from base policy $P_0$, seed harness $H_0$, and random search set $\mathcal{S}_0 \subset \mathcal{D}$:

1. H-step: Keep $P_t$ fixed, use it to propose $K$ harness candidates $\{H_t^k\}_{k=1}^K$, retain the best-scoring harness on $\mathcal{S}_t$ as $H_{t+1}$
2. P-step: Freeze $H_{t+1}$, update $P_t$ to $P_{t+1}$ on $\mathcal{D}$ using reinforcement learning
3. Next search set: Tasks never resolved during the P-step form $\mathcal{S}_{t+1}$
4. Loop stops when the hard-task set stabilizes between rounds

H-step: Failure-Conditioned Harness Search

• Self-proposer: The frozen policy $P_t$ acts as both task solver and harness proposer
• The proposer inspects current failures and produces targeted revisions to prompts or executable runtime logic
• Each of $K$ search iterations revises the current best harness using accumulated experience
• Evolving search set: Initialized uniformly from $\mathcal{D}$, subsequent sets consist of tasks never resolved in any rollout during the preceding P-step

P-step: Policy Optimization under Optimized Harness

• Collects on-policy trajectories over $\mathcal{D}$ as the policy is updated
• Uses Group Relative Policy Optimization (GRPO) combined with Balanced Adaptive Rollout (BAR) from Orchard
• BAR adaptively draws additional trajectories to construct reward-balanced groups, reducing uninformative zero-variance groups under sparse outcome rewards

---

Empirical Validation / Results

Main Results

Table 1: Main results (resolve rates with standard deviations over three runs)

Key Findings

1. Joint optimization wins: Self-HPO outperforms RLVR by 2.0 and Self-Harness by 4.3 points on SWE-Verified
2. Out-of-distribution generalization: Gains transfer to non-Python languages (SWE-Multi) and diverse tasks (SWE-Pro)
3. Harness optimization > SFT distillation: Meta-Harness outperforms SFT by 4.1 points on SWE-Verified and 8.9 points on average
4. Alternating optimization drives continued improvement: The second H-step improves the updated policy, and the following P-step provides further gains

Ablation Study (Table 2)

Cross-Evaluation (Table 3)

Cross-evaluating Self-HPO policies and harnesses on SWE-Bench Verified

The joint gain (8.5 points) exceeds the sum of individual gains (3.5 + 3.4 = 6.9 points) by 1.6 points.

Additional Analysis

• Pass@k: Self-HPO remains above Base for every $k$ from 1 to 8; pass@8 increases from 80.0% to 84.8%
• Pass@1 by difficulty: Gains of 17.2 and 12.9 points on Medium and Easy tasks, 6.1 points on Hard tasks; 2.2-point decrease on already-Resolved tasks
• Training signal: Zero-success rollout groups decline to 6.5% (vs. 11-17% for RLVR), reducing reward sparsity

Case Study: Harness Evolution

The optimized harness adds targeted controls:

1. Budget warnings: Alerts the model as step/time limits approach, prompts best-effort submission
2. Empty-patch guard: Asks model to include newly created files via git add -N
3. Interface verification: Delays first submission to prompt checking API names/signatures
4. Fallback submission: Uses sandbox's Git diff when patch fails the apply check

---

Theoretical and Practical Implications

Theoretical Contributions

• Self-evolving agents: Demonstrates that an agent can use its own policy as a harness proposer without external supervision
• Co-adaptation dynamics: Shows that harness and policy improvements are complementary and super-additive — their joint gain exceeds the sum of individual gains
• Moving target adaptation: The evolving search set mechanism addresses the challenge of optimizing against a moving policy

Practical Implications

• No external model required: Self-HPO achieves strong results without relying on a frontier model as proposer
• Transferable improvements: Gains generalize beyond the training distribution (Python tasks) to other languages and task types
• Reliability + capability: The framework improves both consistency (pass@1) and exploration (pass@k)
• Training efficiency: Reduced zero-success groups during RL suggest more informative gradient signals

---

Conclusion

Self-HPO introduces a recursive self-improving framework for coding agents that alternates harness and policy optimization. The evolving policy proposes its own harness revisions, learns through RL under the selected harness, and supplies hard tasks for the next round of harness search.

Key takeaways:

1. Joint harness-policy optimization consistently outperforms single-component baselines
2. The self-improving setup works without external proposers, though external models provide additional gains
3. Both components retain value independently, and their combination is super-additive
4. The framework improves both reliability and capability frontier

Future directions implied by this work include:

• Extending the framework to more optimization rounds and larger search budgets
• Exploring alternative policy optimization strategies beyond GRPO
• Investigating the applicability to other agent domains beyond coding (e.g., web navigation, computer use)
• Understanding when harness vs. policy optimization provides the greatest marginal benefit at different stages of training