# HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

> HarnessCompass improves LLM agent harnesses from 54% to 66% Pass@1 in 5 iterations by constraining evolution, adding grounded first-person feedback, and optimizing components separately.

- **Source:** [arXiv](https://arxiv.org/abs/2608.01918)
- **Published:** 2026-08-15
- **Permalink:** https://picx.dev/p/pfX4Qf
- **Whiteboard:** https://picx.dev/p/pfX4Qf/image

## Summary

# HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

## Summary (Overview)

- **Novel framework**: HarnessCompass introduces a disciplined automatic harness evolution framework for LLM agents, built on three principles: constrained evolution, proactive feedback, and component-wise optimization.
- **Key results**: On SWE-bench Verified with GPT-5.4, HarnessCompass improves Pass@1 from 54% to 66% in only 5 evolution iterations, outperforming the state-of-the-art AHE method (63% in 20 iterations).
- **Strong generalization**: The evolved harness transfers effectively to held-out tasks (60.4% vs. AHE's 54.7%) and across base models (improving Claude-Sonnet-4.6 from 70.0% to 73.8% without further evolution).
- **Identified limitations**: The paper systematically identifies three limitations of existing automatic harness evolution: search-task overfitting, insufficient feedback signals, and cross-component interference.
- **Efficiency gains**: HarnessCompass reaches higher performance in roughly one-quarter of the iterations required by AHE, demonstrating substantially better evolution efficiency.

## Introduction and Theoretical Foundation

### Background and Motivation

LLM agents deployed in executable environments (code repositories, terminal workflows, tool coordination) depend critically on their **harness**—the software layer orchestrating how the model interacts with its environment, including prompts, tools, middleware, memory, and verification routines. Recent work shows that harness design alone can substantially shift agent performance even with a fixed base model.

The core problem: the optimal harness is **model-specific** and must be re-tuned whenever the base model changes. Manual harness design by developers cannot keep pace with rapid base model advancement, creating a gap between what a model *can* do and what its harness *lets it realize*.

### Three Identified Limitations of Existing Methods

The paper identifies three fundamental limitations in current automatic harness evolution approaches:

1. **Overfitting to search tasks** (❶): Meta-agents revising harnesses freely against verifier feedback from a fixed task set tend to bake in task-specific shortcuts that fail to transfer to held-out tasks.

2. **Reliance solely on trajectory-derived signals** (❷): The meta-agent observes interactions only externally, learning *that* a task failed and *where*, but not *why* the agent found the harness hard to use—leading to misattribution between harness friction and agent errors.

3. **Joint optimization of all components** (❸): When prompts, tools, middleware, and memory are revised together, their edits interfere, capping rather than compounding individual gains.

### Core Insight

> "The core challenge, therefore, is not harness evolution itself, but the lack of structure and discipline governing the evolution process."

The three principles of HarnessCompass address these limitations:
- **Constrained evolution** → counters overfitting
- **Proactive feedback** → enriches evidence
- **Component-wise optimization** → reduces interference

## Methodology

### Formal Setup

Let $M$ denote the base model (fixed throughout evolution), and $\mathcal{H}$ denote a harness as a set of editable components. A harness consists of **seven orthogonal component types** divided into two categories:

- **Structural components**: tool implementations, middleware, sub-agent configurations
- **Guidance components**: system prompt, tool descriptions, skills, memory

Running $M$ with harness $\mathcal{H}$ on task $x \in \mathcal{D}$ produces a trajectory $\tau$ (complete sequence of messages, tool calls, environment observations). A verifier assigns outcome $r(\tau) \in \{0, 1\}$. With $k$ rollouts per task, **Pass@1** is the average success rate over evaluated tasks.

The evolution process starts from a **minimal harness** $\mathcal{H}_0$ (single shell command as tool, short system prompt, no middleware/skills/sub-agents) to ensure every later component is attributable to the evolution loop itself.

### Algorithm Overview

```
Algorithm 1: Evolution procedure of HARNESSCOMPASS
Require: initial harness H₀, base model M, meta agent M, search tasks D, rollouts per task k, iterations N
Ensure: evolved harness H*
1: H* ← H₀
2: for t = 1 to N do
3:   T ← ROLLOUT(M, H*, D, k)                    ▷ evaluate current harness
4:   E ← DISTILL(T)                              ▷ trajectory-based evidence
5:   F ← GROUNDSELFFEEDBACK(M, T)                ▷ first-person evidence
6:   for all track c ∈ {structure, guidance} do  ▷ component-wise optimization
7:     H^c ← GATEDEVOLVE(M, H*, E, F, c)         ▷ constrained evolution
8:     T^c ← ROLLOUT(M, H^c, D, k)
9:   end for
10:  w ← arg max_c Pass@1(T^c); ℓ ← the other track
11:  H_t ← R³(H^w, H^ℓ)                          ▷ integrate complementary edits
12:  if Pass@1(T^w) > Pass@1(T) then
13:    H* ← H_t                                  ▷ accept improved harness
14:  end if
15: end for
16: return H*
```

### Principle 1: Constrained Evolution

Implemented as a **global generalization gate** that every candidate edit must pass, imposing two requirements:

**Content requirement**: Rejects any edit mentioning specific task instances, test functions, or private symbols; rejects code branches triggered by tokens appearing only in particular tasks. Admits reusable decision criteria with applicability conditions evaluable on unseen tasks.

**Placement requirement**: Separates edits into two categories:
- **Capability edits** (new executable functionality): must be implemented as executable code in middleware, tools, or sub-agents
- **Guidance edits** (behavioral instructions): must be placed in system prompt or memory, not executable components

This prevents guidance from being encoded as executable logic, which would lack reliable execution conditions.

### Principle 2: Proactive Feedback

Three-step process to enrich evidence:

**Step 1—Eliciting first-person feedback**: After verification, collect feedback from failed tasks using the same base model $M$, via two reconciled queries:
- **Blind report**: Trajectory provided without revealing the verdict—avoids hindsight bias
- **Hindsight report**: Verdict revealed, agent attributes failure to one of four causes: harness, agent's own reasoning, task ambiguity, or environment

**Step 2—Grounding against trajectories**: A feedback item is retained only when supported by the trajectory. For complaints about existing components, the trajectory must directly reveal the difficulty. For requests for new capabilities, the trajectory must demonstrate a concrete gap the capability would address.

**Step 3—Aggregation and use**: Group reports by implicated harness component, assign confidence scores based on agreement level, number of distinct tasks affected, and consistency between blind/hindsight reports. This serves as *additional* evidence, not a replacement for trajectory evidence.

### Principle 3: Component-wise Optimization

**Independent tracks**: Two harness variants evolve in parallel, each restricted to one component group (structural vs. guidance). Both are evaluated; the higher Pass@1 wins.

**R³ merge procedure** (from winner as base):
1. **Revision**: Retain only loser edits that are independently beneficial, non-duplicative, and non-conflicting with the winner
2. **Recombination**: Apply retained edits with winner taking precedence in file-level conflicts
3. **Refinement**: Remove redundant edits implementing overlapping functionality, keeping the more reliable component's modification

## Empirical Validation / Results

### Main Results

**Table 1: Main results on SWE-bench Verified with GPT-5.4 (non-thinking mode)**

| Method | Sample | Turns | Held-Out | Total |
|--------|--------|-------|----------|-------|
| $\mathcal{H}_0$ (seed) | 54.0% | 0 | 51.6% | 51.8% |
| AHE | 63.0% | 20 | 54.7% | 55.5% |
| **HarnessCompass** | **66.0%** | **5** | **60.4%** | **61.0%** |

Key observations:
- HarnessCompass outperforms AHE by 3 points on the evolution sample and **5.7 points on held-out tasks** (wider gap = stronger generalization)
- Requires only 5 turns vs. AHE's 20 turns (4× more efficient)
- All role agents share one base model; both methods start from identical $\mathcal{H}_0$, isolating the effect of harness edits

### Ablation Study

**Table 2: Progressive ablation (adding one principle at a time)**

| Configuration | Sample | Turns | Held-Out | Total |
|---------------|--------|-------|----------|-------|
| $\mathcal{H}_0$ (seed) | 54.0% | 0 | 51.6% | 51.8% |
| + Generalization Gate | 62.0% | 2 | 58.4% | 58.8% |
| + Proactive Feedback | 66.0% | 12 | 55.8% | 56.8% |
| + R³ Integration | 66.0% | 5 | 60.4% | 61.0% |

Key findings:
- **Generalization gate alone** improves seed by 8 points in only 2 turns, with strong held-out gains (51.6% → 58.4%)
- **Proactive feedback** raises sample to 66.0% but *hurts* generalization without safeguards (held-out drops to 55.8%, turns increase to 12)—feedback-driven edits may overfit
- **R³ integration** resolves the tension: maintains 66.0% while restoring held-out to 60.4% and reducing turns to 5

### Cross-Model Generalization

**Table 3: Cross-model transfer (harness evolved with GPT-5.4, frozen, evaluated on Claude-Sonnet-4.6)**

| Base model | Harness | Sample | Held-Out | Total |
|------------|---------|--------|----------|-------|
| GPT-5.4 | $\mathcal{H}_0$ | 54.0% | 51.6% | 51.8% |
| GPT-5.4 | Ours | 66.0% | 60.4% | 61.0% |
| Claude-Sonnet-4.6 | $\mathcal{H}_0$ | 68.0% | 70.2% | 70.0% |
| Claude-Sonnet-4.6 | Ours | 76.0% | 73.6% | 73.8% |

The frozen harness improves a model it was never evolved on (70.0% → 73.8% total), demonstrating that HarnessCompass learns **reusable engineering improvements** rather than model-specific fixes.

### Evolution Efficiency

Per-iteration Pass@1 curves show:
- HarnessCompass climbs steeply, peaking around iteration 6
- AHE improves slowly, leveling off below HarnessCompass after ~20 iterations
- Two mechanisms explain this: the generalization gate prevents give-back from per-task fixes, and R³ compounds each round's exploration by folding in the losing track's useful edits

### Per-Repository Analysis

Gains are broadly distributed across repositories, concentrated on harness-sensitive ones:
- **Sphinx-doc**: 35.2% → 58.0%
- **pytest-dev**: 52.6% → 68.4%
- **django** (largest repo, 231 tasks): 59.3% → 70.1%
- **scikit-learn** (already strong at 73.4%): barely moves
- Same pattern holds for Claude-Sonnet-4.6 (e.g., astropy: 45.5% → 63.6%), confirming cross-model transfer at the repository level

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Discipline over search**: The paper demonstrates that *how* evolution is constrained matters as much as the search itself. Unrestricted search invites overfitting; disciplined evolution with explicit constraints produces both better and more transferable results.

2. **First-person evidence**: Proactive feedback introduces a novel evidence modality—the agent's own experience of harness friction—that trajectory-derived signals alone cannot provide, while grounding ensures reliability.

3. **Component isolation**: The R³ merge procedure provides a principled approach to combining independently optimized components, addressing the interference problem inherent in joint optimization.

### Practical Implications

1. **Cost efficiency**: Reaching higher performance in 5 turns vs. 20 turns represents substantial computational savings, as each turn requires full benchmark evaluation.

2. **Transferability**: The evolved harness transfers across both tasks and base models, meaning harness improvements are reusable assets rather than task-specific artifacts.

3. **Model-agnostic improvements**: The cross-model results suggest that certain harness design principles are broadly beneficial regardless of the underlying model's reasoning style.

4. **Benchmark design**: The 50/450 task split methodology offers a template for evaluating generalization in harness evolution, addressing the evaluation gap identified in prior work.

## Conclusion

HarnessCompass addresses three fundamental limitations of automatic harness evolution—search-task overfitting, insufficient feedback, and cross-component interference—through three complementary principles:

- **Constrained evolution** (generalization gate) promotes transferable, task-agnostic edits
- **Proactive feedback** (grounded first-person reports) expands the optimization signal beyond trajectory outcomes
- **Component-wise optimization** (R³ integration) preserves complementary gains while removing redundant or non-transferable edits

The framework achieves stronger effectiveness (66% vs. 54% seed), superior generalization (60.4% held-out vs. AHE's 54.7%), and higher evolution efficiency (5 vs. 20 turns) on SWE-bench Verified, with successful cross-model transfer to Claude-Sonnet-4.6.

**Key takeaway**: "Disciplined evolution is as important as search itself," motivating future work toward harnesses that are both effective and broadly generalizable.

---

_Markdown view of https://picx.dev/p/pfX4Qf, served by PicX — AI-generated visual whiteboard summaries of research papers._