# One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models

> Self-evolving coding harnesses are legible compensation layers, shaped by language-specific defect profiles and model capability gaps, not opaque benchmark-tuned scaffolds.

- **Source:** [arXiv](https://arxiv.org/abs/2608.10178)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/XjV02V
- **Whiteboard:** https://picx.dev/p/XjV02V/image

## Summary

# One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models

## Summary (Overview)

- This paper investigates **what self-evolving harnesses actually encode** by holding an evolution recipe fixed across a grid of 8 programming languages (Multi-SWE-Bench) and 3 base models (Claude Haiku 4.5, GPT-5-mini, DeepSeek-V4-Flash), forming a 24-cell experimental grid.
- The authors introduce **TRIAGE** (Typed Routing and Instrumented Attribution for Guided Evolution), a diagnostic-routed framework where every harness edit is linked to a falsifiable failure signal, making modifications attributable after evolution.
- **Four key findings emerge**: (1) The loop improves held-out solve rates over both minimal seed and mini-SWE-agent in most cells, with two null regions (Python across all models, and GPT-5-mini across all languages); (2) Gains compensate **recoverable execution defects** (test-file edits, non-compiling source, missing test runs); (3) Evolved harnesses share an abstract playbook across languages but instantiate it with nearly disjoint language-ecosystem machinery; (4) The shared disciplinary core transfers via transplantation and distillation, but an **ecosystem margin** (20-40% of each harness) resists both and requires native re-evolution.
- The paper recasts evolved harnesses as **legible compensation layers**—shaped jointly by language engineering demands and model behavioral gaps—rather than opaque benchmark-tuned scaffolds.

## Introduction and Theoretical Foundation

### Background and Motivation

Coding-agent performance depends not only on base models but also on **harnesses**—the editable prompts, tools, memory, and workflow scaffolding through which agents act. As base models advance, harness engineering has become an important optimization target: the same base model can improve substantially depending on the surrounding harness.

However, prior work reports only **aggregate gains** from self-evolving harnesses without analyzing what the evolved artifacts encode. Three competing hypotheses remain untested:

1. Harnesses encode **benchmark-specific adaptations** (overfitting to evaluated instances)
2. Harnesses encode **language-specific engineering knowledge**
3. Harnesses encode **compensation for limitations of the underlying model**

### Theoretical Foundation

The paper builds on the concept of **externalized tuning**—optimizing textual and program-level artifacts rather than model weights. Key related work includes:

- **AHE** (Agentic Harness Engineering): observability-driven automatic evolution of coding-agent harnesses
- **DGM** (Darwin Gödel Machine): open-ended evolution of self-improving agents
- **HarnessFix**: diagnoses failures within a seven-layer harness taxonomy
- **DSPy**: represents LM pipelines as declarative programs

The authors argue that distinguishing between these hypotheses requires **holding the evolution recipe fixed while varying language and model**, then analyzing the resulting artifacts. Programming language serves as an ideal analytical axis because language ecosystems impose concrete, comparable constraints (build, test, and dependency tooling), and fixing the code-repair domain reduces task-level variation.

## Methodology

### TRIAGE: The Evolution Framework

The formal setup: For a language ecosystem $L$, frozen task policy $\pi$, and outer-loop driver $\mu$, the finite-round output is:

$$
H_{L, \pi, \mu}^{(R)} = \mathrm{Evolve}^{R}\big(H_{0}, D_{\mathrm{evo}}^{L}; \pi, \mu\big).\tag{1}
$$

The harness exposes four versioned, file-level slots: $H = (P, W, M, T)$:
- **P**: system prompt (persistent behavior)
- **W**: workflow hooks (state-dependent guidance)
- **M**: lessons memory (reusable cross-instance lessons, capacity-bounded to ~12 entries of ≤400 chars)
- **T**: tool implementations (executable capabilities)

### Key Mechanisms

**Typed Attribution Schema**: Each rollout receives a deterministic bucket assigned by fixed priority order:
1. `model_incompat` — API-layer failure
2. `budget_exceeded` — step/cost/time ceiling
3. `patch_broke_tests` — test-file edit colliding with gold test patch
4. `unwinnable` — reference patch too large
5. `patch_pollution` — build artifacts in patch
6. `no_build` — project never built
7. `wrong_fix` — default: patch applied, test ran, still wrong

**Edit Contracts**: Every round writes a machine-readable manifest with five fields: `failure_evidence`, `root_cause`, `targeted_fix`, `predicted_impact`, and `changed_fields`. The prediction commits to task-level outcomes before observation.

**Selection Score**:

$$
\tilde{s}_{r} = s_{r} - \lambda \rho_{r}, \quad \lambda = 2.0
$$

where $s_r$ is the round-$r$ score and $\rho_r$ is the fraction of previously-solved instances that regressed. A non-regression gate floors the shipped harness at the seed.

### Experimental Setup

- **Benchmark**: Multi-SWE-Bench with 8 languages: {cpp, c, java, rust, typescript, javascript, go, python}
- **Splits**: 20 instances for evolution ($D_{\mathrm{evo}}^L$), 50 held-out for testing ($D_{\mathrm{test}}^L$), disjoint
- **Models**: Claude Haiku 4.5, GPT-5-mini, DeepSeek-V4-Flash (same model serves as policy π and driver μ)
- **Metric**: mean_solve@3 (mean solve rate across $k=3$ independent rollouts per held-out instance)
- **Budgets**: 80 agent steps max, $1.5 cost ceiling per rollout, $R=3$ evolution rounds

## Empirical Validation / Results

### C1: Effectiveness

The evolved harness improves over the minimal seed in most cells and matches/exceeds mini-SWE-agent in 14 of 24 cells. **Two null regions** emerge:

| Null Region | Cause |
|---|---|
| **Python** (all models) | Base policies already follow the disciplines detectors cover (defect rate 5.6%, lowest of 8 languages) |
| **GPT-5-mini** (all languages) | Commits few detectable defects; evolved memory contains no test-file rule where other models install one |

### C2: Compensation Mechanism

**Key quantitative results** (Table 3, selected cells):

| Model | Lang | $H_0$ | mini-SWE | TRIAGE | $\Delta$ vs $H_0$ |
|---|---|---|---|---|---|
| Haiku 4.5 | C++ | 12.9 | 61.9 | 87.1 | +74.3 |
| Haiku 4.5 | Java | 2.9 | 51.4 | 82.9 | +80.0 |
| Haiku 4.5 | Python | 75.0 | 72.6 | 73.2 | -1.8 |
| GPT-5-mini | C++ | 22.9 | 27.6 | 27.1 | +4.3 |
| DS-V4-Flash | C++ | 34.3 | 60.0 | 65.7 | +31.4 |
| DS-V4-Flash | Python | 73.2 | 71.4 | 66.1 | -7.1 |

**Defect composition varies by language** (Figure 3): Non-compiling changes account for 70% of C++ rollout defects but under 10% of Go's. Test-file edits dominate elsewhere.

**Recovery decomposition**: Of 256 recovered held-out instances across the grid, **88% carried a detector-covered defect** under the seed (81% test-path edits, 7% build breaks).

### C3: Cell-Specificity

- **Abstract concept Jaccard overlap**: mean 0.55 (Haiku), 0.57 (DeepSeek)—moderate overlap, a shared generic playbook
- **Ecosystem-marker Jaccard overlap**: mean 0.12 (Haiku), 0.14 (DeepSeek)—near-zero overlap, a factor of ~4 reduction
- **Ecosystem-specific fraction**: 20-40% of each harness (mean 0.26 Haiku, 0.29 DeepSeek)

### C4: Bounded Portability

**Universal Distillation** (retention $\rho = (u-m)/(n-m)$ where $m$=seed score, $n$=native score, $u$=transferred score):

| Target | Retention ρ |
|---|---|
| Go | 1.08 |
| JavaScript | 1.13 |
| Java | 0.68 |
| C++ | 0.65 |
| TypeScript | 0.48 |

**Cross-Language Transplantation** (DeepSeek-V4-Flash, 5×5 matrix):
- Transfer positive in **18/20 off-diagonal cells**, mean retention 0.63
- Retention below 1 in **14/20 cells**; full recovery in only 6/20
- The single strongly negative cell: **JS→TS (−2.30)**—JavaScript harness lacks compile-verification since interpreted JS has nothing to compile, but TypeScript's test runner transpiles without type-checking, so the gap opens

**Three independent measurements converge** on the same ecosystem margin: textual (20-40% ecosystem-specific content), distillation (32-52% unrecovered gain on ecosystem-heavy targets), and transplantation (roughly a third not supplied by any source).

## Theoretical and Practical Implications

### Theoretical Implications

1. **Harnesses are compensation layers, not capability extensions**: The paper finds no recovered instance where the evolved harness enables a repair the base policy could not otherwise express. The harness closes the gap between *what a policy can do* and *what it does*.

2. **Language determines which disciplines matter**: Compiled targets elicit build-verification concepts that interpreted ones never raise. The dominant defect is a property of the cell, not the method.

3. **Model capability is qualitative, not just quantitative**: GPT-5-mini is the *weakest* model under mini-SWE-agent but shows no harness gains—suggesting models possess qualitative properties independent of capability level.

4. **The meta-agent's strength is not the driver**: An ablation (Table 7) shows even an ultra-weak 20B open model (GPT-OSS-20B) recovers the same gains (34.2% vs 35.6% for Claude Sonnet 4.6) as a frontier model, confirming diagnostic routing—not external knowledge—governs harness evolution.

### Practical Implications

1. **Defect profiling as a leading indicator**: A base policy's measured defect profile is a cheap predictor of where harness engineering pays off.

2. **When to reuse vs. re-evolve**: 
   - Reuse/distill for generic targets (Go, JavaScript)
   - Native re-evolution remains necessary for complex ecosystems (Java, C++, TypeScript)

3. **Cost efficiency**: Meta-model spend is ~5% of total ($256 vs $3,667 rollout spend across ~15k rollouts, 737k agent steps), making harness evolution cheap relative to weight-space adaptation.

## Conclusion

The paper characterizes self-evolving harnesses as **legible compensation layers**—the language determines which disciplines a task requires, model capability determines how many the base policy already supplies, and evolution installs the difference. The evolved harness improves how a model applies its existing capabilities, with required guidance shaped jointly by model behavior and language ecosystem.

**Key limitations acknowledged**:
- Single task domain (code repair), single evolution recipe, one run per setting
- No head-to-head comparison with related harness-optimization methods
- Defect detectors cover only part of the recoverable space
- Small splits (20 evolution / 50 held-out instances per language)

**Future directions**: Whether the decomposition into a portable disciplinary component and an ecosystem-bound remainder survives at longer horizons remains open. The loop shows effectiveness at *recall* (recognizing recurring failure modes) but no evidence of *discovery* of novel, environment-specific operational knowledge.

---

_Markdown view of https://picx.dev/p/XjV02V, served by PicX — AI-generated visual whiteboard summaries of research papers._