One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models

Summary (Overview)

• This paper investigates what self-evolving harnesses actually encode by holding an evolution recipe fixed across a grid of 8 programming languages (Multi-SWE-Bench) and 3 base models (Claude Haiku 4.5, GPT-5-mini, DeepSeek-V4-Flash), forming a 24-cell experimental grid.
• The authors introduce TRIAGE (Typed Routing and Instrumented Attribution for Guided Evolution), a diagnostic-routed framework where every harness edit is linked to a falsifiable failure signal, making modifications attributable after evolution.
• Four key findings emerge: (1) The loop improves held-out solve rates over both minimal seed and mini-SWE-agent in most cells, with two null regions (Python across all models, and GPT-5-mini across all languages); (2) Gains compensate recoverable execution defects (test-file edits, non-compiling source, missing test runs); (3) Evolved harnesses share an abstract playbook across languages but instantiate it with nearly disjoint language-ecosystem machinery; (4) The shared disciplinary core transfers via transplantation and distillation, but an ecosystem margin (20-40% of each harness) resists both and requires native re-evolution.
• The paper recasts evolved harnesses as legible compensation layers—shaped jointly by language engineering demands and model behavioral gaps—rather than opaque benchmark-tuned scaffolds.

Introduction and Theoretical Foundation

Background and Motivation

Coding-agent performance depends not only on base models but also on harnesses—the editable prompts, tools, memory, and workflow scaffolding through which agents act. As base models advance, harness engineering has become an important optimization target: the same base model can improve substantially depending on the surrounding harness.

However, prior work reports only aggregate gains from self-evolving harnesses without analyzing what the evolved artifacts encode. Three competing hypotheses remain untested:

1. Harnesses encode benchmark-specific adaptations (overfitting to evaluated instances)
2. Harnesses encode language-specific engineering knowledge
3. Harnesses encode compensation for limitations of the underlying model

Theoretical Foundation

The paper builds on the concept of externalized tuning—optimizing textual and program-level artifacts rather than model weights. Key related work includes:

• AHE (Agentic Harness Engineering): observability-driven automatic evolution of coding-agent harnesses
• DGM (Darwin Gödel Machine): open-ended evolution of self-improving agents
• HarnessFix: diagnoses failures within a seven-layer harness taxonomy
• DSPy: represents LM pipelines as declarative programs

The authors argue that distinguishing between these hypotheses requires holding the evolution recipe fixed while varying language and model, then analyzing the resulting artifacts. Programming language serves as an ideal analytical axis because language ecosystems impose concrete, comparable constraints (build, test, and dependency tooling), and fixing the code-repair domain reduces task-level variation.

Methodology

TRIAGE: The Evolution Framework

The formal setup: For a language ecosystem $L$, frozen task policy $\pi$, and outer-loop driver $\mu$, the finite-round output is:

$$H_{L, \pi, \mu}^{(R)} = \mathrm{Evolve}^{R}\big(H_{0}, D_{\mathrm{evo}}^{L}; \pi, \mu\big).\tag{1}$$

The harness exposes four versioned, file-level slots: $H = (P, W, M, T)$:

• P: system prompt (persistent behavior)
• W: workflow hooks (state-dependent guidance)
• M: lessons memory (reusable cross-instance lessons, capacity-bounded to ~12 entries of ≤400 chars)
• T: tool implementations (executable capabilities)

Key Mechanisms

Typed Attribution Schema: Each rollout receives a deterministic bucket assigned by fixed priority order:

1. model_incompat — API-layer failure
2. budget_exceeded — step/cost/time ceiling
3. patch_broke_tests — test-file edit colliding with gold test patch
4. unwinnable — reference patch too large
5. patch_pollution — build artifacts in patch
6. no_build — project never built
7. wrong_fix — default: patch applied, test ran, still wrong

Edit Contracts: Every round writes a machine-readable manifest with five fields: failure_evidence, root_cause, targeted_fix, predicted_impact, and changed_fields. The prediction commits to task-level outcomes before observation.

Selection Score:

$$\tilde{s}_{r} = s_{r} - \lambda \rho_{r}, \quad \lambda = 2.0$$

where $s_r$ is the round-$r$ score and $\rho_r$ is the fraction of previously-solved instances that regressed. A non-regression gate floors the shipped harness at the seed.

Experimental Setup

• Benchmark: Multi-SWE-Bench with 8 languages: {cpp, c, java, rust, typescript, javascript, go, python}
• Splits: 20 instances for evolution ($D_{\mathrm{evo}}^L$), 50 held-out for testing ($D_{\mathrm{test}}^L$), disjoint
• Models: Claude Haiku 4.5, GPT-5-mini, DeepSeek-V4-Flash (same model serves as policy π and driver μ)
• Metric: mean_solve@3 (mean solve rate across $k=3$ independent rollouts per held-out instance)
• Budgets: 80 agent steps max, $1.5 cost ceiling per rollout,$R=3$ evolution rounds

Empirical Validation / Results

C1: Effectiveness

The evolved harness improves over the minimal seed in most cells and matches/exceeds mini-SWE-agent in 14 of 24 cells. Two null regions emerge:

C2: Compensation Mechanism

Key quantitative results (Table 3, selected cells):

Defect composition varies by language (Figure 3): Non-compiling changes account for 70% of C++ rollout defects but under 10% of Go's. Test-file edits dominate elsewhere.

Recovery decomposition: Of 256 recovered held-out instances across the grid, 88% carried a detector-covered defect under the seed (81% test-path edits, 7% build breaks).

C3: Cell-Specificity

• Abstract concept Jaccard overlap: mean 0.55 (Haiku), 0.57 (DeepSeek)—moderate overlap, a shared generic playbook
• Ecosystem-marker Jaccard overlap: mean 0.12 (Haiku), 0.14 (DeepSeek)—near-zero overlap, a factor of ~4 reduction
• Ecosystem-specific fraction: 20-40% of each harness (mean 0.26 Haiku, 0.29 DeepSeek)

C4: Bounded Portability

Universal Distillation (retention $\rho = (u-m)/(n-m)$ where $m$=seed score, $n$=native score, $u$=transferred score):

Cross-Language Transplantation (DeepSeek-V4-Flash, 5×5 matrix):

• Transfer positive in 18/20 off-diagonal cells, mean retention 0.63
• Retention below 1 in 14/20 cells; full recovery in only 6/20
• The single strongly negative cell: JS→TS (−2.30)—JavaScript harness lacks compile-verification since interpreted JS has nothing to compile, but TypeScript's test runner transpiles without type-checking, so the gap opens

Three independent measurements converge on the same ecosystem margin: textual (20-40% ecosystem-specific content), distillation (32-52% unrecovered gain on ecosystem-heavy targets), and transplantation (roughly a third not supplied by any source).

Theoretical and Practical Implications

Theoretical Implications

1. Harnesses are compensation layers, not capability extensions: The paper finds no recovered instance where the evolved harness enables a repair the base policy could not otherwise express. The harness closes the gap between what a policy can do and what it does.

2. Language determines which disciplines matter: Compiled targets elicit build-verification concepts that interpreted ones never raise. The dominant defect is a property of the cell, not the method.

3. Model capability is qualitative, not just quantitative: GPT-5-mini is the weakest model under mini-SWE-agent but shows no harness gains—suggesting models possess qualitative properties independent of capability level.

4. The meta-agent's strength is not the driver: An ablation (Table 7) shows even an ultra-weak 20B open model (GPT-OSS-20B) recovers the same gains (34.2% vs 35.6% for Claude Sonnet 4.6) as a frontier model, confirming diagnostic routing—not external knowledge—governs harness evolution.

Practical Implications

1. Defect profiling as a leading indicator: A base policy's measured defect profile is a cheap predictor of where harness engineering pays off.

2. When to reuse vs. re-evolve:

   • Reuse/distill for generic targets (Go, JavaScript)
   • Native re-evolution remains necessary for complex ecosystems (Java, C++, TypeScript)

3. Cost efficiency: Meta-model spend is ~5% of total ($256 vs$3,667 rollout spend across ~15k rollouts, 737k agent steps), making harness evolution cheap relative to weight-space adaptation.

Conclusion

The paper characterizes self-evolving harnesses as legible compensation layers—the language determines which disciplines a task requires, model capability determines how many the base policy already supplies, and evolution installs the difference. The evolved harness improves how a model applies its existing capabilities, with required guidance shaped jointly by model behavior and language ecosystem.

Key limitations acknowledged:

• Single task domain (code repair), single evolution recipe, one run per setting
• No head-to-head comparison with related harness-optimization methods
• Defect detectors cover only part of the recoverable space
• Small splits (20 evolution / 50 held-out instances per language)

Future directions: Whether the decomposition into a portable disciplinary component and an ecosystem-bound remainder survives at longer horizons remains open. The loop shows effectiveness at recall (recognizing recurring failure modes) but no evidence of discovery of novel, environment-specific operational knowledge.