# Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

> Harness-of-Harness enables multi-day autonomous software development by organizing coding agents into iterative planning-coding-testing loops, achieving 52.25% average relative gains across benchmarks.

- **Source:** [arXiv](https://arxiv.org/abs/2609.01481)
- **Published:** 2026-09-05
- **Permalink:** https://picx.dev/p/m6Zzv4
- **Whiteboard:** https://picx.dev/p/m6Zzv4/image

## Summary

# Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

## Summary (Overview)

- **Introduces Harness-of-Harness (HoH)**, a framework that enables LLM-based coding agents to perform autonomous, multi-day software development from high-level requirements without human intervention, by organizing existing coding-agent harnesses into iterative planning–coding–testing loops.
- **Achieves consistent improvements across three benchmarks** (GameCraft-Bench, FrontierSWE, ProgramBench) and three harness–model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, Pi with MiniMax-M3), with an average relative gain of **52.25%** and a maximum gain of **82.86%** after three iterations.
- **Demonstrates sustained improvement over long horizons**: On FrontierSWE, HoH with Codex and GPT-5.5 (high) improves from 22% to 72.67% over ten iterations, showing that gains continue well beyond the initial three loops.
- **Showcases multi-day autonomous development**: In a 70-loop deployment, HoH autonomously develops "Fusepoint," a complete first-person-shooter game with coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals, and integrated audio.
- **Key design principles**: Balances repair with capability growth, scopes development into small verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows.

## Introduction and Theoretical Foundation

### Background and Motivation

Software development has become a prominent application of large language models (LLMs), with coding agents progressing from localized assistance (function completion) to complex repository-level tasks. However, most coding agents still operate under a **human-in-the-loop** paradigm where developers must define tasks, guide intermediate decisions, review changes, and intervene during failures.

The paper pursues a more ambitious goal: **autonomous software development**—given only high-level requirements, agents independently transform them into complete, functional, and deployable software systems without further human guidance.

### The Long-Horizon Challenge

Autonomous development poses a fundamentally longer-horizon problem than conventional agentic coding. Building software from scratch requires:
- Translating high-level requirements into executable plans
- Coordinating interdependent tasks
- Designing and integrating components
- Continuously testing and debugging the evolving system

As trajectories grow, agents face several challenges:
- **Loss of context**: Agents may lose track of earlier requirements and design decisions
- **Constraint violations**: Local fixes may violate constraints elsewhere
- **Error accumulation**: Failed attempts and suboptimal decisions accumulate
- **Invalidated assumptions**: New evidence from testing can invalidate earlier assumptions
- **Repetitive cycles**: Inspection and repair loops, redundant verification, premature completion declarations

The authors argue that autonomous software development is not simply a problem of longer execution—the real challenge is **sustaining coherent and effective progress over time**.

### Theoretical Foundation

HoH builds on the principle of **iterative and incremental development** from software engineering. Modern coding agents operate within a *harness*—the surrounding system providing tools, managing execution, and mediating LLM interaction with the development environment. HoH operates *on top of* these harnesses rather than replacing them.

The framework is grounded in the insight that end-to-end software development requires three distinct decision types:
1. **What to change next** (requires project-level view)
2. **How to realize the change** (requires write access and technical autonomy)
3. **Whether the result satisfies requirements** (requires independent assessment)

## Methodology

### Problem Formulation

Given a software specification $S$, the task is to construct a complete software artifact $A$ satisfying functional and quality requirements. Let $M$ denote a language model and $H$ the coding harness. HoH applies a fixed harness–model configuration:

$$
\mathrm{HoH}_{M, H}: \mathcal{S} \longmapsto A.
\tag{1}
$$

### Cross-Loop State Management

HoH maintains two complementary states across loop boundaries:

- **Artifact state** $A_t$: The current implementation after loop $t$ (source code, configuration, resources, project metadata)
- **Evidence state** $\mathcal{E}_t$: Validated knowledge obtained by evaluating $A_t$ against the specification (verified behaviors, unsupported claims, observed failures)

The transition across loop $t$ is:

$$
\left(A_{t-1}, \mathcal{E}_{t-1}\right) \xrightarrow{\text{loop } t \text{ under } \mathcal{S}} \left(A_{t}, \mathcal{E}_{t}\right).
\tag{2}
$$

Neither state subsumes the other: $A_t$ supplies the implementation, while $\mathcal{E}_t$ supplies validated project knowledge to direct development.

### Three-Role Architecture

HoH assigns responsibilities to three distinct roles, each instantiated as a separate invocation of the same fixed harness–model configuration:

| Role | Responsibility | Authority | Key Constraint |
|------|---------------|-----------|----------------|
| **Project Planner** | Selects bounded, locally complete development objective | Read-only access to artifact | Cannot modify artifact |
| **Developer** | Implements the planned increment | Write access (single-writer boundary) | Only role that can modify artifact |
| **QA Tester** | Independently evaluates the candidate | Read-only access to frozen candidate | Cannot repair or modify |

### Algorithm

```
Algorithm 1 Harness-of-Harness
Input: specification S, initial artifact A₀, iteration budget T
Fixed: model M, harness H, and role contracts
Output: final artifact A_T

1:  E₀ ← ∅
2:  for t = 1, ..., T do
3:      Dₜ ← ProjectPlanner(S, Eₜ₋₁; read_only(Aₜ₋₁))
4:      Aₜ ← Developer(Aₜ₋₁; S, Dₜ)
5:      Eₜ ← QATester(read_only(Aₜ); S, Dₜ, Runtime.check(Aₜ))
6:  end for
7:  return A_T
```

### Key Mechanisms

1. **Iterative Planning–Coding–Testing**: Each loop produces a bounded software increment, verifies the result, and carries both candidate and evidence forward.

2. **Progressive Disclosure**: Plans, reports, and histories are persisted in the file system, exposed through concise categorized indexes, with details retrieved only when relevant—avoiding context window overflow.

3. **Role-Specific Tools and Skills**: Tools (MCP servers, expert models, domain-specific algorithms) are organized by role, with lightweight Markdown-based skills providing on-demand guidance.

4. **Versioned Project History**: Software state is preserved with concise accounts of changes, enabling rollback to verified states after regressions and informed diagnosis of recurring failures.

5. **Constrained Outputs, Not Workflows**: Each role must return structured artifacts; schema violations trigger retries, but agents retain autonomy over reasoning, tool use, and implementation strategy.

## Empirical Validation / Results

### Benchmark Evaluation Setup

- **GameCraft-Bench**: 45 tasks sampled from 15 game families (stratified random sampling), scored 0–100 on Overall score combining Core Mechanics, Content Depth, Functional Visuals, and Art and Presentation
- **FrontierSWE**: 15 of 17 tasks (4 Implementation, 9 Performance, 2 Research), reporting mean reward and official dominance score
- **ProgramBench**: Cleanroom program-reconstruction benchmark, reporting Avg. Test Pass Rate

### Main Results (HoH@3 vs. Vanilla)

| Configuration | GameCraft-Bench (Overall) | FrontierSWE (Reward) | ProgramBench (Pass Rate†) |
|--------------|--------------------------|---------------------|--------------------------|
| **Codex + GPT-5.5 (high)** | 49.58 → **71.52** (+21.93) | 0.31 → **0.54** | 60.41 → **66.50** (+6.09) |
| **OpenCode + DeepSeek-V4-Pro** | 26.90 → **48.98** (+22.08) | 0.23 → **0.31** | 45.27 → **57.56** (+12.29) |
| **Pi + MiniMax-M3** | 42.16 → **58.78** (+16.62) | 0.26 → **0.55** | 35.83 → **52.68** (+16.85) |

**Dominance scores on FrontierSWE** improved from 44% to 71% (Codex), 25% to 44% (OpenCode), and 35% to 64% (Pi).

### Sustained Improvement Over Ten Loops

On FrontierSWE with Codex + GPT-5.5 (high), dominance increased from **39.33% at HoH@3 to 72.67% at HoH@10**, reaching 76.00% at HoH@9, versus 27.33% for Vanilla—a 45.34-point improvement over baseline.

### Pass-Controlled Comparison

To distinguish HoH's contribution from merely running more passes, HoH was compared against Vanilla Continuation:

| Method | Dev. Passes | Score | Tokens (M) |
|--------|------------|-------|------------|
| Vanilla | 1 | 49.58 | 2.59 |
| Vanilla Continuation | 2 | 54.99 | 4.56 |
| Vanilla Continuation | 3 | 58.24 | 6.33 |
| **HoH** | **1** | **59.71** | **2.88** |
| **HoH** | **2** | **64.84** | **5.67** |
| **HoH** | **3** | **71.52** | **8.41** |

Notably, HoH@2 (64.84 with 5.67M tokens) exceeds three-pass Vanilla Continuation (58.24 with 6.33M tokens), showing HoH produces higher quality with comparable or better inference budgets.

### Ablation Study

| Variant | Score | Tokens (M) |
|---------|-------|------------|
| w/o Plan Update | 63.39 (−8.13) | 7.56 |
| w/o Evidence Feedback | 65.23 (−6.28) | 7.46 |
| w/o Warm-Start | 63.67 (−7.85) | 11.12 |
| **Full HoH@3** | **71.52** | **8.41** |

All ablations underperform Full HoH, confirming that plan revision, evidence feedback, and artifact warm-start each contribute meaningfully to performance.

### Multi-Day FPS Game Development (Fusepoint)

In a 70-loop autonomous deployment with Codex CLI and GPT-5.6-Sol:
- **Three development phases observed**: Initial construction (Loops 1–27, establishing executable project), capability expansion (Loops 28–49, adding functionality while repairing), and stabilization (active backlog declining)
- **Issue tracking**: 65 of 81 recorded issues closed by Loop 70; 17 issues reopened after regression from later changes
- **Human involvement**: Limited to restoring network/API availability—no planning, implementation, debugging, testing, or acceptance
- **Deliverable**: Complete single-player narrative FPS with two control points, three-stage defusal, 18 enemies (distributed 3/5/10 across regions), distinct success/detonation branches, polished visuals, and integrated audio

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Harness-of-Harness as a meta-layer**: Just as a coding harness structures model operation, HoH structures harness participation in long-horizon development—a new abstraction level for autonomous software engineering.

2. **Evidence-grounded orchestration**: The separation of artifact state from evidence state provides a principled framework for maintaining continuity in long-horizon agentic tasks, addressing context-window limitations without dedicated memory modules.

3. **Role separation as architectural principle**: The strict separation of planning, development, and independent QA—with read-only/write-only boundaries—prevents the collapse of implementation and acceptance into a single decision process.

4. **Iterative and incremental development formalized**: HoH operationalizes software engineering principles (shift-left testing, single-writer boundaries, iterative development) within an autonomous agent framework.

### Practical Implications

1. **Vendor-agnostic improvement**: HoH works across different harness–model configurations, improving even the strongest baselines (Codex + GPT-5.5) and substantially boosting weaker ones (OpenCode + DeepSeek-V4-Pro).

2. **Token efficiency**: Pass-controlled comparisons show HoH produces better artifacts than repeated vanilla development at comparable or lower token budgets.

3. **Real-world viability**: The 70-loop Fusepoint case demonstrates that autonomous agents can sustain coherent multi-day development projects with versioned histories, issue tracking, and regression management.

4. **Reusable skills and tools**: The framework's role-specific tools and lightweight skills (asset generation, UI/UX presentation, testing) provide a template for domain-specific autonomous development.

## Conclusion

### Main Takeaways

- HoH enables autonomous software development by organizing fixed coding-agent harnesses into iterative planning–coding–testing loops with persistent artifact and evidence states.
- It achieves substantial, consistent gains across diverse benchmarks and configurations, with improvements continuing over ten or more iterations.
- The framework successfully handles multi-day, open-ended development tasks, producing a complete, human-playable game from scratch.

### Future Directions

- Extend HoH to a broader range of real-world development scenarios (different game types, other software systems)
- Develop toward a general framework for autonomous software development
- Investigate additional mechanisms for sustaining improvement over even longer horizons
- Explore applications in domains beyond game development where long-horizon autonomous creation is valuable

---

_Markdown view of https://picx.dev/p/m6Zzv4, served by PicX — AI-generated visual whiteboard summaries of research papers._