Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

Summary (Overview)

• Introduces Harness-of-Harness (HoH), a framework that enables LLM-based coding agents to perform autonomous, multi-day software development from high-level requirements without human intervention, by organizing existing coding-agent harnesses into iterative planning–coding–testing loops.
• Achieves consistent improvements across three benchmarks (GameCraft-Bench, FrontierSWE, ProgramBench) and three harness–model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, Pi with MiniMax-M3), with an average relative gain of 52.25% and a maximum gain of 82.86% after three iterations.
• Demonstrates sustained improvement over long horizons: On FrontierSWE, HoH with Codex and GPT-5.5 (high) improves from 22% to 72.67% over ten iterations, showing that gains continue well beyond the initial three loops.
• Showcases multi-day autonomous development: In a 70-loop deployment, HoH autonomously develops "Fusepoint," a complete first-person-shooter game with coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals, and integrated audio.
• Key design principles: Balances repair with capability growth, scopes development into small verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows.

Introduction and Theoretical Foundation

Background and Motivation

Software development has become a prominent application of large language models (LLMs), with coding agents progressing from localized assistance (function completion) to complex repository-level tasks. However, most coding agents still operate under a human-in-the-loop paradigm where developers must define tasks, guide intermediate decisions, review changes, and intervene during failures.

The paper pursues a more ambitious goal: autonomous software development—given only high-level requirements, agents independently transform them into complete, functional, and deployable software systems without further human guidance.

The Long-Horizon Challenge

Autonomous development poses a fundamentally longer-horizon problem than conventional agentic coding. Building software from scratch requires:

• Translating high-level requirements into executable plans
• Coordinating interdependent tasks
• Designing and integrating components
• Continuously testing and debugging the evolving system

As trajectories grow, agents face several challenges:

• Loss of context: Agents may lose track of earlier requirements and design decisions
• Constraint violations: Local fixes may violate constraints elsewhere
• Error accumulation: Failed attempts and suboptimal decisions accumulate
• Invalidated assumptions: New evidence from testing can invalidate earlier assumptions
• Repetitive cycles: Inspection and repair loops, redundant verification, premature completion declarations

The authors argue that autonomous software development is not simply a problem of longer execution—the real challenge is sustaining coherent and effective progress over time.

Theoretical Foundation

HoH builds on the principle of iterative and incremental development from software engineering. Modern coding agents operate within a harness—the surrounding system providing tools, managing execution, and mediating LLM interaction with the development environment. HoH operates on top of these harnesses rather than replacing them.

The framework is grounded in the insight that end-to-end software development requires three distinct decision types:

1. What to change next (requires project-level view)
2. How to realize the change (requires write access and technical autonomy)
3. Whether the result satisfies requirements (requires independent assessment)

Methodology

Problem Formulation

Given a software specification $S$, the task is to construct a complete software artifact $A$ satisfying functional and quality requirements. Let $M$ denote a language model and $H$ the coding harness. HoH applies a fixed harness–model configuration:

$$\mathrm{HoH}_{M, H}: \mathcal{S} \longmapsto A. \tag{1}$$

Cross-Loop State Management

HoH maintains two complementary states across loop boundaries:

• Artifact state $A_t$: The current implementation after loop $t$ (source code, configuration, resources, project metadata)
• Evidence state $\mathcal{E}_t$: Validated knowledge obtained by evaluating $A_t$ against the specification (verified behaviors, unsupported claims, observed failures)

The transition across loop $t$ is:

$$\left(A_{t-1}, \mathcal{E}_{t-1}\right) \xrightarrow{\text{loop } t \text{ under } \mathcal{S}} \left(A_{t}, \mathcal{E}_{t}\right). \tag{2}$$

Neither state subsumes the other: $A_t$ supplies the implementation, while $\mathcal{E}_t$ supplies validated project knowledge to direct development.

Three-Role Architecture

HoH assigns responsibilities to three distinct roles, each instantiated as a separate invocation of the same fixed harness–model configuration:

Algorithm

Algorithm 1 Harness-of-Harness
Input: specification S, initial artifact A₀, iteration budget T
Fixed: model M, harness H, and role contracts
Output: final artifact A_T

1:  E₀ ← ∅
2:  for t = 1, ..., T do
3:      Dₜ ← ProjectPlanner(S, Eₜ₋₁; read_only(Aₜ₋₁))
4:      Aₜ ← Developer(Aₜ₋₁; S, Dₜ)
5:      Eₜ ← QATester(read_only(Aₜ); S, Dₜ, Runtime.check(Aₜ))
6:  end for
7:  return A_T

Key Mechanisms

1. Iterative Planning–Coding–Testing: Each loop produces a bounded software increment, verifies the result, and carries both candidate and evidence forward.

2. Progressive Disclosure: Plans, reports, and histories are persisted in the file system, exposed through concise categorized indexes, with details retrieved only when relevant—avoiding context window overflow.

3. Role-Specific Tools and Skills: Tools (MCP servers, expert models, domain-specific algorithms) are organized by role, with lightweight Markdown-based skills providing on-demand guidance.

4. Versioned Project History: Software state is preserved with concise accounts of changes, enabling rollback to verified states after regressions and informed diagnosis of recurring failures.

5. Constrained Outputs, Not Workflows: Each role must return structured artifacts; schema violations trigger retries, but agents retain autonomy over reasoning, tool use, and implementation strategy.

Empirical Validation / Results

Benchmark Evaluation Setup

• GameCraft-Bench: 45 tasks sampled from 15 game families (stratified random sampling), scored 0–100 on Overall score combining Core Mechanics, Content Depth, Functional Visuals, and Art and Presentation
• FrontierSWE: 15 of 17 tasks (4 Implementation, 9 Performance, 2 Research), reporting mean reward and official dominance score
• ProgramBench: Cleanroom program-reconstruction benchmark, reporting Avg. Test Pass Rate

Main Results (HoH@3 vs. Vanilla)

Dominance scores on FrontierSWE improved from 44% to 71% (Codex), 25% to 44% (OpenCode), and 35% to 64% (Pi).

Sustained Improvement Over Ten Loops

On FrontierSWE with Codex + GPT-5.5 (high), dominance increased from 39.33% at HoH@3 to 72.67% at HoH@10, reaching 76.00% at HoH@9, versus 27.33% for Vanilla—a 45.34-point improvement over baseline.

Pass-Controlled Comparison

To distinguish HoH's contribution from merely running more passes, HoH was compared against Vanilla Continuation:

Notably, HoH@2 (64.84 with 5.67M tokens) exceeds three-pass Vanilla Continuation (58.24 with 6.33M tokens), showing HoH produces higher quality with comparable or better inference budgets.

Ablation Study

All ablations underperform Full HoH, confirming that plan revision, evidence feedback, and artifact warm-start each contribute meaningfully to performance.

Multi-Day FPS Game Development (Fusepoint)

In a 70-loop autonomous deployment with Codex CLI and GPT-5.6-Sol:

• Three development phases observed: Initial construction (Loops 1–27, establishing executable project), capability expansion (Loops 28–49, adding functionality while repairing), and stabilization (active backlog declining)
• Issue tracking: 65 of 81 recorded issues closed by Loop 70; 17 issues reopened after regression from later changes
• Human involvement: Limited to restoring network/API availability—no planning, implementation, debugging, testing, or acceptance
• Deliverable: Complete single-player narrative FPS with two control points, three-stage defusal, 18 enemies (distributed 3/5/10 across regions), distinct success/detonation branches, polished visuals, and integrated audio

Theoretical and Practical Implications

Theoretical Contributions

1. Harness-of-Harness as a meta-layer: Just as a coding harness structures model operation, HoH structures harness participation in long-horizon development—a new abstraction level for autonomous software engineering.

2. Evidence-grounded orchestration: The separation of artifact state from evidence state provides a principled framework for maintaining continuity in long-horizon agentic tasks, addressing context-window limitations without dedicated memory modules.

3. Role separation as architectural principle: The strict separation of planning, development, and independent QA—with read-only/write-only boundaries—prevents the collapse of implementation and acceptance into a single decision process.

4. Iterative and incremental development formalized: HoH operationalizes software engineering principles (shift-left testing, single-writer boundaries, iterative development) within an autonomous agent framework.

Practical Implications

1. Vendor-agnostic improvement: HoH works across different harness–model configurations, improving even the strongest baselines (Codex + GPT-5.5) and substantially boosting weaker ones (OpenCode + DeepSeek-V4-Pro).

2. Token efficiency: Pass-controlled comparisons show HoH produces better artifacts than repeated vanilla development at comparable or lower token budgets.

3. Real-world viability: The 70-loop Fusepoint case demonstrates that autonomous agents can sustain coherent multi-day development projects with versioned histories, issue tracking, and regression management.

4. Reusable skills and tools: The framework's role-specific tools and lightweight skills (asset generation, UI/UX presentation, testing) provide a template for domain-specific autonomous development.

Conclusion

Main Takeaways

• HoH enables autonomous software development by organizing fixed coding-agent harnesses into iterative planning–coding–testing loops with persistent artifact and evidence states.
• It achieves substantial, consistent gains across diverse benchmarks and configurations, with improvements continuing over ten or more iterations.
• The framework successfully handles multi-day, open-ended development tasks, producing a complete, human-playable game from scratch.

Future Directions

• Extend HoH to a broader range of real-world development scenarios (different game types, other software systems)
• Develop toward a general framework for autonomous software development
• Investigate additional mechanisms for sustaining improvement over even longer horizons
• Explore applications in domains beyond game development where long-horizon autonomous creation is valuable