# Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents — A Source-Code Study of Eleven Systems

> Production harnesses replace agentic frameworks and RAG, relying on hand-rolled loops and deterministic retrieval, marking a platform turn in agent design.

- **Source:** [arXiv](https://arxiv.org/abs/2609.00006)
- **Published:** 2026-09-05
- **Permalink:** https://picx.dev/p/S7cZF4
- **Whiteboard:** https://picx.dev/p/S7cZF4/image

## Summary

# Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents

## Summary (Overview)

- **Comprehensive source-code anatomy**: This paper provides the first large-scale empirical foundation for "harness engineering" — the discipline of designing the runtime that couples LLMs to the world — by dissecting eleven production coding harnesses at source level, spanning major commercial providers (Claude Code, Codex, Gemini CLI, Mistral Vibe) and open-source systems (OpenHands, Aider, Mini-SWE-Agent, Hermes, Pi, OpenCode, OpenClaw), plus Omnigent as a meta-harness contrast point.

- **Seven canonical subsystems identified**: The paper defines the harness anatomy as seven subsystems (agent loop, LLM integration, tools & actions, memory & context, safety & permissions, orchestration, extensibility), each with documented minimal and maximal implementations observed across the corpus.

- **Two striking empirical absences**: Across ~4 million lines of code, no agent runtime imports a general-purpose agentic framework (LangChain, LangGraph, AutoGen, etc.), and none uses vector-embedding RAG for code retrieval — the field runs on hand-rolled async loops and deterministic retrieval (ripgrep, tree-sitter, glob).

- **The platform turn thesis**: The harness has completed a transformation from tool to platform — harnesses became importable SDKs, framework vendors shipped harnesses, marketplaces and governance layers appeared, and a meta-harness (Omnigent) now orchestrates eleven vendor harnesses behind one API.

- **Longitudinal evolution captured**: Re-pinning eight systems from April to July 2026 reveals convergence becoming imitation (Codex adopting Claude Code's hook vocabulary verbatim), policy migrating from prompt prose to configuration, and 13 cross-cutting observations plus 29 recurring design patterns cataloged.

## Introduction and Theoretical Foundation

The paper establishes harness engineering as a discipline by first defining its core object. The foundational equation is:

$$\text{Agent} = \text{Model} + \text{Harness}$$

The harness is defined as "everything except the model: the runtime that couples an LLM to the world—its loop, its tools, its context, its safety controls, its orchestration, and its extension surfaces." This definition builds on Macedo's operational definition (the layer that wraps a language model and turns it into an agent able to act on a repository) and extends it with a subsystem decomposition.

The paper carefully distinguishes the harness from four boundary cases:
- **Not a scaffold** (scaffold names the structural code; harness is the shipped runtime artifact)
- **Not an agentic framework** (a framework is imported; a harness is worked inside of — though this distinction is dissolving)
- **Not an evaluation harness** (opposite direction of wrapping)
- **Not an orchestrator** (orchestrators coordinate harnesses from above and implement no editing loop)

The seven canonical subsystems are: agent loop, LLM integration, tools & actions, memory & context, safety & permissions, orchestration, and extensibility. The corpus provides existence proofs at both extremes — Mini-SWE-Agent implements all seven in ~100 lines, while Codex spans ~1.1M lines of Rust.

The theoretical foundation draws on prior work including SWE-Bench, the Agent-Computer Interface (ACI), the CodeAct paradigm, and ReAct's reasoning-action loop, with the key insight that production harnesses operate on a different complexity budget than generic LLM applications.

## Methodology

**System Selection**: Eleven systems chosen for spread along design philosophy, maturity, and market position axes, spanning three programming languages (Python, TypeScript, Rust) and three orders of magnitude in code size. All snapshots pinned to July 2026 releases, with April 2026 snapshots retained for longitudinal comparison.

**Dimensions of Analysis**: Each system analyzed along seven dimensions (D1–D7):
- D1: Agent loop design
- D2: LLM integration and model–agent co-design  
- D3: Tool and action systems
- D4: Memory and context management
- D5: Safety and permission models
- D6: Multi-agent orchestration
- D7: Extensibility mechanisms

**Analytical approach**: The study is descriptive, not benchmark-based. It compares how systems are built, not how they perform. The methodology includes source-diffing across the longitudinal sample, dependency manifest inspection, and systematic grepping for framework imports and vector-store dependencies.

**Key comparison framework**: The paper identifies five fundamental trade-off axes:
1. Simplicity vs. Capability
2. Safety vs. Autonomy
3. Provider Coupling vs. Agnosticism
4. Monolithic vs. Modular
5. Scaffold Complexity vs. Model Capability

## Empirical Validation / Results

The audit yields **13 cross-cutting observations** and a **catalog of 29 recurring design patterns**. Key findings include:

### Agent Loop Taxonomies
Three loop paradigms identified: iterative action-observation (9 systems), reflection-augmented (Aider), and coordinator-worker (Claude Code, Codex, Hermes). The corpus shows that loop sophistication does not predict benchmark performance — Mini-SWE-Agent's minimal linear loop achieves results in the same range as OpenHands's event-sourced conversation engine.

### The Twin Absences
Two technologies central to the broader LLM literature are completely absent:

**Absence 1: Agentic frameworks**. Across ~4M lines, no production agent code path imports LangChain, LangGraph, AutoGen, CrewAI, or any of a dozen others. Gemini CLI uses neither of Google's own frameworks.

**Absence 2: RAG over code**. Zero systems use vector-embedding retrieval for code. The production answer is deterministic retrieval:

| System | Code-retrieval mechanism |
|--------|-------------------------|
| Claude Code | ripgrep keyword search + BashTool |
| Codex | Rust-native file search |
| Gemini CLI | Bundled ripgrep + glob tools |
| OpenHands | GrepTool/GlobTool + terminal |
| Aider | RepoMap (tree-sitter symbol extraction) |
| Hermes | ripgrep + SQLite FTS5 (BM25) |
| Pi | Auto-downloaded ripgrep + fd binaries |

### Extensibility Standards Resolved
- **Skills (9/11)** overtook **MCP (8/11)** in adoption
- SKILL.md format and `.agents/skills/` path became near-universal conventions
- ACP ships in six systems with a third role: harness hosting

### Longitudinal Evolution (April → July 2026)
- Convergence became imitation (Codex adopted Claude Code's hook vocabulary verbatim)
- Policy migrated from prompt prose to configuration
- Three of April's observations required substantive revision

### File Editing Strategies
Eight fundamentally different strategies identified, with a visible cross-harness genealogy (Hermes's matcher "inspired by OpenCode"; OpenCode's cascade crediting Cline and Gemini CLI).

### The Platform Turn Evidence
Four convergent signals documented:
1. Skills as declarative programs
2. Hooks and event buses as extension substrate (9/11 systems)
3. Disappearing boundary between tool and workflow
4. Harness as a service surface (OpenCode's embedded HTTP server, OpenHands's OpenAI-compatible gateway)

## Theoretical and Practical Implications

**The Harness–Framework Merger**: The paper documents both directions of convergence — harnesses became importable SDKs (Claude Agent SDK, openai-codex, OpenHands agent SDK) while framework vendors shipped harnesses (LangChain's Deep Agents, Pydantic AI Harness, Strands harness-sdk). The twin absence of frameworks gets its historical resolution: harnesses didn't adopt frameworks; they replaced them.

**Platform Economics**: Three developments signal platform dynamics:
- **Marketplaces**: Codex's plugin marketplace, skills registries in four systems
- **Switching costs**: Codex ships an importer for Claude Code's on-disk state
- **Enterprise governance**: MDM-managed preferences, host-wide system files, constraint engines

**Safety Architecture**: The paper reveals that OS-level sandboxing is a choice, not a consequence of scale. Hermes (~642K lines) ships zero OS-level isolation primitives, while Codex vendors Bubblewrap into its tree. The meta-harness re-pays the entire sandbox bill one layer up.

**The Scaffold–Capability Frontier**: A guiding intuition (not proven) that scaffold complexity and task success trace a roughly concave curve: a floor below which agents can't operate, a steep early-gain region, and a diminishing-returns plateau where further work addresses operational concerns rather than completion rates.

**18 Design Recommendations**: The analysis condenses into a practitioner's guide, including:
1. Start with a linear while loop
2. Couple tightly to home provider with generic fallback
3. Begin with just a bash tool
4. Adopt deferred tool loading beyond ~15 tools
5. Match edit-tool contract to model tier
6. Auto-discover hierarchical Markdown context files
7. Implement threshold compaction with incremental merging
8. **Do not** build RAG over code
9-11. Tiered safety architecture by deployment context
12. Stay single-agent until breadth-first exploration demands parallelism
13. Ship an ACP server
14. Use Skills for capability templates, MCP for external integrations
15-18. Explicit anti-patterns (no frameworks, no vector RAG, no 1-to-1 API wrapping, no over-engineered stuck detection)

**90-Line Minimum Viable Harness**: A concrete scaffold implementing 10 of 18 recommendations directly, with no framework dependencies, no RAG, no multi-agent orchestration, and no sandbox.

## Conclusion

The paper's key takeaways:

1. **Architecture matters** but not in the way current discourse implies — loop sophistication doesn't predict benchmark performance but does predict production readiness
2. **Model–agent relationship is co-evolutionary**, and coupling is about the update loop, not capability
3. **Safety is architecturally expensive** — and a choice, not a consequence of scale
4. **Multi-agent orchestration converged** on hierarchical patterns; ACP found a third role (harness hosting)
5. **Skills lead MCP** as the extensibility standard (9/11 vs. 8/11)
6. **The twin absences are real and historically explained** — harnesses replaced frameworks, not adopted them
7. **The Anthropic effective-agents series anticipates observed architectures** across independently developed systems
8. **The analysis is directly actionable** via 18 evidence-anchored recommendations
9. **The platform turn is no longer a hypothesis** — documented in named artifacts across the corpus

**Future work directions** include:
- Unified evaluation frameworks assessing safety, UX, cost efficiency alongside correctness
- Reference architecture specification formalizing identified patterns
- Formal verification of safety policies
- Empirical study of model–agent co-evolution
- Cross-system benchmark suite
- Inter-agent protocol adoption studies
- Longitudinal continuation (quarterly re-pinning)
- Meta-harness economics analysis
- Causation analysis of Anthropic–industry alignment

The paper closes with a notable methodological lesson: "in this field, inventory claims (tool counts, feature cells, version pins) decay in weeks, while structural claims (loop taxonomy, subsystem anatomy, the absences) have so far proven durable."

---

_Markdown view of https://picx.dev/p/S7cZF4, served by PicX — AI-generated visual whiteboard summaries of research papers._