HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Summary (Overview)

• New evaluation paradigm: HarnessDev shifts the unit of evaluation from task outputs to runnable infrastructure, measuring whether LLMs can build (Creation) and iteratively improve (Evolution) persistent agent harnesses—the execution systems that manage tool use, context, and verification.

• Two-stage benchmark: Creation starts from a deliberately weak but runnable seed harness and evaluates whether a creator LLM can build a complete execution system; Evolution starts from the creator's own harness and measures improvement via downstream execution feedback.

• Key finding on Creation: Model-built harnesses match or exceed human-engineered references on writing (EQ-Bench3) and machine-learning experimentation (MLE-bench), but remain substantially behind on code (SWE-Pro, Terminal-Bench) and search/research (BrowseComp), with large variation in execution cost.

• Key finding on Evolution: Models can produce local improvements from feedback (e.g., Opus 4.8 gains +4.44 points on held-out tasks), but gains are unstable, shrink on held-out tasks, and transfer only partially across runtime models—visible feedback and held-out scores move in the same direction only 53.1% of the time.

• Executor dependence: Switching the runtime model (Self-Eval vs. Unified-Eval with fixed Gemini) substantially changes both starting performance and whether evolutionary revisions help, indicating limited transfer of harness improvements across models.

Introduction and Theoretical Foundation

Motivation

The paper addresses a critical gap in agent evaluation: most benchmarks report downstream task performance under a chosen harness, treating the harness as part of the experimental configuration rather than as an artifact to be developed. Yet the harness's impact is substantial—with identical weights, GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 but 49.6% inside Codex CLI.

The authors draw an analogy to the forward-deployed engineer (FDE) role popularized by Palantir: an engineer who turns a general-purpose model into something that runs against a customer's specific data formats, workflows, and compliance constraints. FDE work supplies three structural pieces that benchmark designers normally presuppose:

1. Vague targets: informal business intent must be translated into concrete objectives and success criteria
2. Absent feedback signals: tests, judges, and traces must be constructed before improvement can be measured
3. Non-existent execution systems: tools, context management, state, and verification interfaces must be built and maintained

Theoretical Foundation

Harness engineering differs fundamentally from ordinary code editing. When a model modifies its own harness, it edits the execution substrate through which it acts—the change alters how the model observes, plans, and recovers in all future tasks. Effective harness improvement requires the model to:

• Recognize its own behavioral limitations from execution traces
• Diagnose structural bottlenecks in the system it runs inside
• Commit targeted changes that accumulate into lasting capability gains rather than one-off fixes

The formal pipeline is:

$$(L_C, D) \to H, \quad (H, L_E, x) \to y \xrightarrow{J} \mathrm{score}$$

where $L_C$ is the creator LLM working in development environment $D$ to produce harness $H$; executor LLM $L_E$ then runs inside $H$ on downstream task $x$; and evaluator $J$ scores output $y$.

Methodology

Benchmark Design

HarnessDev evaluates two stages of harness development:

The Weak Seed Harness

Every creator receives the same $H_{\text{seed}}$: a runnable compatibility layer, not a task-solving agent. It parses task/model configuration, exposes passive low-level tools, and writes required results/trajectories/logs. Crucially, it has:

• No agent loop, task decomposition, tool policy, context management, or persistent state
• No verifier, retry/recovery logic, or stopping rule
• Unmodified, it scores zero on every downstream benchmark

This design avoids two extremes: an empty repository (which would mix harness design with setup) and a mature agent (which would give away the planning structure being tested).

Domains and Benchmarks

Evolution uses a 100-task SWE-Pro feedback set, all 89 Terminal-Bench tasks as feedback, and a disjoint 630-instance SWE-Pro held-out split for generalization testing.

Evaluation Protocol

• Self-Eval: $L_E = L_C$ (creator also executes), measuring the complete creator–harness system
• Unified-Eval: every harness runs with the same fixed $L_E$ (Gemini 3.1 Pro), making harnesses directly comparable
• Constraint compliance: harnesses cannot hardcode solutions, consult hidden tests/answers, or replace the runtime interface; every run retains auditable trajectory artifacts

Creators and Setup

Six creator LLMs were evaluated: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro. Claude Code 2.1.177 served as the development environment (Codex 0.144.3 for GPT-5.5). Three independent harnesses per creator–benchmark pair were created (avg@3).

Empirical Validation / Results

Harness Creation (RQ1)

Overall performance under Self-Eval (avg@3, native metrics):

Token costs in millions; MLE-bench covers 33 physical cells and 2,475 results.

Key findings:

• Opus 4.8 achieves the highest overall score (67.8) but remains below the human reference (86.2)
• Writing harnesses approach the reference (84.6 vs. 83.7 for Opus); Search shows the largest gap (52.4 vs. 92.2)
• 77.8% of failed Data tasks are attributed to harness defects, not executor capability
• MLE-bench token use varies ~19-fold across creators, yet higher cost does not reliably produce higher scores

Executor dependence: Under fixed Gemini executor, rankings change substantially—Qwen gains 17.6 points on BrowseComp and 12.9 on MLE-bench, while Opus's SWE-Pro score collapses from 69.3 to 33.0, and its duplicate-query rate in Search rises from 10.1% to 88.2%.

Implementation behavior: 18 Code artifacts add 17,111 net lines total, but edit size does not predict performance. Gemini adds the fewest lines (1,006) yet achieves the best Terminal-Bench score (68.8). State/memory is the clearest gap: only 1/18 artifacts exposes a state-saving interface; no checkpoint event appears in 26,679 recorded trajectories.

Harness Evolution (RQ2)

Feedback vs. held-out generalization (pair score $\bar{P}_t = \frac{1}{2}(P_t^{\mathrm{SWE100}} + P_t^{\mathrm{Term89}})$):

Key findings:

• All self-runtime creators improve on visible feedback, but gains shrink on held-out tasks (Opus largest at +4.44)
• Under fixed Gemini, only Opus improves on held-out; other three lineages regress
• Evolution is not monotonic: of 64 official switches, 8 regress on both benchmarks, 16 show single-benchmark regression
• Visible feedback and held-out scores move in the same direction only 34/64 times (53.1%)
• Only 2/9 declared versions are held-out optimal
• 113 of 169 new functions are reachable; 31 reachable only through dead code; 25 have no caller

Positive example: Opus finds that 99/100 runs report success while only 48 pass, traces the gap to premature completion, and adds a completion check—a targeted, verified fix.

Theoretical and Practical Implications

Theoretical Contributions

1. Separation of harness quality from model capability: HarnessDev demonstrates that the persistent execution system and the model running inside it are separable axes of evaluation. Changing only the runtime binding can substantially move $H_0$ and alter which harness changes are useful.

2. Creator–executor co-adaptation: Harnesses can become specialized to their creator model (e.g., Opus's 120-step limit hard-coded around the original executor), such that capability transfers only when prompts, tool protocols, budgets, and stopping rules remain compatible.

3. Feedback overfitting: Repeatedly optimizing a noisy feedback score can favor lucky runs and amplify overfitting—visible feedback is useful for local search but unreliable for final selection.

Practical Implications

• For benchmark design: HarnessDev's protocol—freezing runnable artifacts, recording development trajectories, measuring execution cost, and evaluating transfer across runtime models—provides a template for evaluating infrastructure development rather than task outputs.

• For agent deployment: The finding that 77.8% of Data failures stem from harness defects (not executor capability) suggests that improving harness quality may be more cost-effective than upgrading models.

• For automated development: Models can make useful local improvements (Opus's completion gate), but robust evolution across unseen tasks and runtime models remains an open challenge—human oversight of the development loop is still needed.

Conclusion

HarnessDev moves agent evaluation from whether a model can solve tasks inside a fixed system to whether it can create and maintain the systems that solve future tasks. The benchmark makes agent-built execution harnesses a measurable object through:

• A four-domain human baseline corpus
• From-scratch creation tasks from a weak seed
• Feedback-driven evolution with separated visible and held-out evaluation
• Self- and Unified-Eval views that separate harness quality from executor capability
• Execution-token cost measurement

Main takeaways: Current models can construct runnable harnesses and make useful local improvements, but reliable evolution remains difficult. Gains are unstable, transfer only partially across models, and state/memory management is a consistent weakness. The authors conclude: "If model weights are one place intelligence accumulates, the harness is another: explicit, inspectable, testable, reusable, and continually improvable through failure, feedback, and real engineering pressure."

Future directions: The authors note that whether an evolved harness can itself serve as the development environment for further evolution is left to future work, as is matched-search evaluation and population-level comparisons with multiple trajectories per cell.