Summary

• Novel Benchmark: Introduces Harness-Bench, a diagnostic benchmark with 106 sandboxed offline tasks across 8 workflow categories, designed to evaluate configuration-level harness effects in realistic agent workflows.
• Key Finding: Across 5,194 execution trajectories, substantial performance variation (up to 23.8 points in aggregate score) is observed across model–harness pairings, demonstrating that agent capability should be reported at the model–harness configuration level rather than attributed to the base model alone.
• Core Contribution: First benchmark to make the "harness" (the system layer managing context, tools, state, permissions, and recovery) a primary axis of evaluation under shared external task conditions.
• Identified Failure Patterns: Documents recurring "execution-alignment" failures where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts.
• Practical Resource: Provides a reproducible foundation with task manifests, fixtures, evaluators, and execution traces for diagnosing reliable, efficient, and auditable agent execution stacks.

Introduction and Theoretical Foundation

The paper addresses a critical gap in LLM agent evaluation. While LLMs are increasingly deployed as executable systems that use tools, modify workspaces, and produce artifacts, existing benchmarks fail to properly account for the execution layer:

1. Static benchmarks (MMLU, GSM8K, BIG-bench, HELM) measure text-based capabilities but abstract away execution entirely.
2. Agent benchmarks (SWE-bench, WebArena, OSWorld, Terminal-Bench) evaluate complete systems in executable environments but either conflate the harness with the full agent system or fix the harness when comparing models.
3. Workflow-oriented benchmarks (AgentBench, GAIA, Claw-Eval) compare model backends under shared execution setups but do not vary the harness as a variable of interest.

The paper formalizes the agent decomposition as:

$$\text{Agent} = \text{Model} + \text{Harness}$$

where the harness encompasses prompt templates, action formats, context construction, tool invocation, workspace access, permissions, budget control, tracing, and recovery mechanisms. The theoretical foundation rests on the observation that a harness implicitly defines the "operational representation of the task"—what counts as a pending obligation, observed evidence, recoverable tool failure, and completed work.

Methodology

Benchmark Design

Harness-Bench employs a setup–execution–judge pipeline:

1. Setup: Renders task specification, constructs runtime environment, initializes fresh sandbox
2. Execution: Agent attempts task under specified budget and workspace constraints; records model requests/responses, tool calls, workspace changes, and usage statistics
3. Judge: Evaluator inspects final workspace and execution evidence

The formal run is defined as:

$$R = \operatorname{Run}(M, H, E, T), \quad \text{TaskScore} = \operatorname{Eval}(R; J)$$

Task Suite

106 tasks across 8 categories, each manually reviewed for four criteria: Realism, Solvability, Oracle-checkability, and Integrity. Categories include:

Scoring

The task score multiplicatively combines security, completion, and process components:

$$\text{TaskScore}_i = \text{Security}_i \cdot \text{Completion}_i \cdot \text{Process}_i$$

where the process score averages robustness, tool use appropriateness, and consistency:

$$\text{Process}_i = \frac{\text{Robustness}_i + \text{ToolUse}_i + \text{Consistency}_i}{3}$$

Experimental Design

• 6 configurable harnesses (OpenClaw, NanoBot, Hermes, ZeroClaw, NullClaw, Moltis) × 8 API model backends × 106 tasks = 5,088 trajectories
• Plus: 106 trajectories for Codex (model-bound coding agent, reported separately)
• Controlled factors: task prompt/fixtures, initial sandbox state, budget, timeout, evaluator
• Varying factors: model backend, harness configuration
• Native factors (preserved per harness): prompting, tool interface, state policy, retry/recovery behavior

Empirical Validation / Results

Main Results by Harness

Key findings:

• 23.8-point gap between best (NanoBot, 76.2) and worst (OpenClaw, 52.4) configurable harnesses
• Higher-scoring harnesses show stronger process profiles (tool use, consistency, robustness)
• Token efficiency varies dramatically (68.7K–175.1K tokens); higher token use doesn't guarantee better performance
• Codex achieves the highest overall score (80.4) using GPT-5.4 but is reported separately as a model-bound agent

Harness Dependence

• Stronger model backends tend to achieve higher mean scores and lower cross-harness variance, suggesting greater tolerance for execution-layer differences
• Weaker backends show larger variance across harnesses, indicating higher sensitivity to the execution substrate
• Cross-harness variation is larger in categories requiring structured data analysis, tool sequencing, and workspace manipulation

Recurring Failure Symptoms

Theoretical and Practical Implications

Theoretical Contributions

1. Configuration-level capability attribution: The paper challenges the practice of attributing agent performance to the base model alone, providing evidence that the execution layer (harness) is a significant source of variation.

2. Execution alignment framework: Introduces the concept of "execution alignment"—the degree to which a harness preserves correspondence among agent reasoning, observed workspace state, tool actions, and evaluator conditions. This provides a theoretical lens for understanding why harnesses matter.

3. Diagnostic methodology: Defines a protocol for measuring harness effects while controlling for external task conditions, enabling reproducible comparison across configurations.

Practical Implications

1. For practitioners: Results suggest careful harness selection can be as important as model selection. The 23.8-point gap between harnesses using the same models indicates significant untapped performance potential in execution-layer optimization.

2. For benchmark design: The study demonstrates that agent benchmarks should report both model and harness conditions, similar to how systems research reports hardware/software stack configurations.

3. For agent development: The identified failure modes (especially contract/format at 36.4%) highlight specific areas for improvement in harness design, including better schema validation, recovery mechanisms, and artifact commitment tracking.

Conclusion

Harness-Bench demonstrates that agent capability is a joint property of the model and its execution harness. The 5,194-trajectory study reveals substantial configuration-level variation (up to 23.8 points in aggregate score) across model–harness pairings under shared external conditions. The benchmark's sandboxed, offline design ensures reproducibility while covering diverse workflow categories from software engineering to DevOps.

Key limitations acknowledged by the authors include: (1) focus on controlled offline workflows excludes live services and user feedback; (2) evaluation of complete harness configurations cannot isolate individual mechanisms; (3) some process scores rely on LLM-assisted rubric assessment.

Future directions implied by this work include: developing harness-adaptive selection strategies (matching harnesses to tasks and models), automated harness optimization, deeper analysis of execution-alignment failures, and extension to multi-modal and long-horizon autonomous settings. The benchmark's public release provides a foundation for the community to diagnose and improve reliable, efficient, permission-aware, and auditable agent execution stacks.