# Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

> Claw-SWE-Bench isolates harness design as a controlled variable, showing adapter design shifts Pass@1 from 19.1% to 73.4%, rivaling model choice in coding-agent evaluation.

- **Source:** [arXiv](https://arxiv.org/abs/2606.12344)
- **Published:** 2026-06-12
- **Permalink:** https://picx.dev/p/xr83Ku

## Summary

## Summary (Overview)

- **Claw-SWE-Bench** introduces a multilingual SWE-bench-style benchmark (350 GitHub issue-resolution instances across 8 languages and 43 repositories) that treats the agent harness ("claw") as a controlled experimental variable, separating LLM effects from harness design effects.
- A shared **adapter protocol** standardizes the interface between heterogeneous agent harnesses and the SWE-bench evaluation lifecycle, enabling general-purpose agents like OpenClaw to be fairly evaluated on coding tasks.
- **Claw-SWE-Bench Lite** is an 80-instance cost-aware subset that reproduces full-benchmark Pass@1 within ~0.4 pp while reducing evaluation cost to ~22.9% of the full run.
- Key finding: **adapter design is essential** — a bare adapter scores only 19.1% Pass@1, while the full adapter reaches 73.4% with the same GLM 5.1 backbone.
- Model choice changes Pass@1 by **29.4 pp** and harness choice by **27.4 pp** under fixed models, demonstrating that harness selection is a first-order factor in coding-agent evaluation.

---

## Introduction and Theoretical Foundation

### Background and Motivation

General-purpose agents like OpenClaw have expanded into productivity tools, browser automation, and scientific assistance, but their repository-level coding ability remains difficult to measure. The natural evaluation standard is SWE-bench, which has become the de facto benchmark for repository-level coding agents.

### The Core Problem: Confounded Evaluation

Leading SWE-bench-style reports package prompt templates, agent loops, tool interfaces, timeouts, patch extraction strategies, and stopping logic into a single released system. The resulting resolved rate conflates three causally distinct factors:

1. **The evaluated LLM** — the underlying model's coding capability
2. **The harness** — the agent loop that turns the LLM into an autonomous coding system
3. **The task instances** — the specific GitHub issues being solved

Prior work (HAL, SWE-Bench Pro, SWE-Effi) partially identifies this issue but does not treat the harness as a controlled variable. **No prior SWE-bench-style benchmark has made the agent harness an experimental variable.**

### The SWE-bench Scoring Contract

The official SWE-bench evaluation harness reads a prediction file containing `instance_id`, `model_name_or_path`, and a string-valued `model_patch`. The evaluator applies the patch to a Docker checkout at `/testbed` and runs repository-level tests. The core interface is an **evaluator-facing patch prediction**, not a generic agent session.

### Key Insight: Contract Mismatch

General-purpose agents like OpenClaw face three barriers to SWE-bench participation:

1. **Runtime mismatch**: SWE-bench Docker images don't provide agent lifecycle, tool configuration, or workspace management
2. **Output mismatch**: General agents signal completion through text/messages, while SWE-bench reads only `model_patch`
3. **Artifact contamination**: Session files and caches can contaminate the git diff

---

## Methodology

### Benchmark Architecture

Claw-SWE-Bench operates in two layers:

1. **Adapter layer**: Connects harness-native execution to SWE-bench's patch-prediction contract
2. **Shared orchestrator**: Fixes task set, repository state, prompt, Docker runtime, outer budget, patch extraction, prediction format, and evaluation

### Adapter Protocol

Each harness implements five abstract methods:

- `create_agent` — instantiate the agent
- `send_task` — dispatch the instantiated task
- `backup_session` — save run artifacts
- `delete_agent` — clean harness state
- `get_docker_args` — provide container configuration

### Workload Composition

| Source | Instances | Languages |
|--------|-----------|-----------|
| SWE-bench-Multilingual | 300 | Java, Go, Rust, JS/TS, C/C++, Ruby, PHP |
| SWE-bench-Verified-Mini | 50 | Python |
| **Total** | **350** | **8 languages, 43 repositories** |

### Standardized Execution Pipeline

- **Runtime**: 3600-second wall-clock timeout, one run per instance, fixed worker concurrency (3)
- **Workspace**: Repository reset to `base_commit`, mounted at `/testbed`
- **Future-commit cleanup**: Removes reachable Git commits after `base_commit` to prevent information leakage
- **Prompt instantiation**: Identical task-prompt template across all harnesses
- **Patch extraction**: Collected from repository state (git diff), not parsed from agent responses

### Bare vs. Full Adapter Diagnostic

- **Bare adapter**: Minimal integration — sends issue to OpenClaw, asks model to output unified diff directly
- **Full adapter**: Requires file edits in `/testbed`, runner exports patch from Git state

### Lite Subset Selection

Lite-80 selects 10 instances per language with a fixed within-language difficulty-quartile quota of 2/3/3/2 over Q₁/Q₂/Q₃/Q₄. Selection optimizes three objectives over 17 calibration columns:

1. **Resolve-rate parity**: L1 difference between Lite-estimated and full-350 rates
2. **Pairwise ranking hinge**: Penalizes ranking reversals when columns differ by >0.03 on full-350
3. **Cost parity**: Minimizes log-cost discrepancy between Lite and full-350

$$\text{Objective} = \text{ResolveParity} + \lambda \cdot \text{RankHinge} + \text{CostParity}$$

Selection uses per-language 200-restart within-quartile 1-swap local search.

---

## Empirical Validation / Results

### Adapter Diagnostic (Table 1)

| Configuration | Resolved | Pass@1 | Apply Failed |
|--------------|----------|--------|--------------|
| Bare adapter | 67/350 | 19.1% | 69.1% |
| Full adapter | 257/350 | 73.4% | < 1.5% |

**Key finding**: Minimal access is insufficient — direct unified-diff generation fails due to line numbers, context, hunk headers, and trailing newlines causing patch application failures.

### LLM-Axis Variation (Table 2): OpenClaw × 9 Models

| Model | Type | Pass@1 | Cost (USD) | Duration | In(M) | Out(M) | Turns | Cache |
|-------|------|--------|------------|----------|-------|--------|-------|-------|
| GPT 5.5 | Flagship | 78.0% | 1399.1 | 603.7 | 40.3 | 15.7 | 67.0 | 97.3 |
| Claude Opus 4.7 | Flagship | 77.1% | 1082.0 | 424.6 | 35.6 | 6.2 | 61.6 | 97.0 |
| GLM 5.1 | Flagship | 73.4% | 277.0 | 586.8 | 27.6 | 9.3 | 80.6 | 96.5 |
| DeepSeek-V4 Pro | Flagship | 71.7% | 81.3 | 662.3 | 19.3 | 11.0 | 47.1 | 97.4 |
| DeepSeek-V4 Flash | Flash | 70.3% | 8.2 | 430.0 | 13.8 | 12.9 | 51.2 | 98.5 |
| Kimi K2.6 | Flagship | 66.9% | 633.7 | 1235.3 | 75.6 | 12.1 | 78.7 | 92.1 |
| Qwen 3.6-flash | Flash | 66.0% | 71.5 | 636.0 | 38.9 | 7.5 | 87.9 | 97.6 |
| MiniMax M2.7 | Flagship | 61.4% | 196.7 | 1165.6 | 25.0 | 9.0 | 94.8 | 96.2 |
| Seed 2.0-mini | Flash | 48.6% | 19.4 | 1153.0 | 89.8 | 21.6 | 44.4 | 79.4 |

**Key findings**:
- Model spread: **29.4 pp** (48.6% to 78.0%)
- Cost ranking ≠ accuracy ranking: DeepSeek-V4 Flash achieves 70.3% at $8.2 vs. GPT 5.5 at 78.0% costing $1399.1
- Cache hit rate explains some cost differences but not all

### Claw-Axis Variation (Table 3): 5 Claws × 2 Models

| Claw | Model | Pass@1 | Cost (USD) | Duration | In(M) | Out(M) | Cache |
|------|-------|--------|------------|----------|-------|--------|-------|
| openclaw | GLM 5.1 | 73.4% | 277.0 | 586.8 | 27.6 | 9.3 | 96.5 |
| hermes-agent | GLM 5.1 | 71.1% | 330.6 | 675.1 | 93.1 | 5.5 | 91.3 |
| zeroclaw | GLM 5.1 | 70.3% | 383.4 | 538.2 | 989.7 | 4.2 | 90.4 |
| genericagent | GLM 5.1 | 63.1% | 85.8 | 576.4 | 99.7 | 5.1 | 66.8 |
| nanobot | GLM 5.1 | 60.9% | 768.8 | 1166.3 | 333.9 | 8.5 | 77.2 |
| openclaw | Qwen 3.6-flash | 66.0% | 71.5 | 636.0 | 38.9 | 7.5 | 97.6 |
| hermes-agent | Qwen 3.6-flash | 62.6% | 103.3 | 638.6 | 44.3 | 7.2 | 97.4 |
| zeroclaw | Qwen 3.6-flash | 58.3% | 49.3 | 428.9 | 1057.9 | 6.1 | 96.9 |
| nanobot | Qwen 3.6-flash | 47.4% | 133.1 | 562.7 | 418.8 | 8.2 | 63.9 |
| genericagent | Qwen 3.6-flash | 38.6% | 14.5 | 321.4 | 103.1 | 2.8 | 74.7 |

**Key findings**:
- Claw spread: **12.5 pp** on GLM 5.1, **27.4 pp** on Qwen 3.6-flash
- Harness choice can reorder leaderboard conclusions

### Future-Commit Cleanup Effect

Pass@1 after cleanup is never higher than before cleanup. Drops range from 0.6 to 8.0 pp:
- Claude Opus 4.7: 84.7% → 76.7% (−8.0 pp)
- Kimi 2.6: −5.0 pp
- Qwen 3.6-flash: −2.0 pp

### Lite-80 Validation

- Mean Pass@1: 0.639 (full-350) vs. 0.643 (Lite-80), difference ≈ +0.4 pp
- Cross-claw mean absolute difference: 1.88 pp; max 3.68 pp
- Cost reduction: ~22.9% of full run (input tokens 22.2%, output tokens 23.6%, cache-read 22.6%, wall-clock 23.0%)
- K-sweep: minimum acceptable per-language size $K^* \in [8, 10]$; released at $K = 10$ (80 instances)

### Pareto Frontier Analysis

The cost-accuracy Pareto frontier (Figure 1) shows:
- Lowest cost: generic × Qwen 3.6-flash ($14.50, 38.6%)
- OpenClaw × GLM 5.1 is the high-accuracy endpoint (73.4%, $277.00)
- hermes-agent and zeroclaw are dominated by OpenClaw × GLM 5.1 (more expensive and less accurate)

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Harness as controlled variable**: Claw-SWE-Bench is the first SWE-bench-style benchmark to treat the agent harness as an experimental variable, decomposing the evaluation stack into fixed base components and a replaceable harness slot.

2. **Model-harness non-separability**: Results demonstrate that model and harness effects are not independent — harness brittleness matters more with smaller models (27.4 pp spread on Qwen vs. 12.5 pp on GLM 5.1).

3. **Cost-aware evaluation framework**: The benchmark establishes that accuracy and cost must be reported jointly, with cache hit rate as a diagnostic for cost accounting.

### Practical Implications

1. **Adapter necessity**: General-purpose agents require substantial adaptation to satisfy SWE-bench scoring contracts — direct diff generation leads to 69.1% apply failures.

2. **Cost-efficient evaluation**: Lite-80 enables iterative development at ~23% of full-benchmark cost while preserving ranking stability and aggregate accuracy.

3. **Reproducibility**: The released protocol (identical prompt, budget, and orchestrator) allows fair cross-system comparison without conflating implementation details.

4. **Tool design guidance**: The Pareto analysis reveals that systems with similar accuracy can differ by orders of magnitude in cost, guiding deployment decisions.

---

## Conclusion

### Main Takeaways

- **Claw-SWE-Bench** provides a 350-instance multilingual benchmark with a shared adapter protocol, making heterogeneous agent harnesses comparable under fixed prompt, budget, and evaluation settings.
- **Claw-SWE-Bench Lite** offers an 80-instance cost-aware subset preserving full-set evaluation scale at ~23% of the cost.
- **Adapter design is critical**: The full adapter (73.4% Pass@1) dramatically outperforms the bare adapter (19.1%) with the same model.
- **Harness choice is a first-order factor**: Changing only the harness produces 12.5–27.4 pp differences in Pass@1.
- **Cost-aware reporting is essential**: Similar resolved rates can correspond to evaluation costs differing by orders of magnitude.

### Limitations and Future Directions

1. **Single-run aggregates**: Differences of a few pp should not be overinterpreted; multi-seed replication is needed
2. **Limited claw × model grid**: Five claws × two models is insufficient to fully decompose harness × model interactions
3. **Provider-dependent cost**: Cost analysis depends on provider pricing and cache accounting; raw token traces should be retained for auditing
4. **Generalization questions**: Whether model-harness non-separability extends to web agents or computer-use agents remains open

### Key Formulas

**Pass@1**:
$$\text{Pass@1} = \frac{\#\text{RESOLVED}}{\#\text{INSTANCES}}$$

**Cache Hit Rate**:
$$\text{CacheHit} = \frac{\#\text{CacheReadTokens}}{\#\text{InputTokens} + \#\text{CacheReadTokens}}$$

---

_Markdown view of https://picx.dev/p/xr83Ku, served by PicX — AI-generated visual whiteboard summaries of research papers._