Summary (Overview)

• Claw-SWE-Bench introduces a multilingual SWE-bench-style benchmark (350 GitHub issue-resolution instances across 8 languages and 43 repositories) that treats the agent harness ("claw") as a controlled experimental variable, separating LLM effects from harness design effects.
• A shared adapter protocol standardizes the interface between heterogeneous agent harnesses and the SWE-bench evaluation lifecycle, enabling general-purpose agents like OpenClaw to be fairly evaluated on coding tasks.
• Claw-SWE-Bench Lite is an 80-instance cost-aware subset that reproduces full-benchmark Pass@1 within ~0.4 pp while reducing evaluation cost to ~22.9% of the full run.
• Key finding: adapter design is essential — a bare adapter scores only 19.1% Pass@1, while the full adapter reaches 73.4% with the same GLM 5.1 backbone.
• Model choice changes Pass@1 by 29.4 pp and harness choice by 27.4 pp under fixed models, demonstrating that harness selection is a first-order factor in coding-agent evaluation.

---

Introduction and Theoretical Foundation

Background and Motivation

General-purpose agents like OpenClaw have expanded into productivity tools, browser automation, and scientific assistance, but their repository-level coding ability remains difficult to measure. The natural evaluation standard is SWE-bench, which has become the de facto benchmark for repository-level coding agents.

The Core Problem: Confounded Evaluation

Leading SWE-bench-style reports package prompt templates, agent loops, tool interfaces, timeouts, patch extraction strategies, and stopping logic into a single released system. The resulting resolved rate conflates three causally distinct factors:

1. The evaluated LLM — the underlying model's coding capability
2. The harness — the agent loop that turns the LLM into an autonomous coding system
3. The task instances — the specific GitHub issues being solved

Prior work (HAL, SWE-Bench Pro, SWE-Effi) partially identifies this issue but does not treat the harness as a controlled variable. No prior SWE-bench-style benchmark has made the agent harness an experimental variable.

The SWE-bench Scoring Contract

The official SWE-bench evaluation harness reads a prediction file containing instance_id, model_name_or_path, and a string-valued model_patch. The evaluator applies the patch to a Docker checkout at /testbed and runs repository-level tests. The core interface is an evaluator-facing patch prediction, not a generic agent session.

Key Insight: Contract Mismatch

General-purpose agents like OpenClaw face three barriers to SWE-bench participation:

1. Runtime mismatch: SWE-bench Docker images don't provide agent lifecycle, tool configuration, or workspace management
2. Output mismatch: General agents signal completion through text/messages, while SWE-bench reads only model_patch
3. Artifact contamination: Session files and caches can contaminate the git diff

---

Methodology

Benchmark Architecture

Claw-SWE-Bench operates in two layers:

1. Adapter layer: Connects harness-native execution to SWE-bench's patch-prediction contract
2. Shared orchestrator: Fixes task set, repository state, prompt, Docker runtime, outer budget, patch extraction, prediction format, and evaluation

Adapter Protocol

Each harness implements five abstract methods:

• create_agent — instantiate the agent
• send_task — dispatch the instantiated task
• backup_session — save run artifacts
• delete_agent — clean harness state
• get_docker_args — provide container configuration

Workload Composition

Standardized Execution Pipeline

• Runtime: 3600-second wall-clock timeout, one run per instance, fixed worker concurrency (3)
• Workspace: Repository reset to base_commit, mounted at /testbed
• Future-commit cleanup: Removes reachable Git commits after base_commit to prevent information leakage
• Prompt instantiation: Identical task-prompt template across all harnesses
• Patch extraction: Collected from repository state (git diff), not parsed from agent responses

Bare vs. Full Adapter Diagnostic

• Bare adapter: Minimal integration — sends issue to OpenClaw, asks model to output unified diff directly
• Full adapter: Requires file edits in /testbed, runner exports patch from Git state

Lite Subset Selection

Lite-80 selects 10 instances per language with a fixed within-language difficulty-quartile quota of 2/3/3/2 over Q₁/Q₂/Q₃/Q₄. Selection optimizes three objectives over 17 calibration columns:

1. Resolve-rate parity: L1 difference between Lite-estimated and full-350 rates
2. Pairwise ranking hinge: Penalizes ranking reversals when columns differ by >0.03 on full-350
3. Cost parity: Minimizes log-cost discrepancy between Lite and full-350

$$\text{Objective} = \text{ResolveParity} + \lambda \cdot \text{RankHinge} + \text{CostParity}$$

Selection uses per-language 200-restart within-quartile 1-swap local search.

---

Empirical Validation / Results

Adapter Diagnostic (Table 1)

Key finding: Minimal access is insufficient — direct unified-diff generation fails due to line numbers, context, hunk headers, and trailing newlines causing patch application failures.

LLM-Axis Variation (Table 2): OpenClaw × 9 Models

Key findings:

• Model spread: 29.4 pp (48.6% to 78.0%)
• Cost ranking ≠ accuracy ranking: DeepSeek-V4 Flash achieves 70.3% at $8.2 vs. GPT 5.5 at 78.0$1399.1
• Cache hit rate explains some cost differences but not all

Claw-Axis Variation (Table 3): 5 Claws × 2 Models

Key findings:

• Claw spread: 12.5 pp on GLM 5.1, 27.4 pp on Qwen 3.6-flash
• Harness choice can reorder leaderboard conclusions

Future-Commit Cleanup Effect

Pass@1 after cleanup is never higher than before cleanup. Drops range from 0.6 to 8.0 pp:

• Claude Opus 4.7: 84.7% → 76.7% (−8.0 pp)
• Kimi 2.6: −5.0 pp
• Qwen 3.6-flash: −2.0 pp

Lite-80 Validation

• Mean Pass@1: 0.639 (full-350) vs. 0.643 (Lite-80), difference ≈ +0.4 pp
• Cross-claw mean absolute difference: 1.88 pp; max 3.68 pp
• Cost reduction: ~22.9% of full run (input tokens 22.2%, output tokens 23.6%, cache-read 22.6%, wall-clock 23.0%)
• K-sweep: minimum acceptable per-language size $K^* \in [8, 10]$; released at $K = 10$ (80 instances)

Pareto Frontier Analysis

The cost-accuracy Pareto frontier (Figure 1) shows:

• Lowest cost: generic × Qwen 3.6-flash ($14.50, 38.6%)
• OpenClaw × GLM 5.1 is the high-accuracy endpoint (73.4%, $277.00)
• hermes-agent and zeroclaw are dominated by OpenClaw × GLM 5.1 (more expensive and less accurate)

---

Theoretical and Practical Implications

Theoretical Contributions

1. Harness as controlled variable: Claw-SWE-Bench is the first SWE-bench-style benchmark to treat the agent harness as an experimental variable, decomposing the evaluation stack into fixed base components and a replaceable harness slot.

2. Model-harness non-separability: Results demonstrate that model and harness effects are not independent — harness brittleness matters more with smaller models (27.4 pp spread on Qwen vs. 12.5 pp on GLM 5.1).

3. Cost-aware evaluation framework: The benchmark establishes that accuracy and cost must be reported jointly, with cache hit rate as a diagnostic for cost accounting.

Practical Implications

1. Adapter necessity: General-purpose agents require substantial adaptation to satisfy SWE-bench scoring contracts — direct diff generation leads to 69.1% apply failures.

2. Cost-efficient evaluation: Lite-80 enables iterative development at ~23% of full-benchmark cost while preserving ranking stability and aggregate accuracy.

3. Reproducibility: The released protocol (identical prompt, budget, and orchestrator) allows fair cross-system comparison without conflating implementation details.

4. Tool design guidance: The Pareto analysis reveals that systems with similar accuracy can differ by orders of magnitude in cost, guiding deployment decisions.

---

Conclusion

Main Takeaways

• Claw-SWE-Bench provides a 350-instance multilingual benchmark with a shared adapter protocol, making heterogeneous agent harnesses comparable under fixed prompt, budget, and evaluation settings.
• Claw-SWE-Bench Lite offers an 80-instance cost-aware subset preserving full-set evaluation scale at ~23% of the cost.
• Adapter design is critical: The full adapter (73.4% Pass@1) dramatically outperforms the bare adapter (19.1%) with the same model.
• Harness choice is a first-order factor: Changing only the harness produces 12.5–27.4 pp differences in Pass@1.
• Cost-aware reporting is essential: Similar resolved rates can correspond to evaluation costs differing by orders of magnitude.

Limitations and Future Directions

1. Single-run aggregates: Differences of a few pp should not be overinterpreted; multi-seed replication is needed
2. Limited claw × model grid: Five claws × two models is insufficient to fully decompose harness × model interactions
3. Provider-dependent cost: Cost analysis depends on provider pricing and cache accounting; raw token traces should be retained for auditing
4. Generalization questions: Whether model-harness non-separability extends to web agents or computer-use agents remains open

Key Formulas

Pass@1:

$$\text{Pass@1} = \frac{\#\text{RESOLVED}}{\#\text{INSTANCES}}$$

Cache Hit Rate:

$$\text{CacheHit} = \frac{\#\text{CacheReadTokens}}{\#\text{InputTokens} + \#\text{CacheReadTokens}}$$