# Beyond the Model: Demystifying Harness Effects in Software Engineering Agents

> Harness design is a first-class performance factor in software engineering agents, with gains depending jointly on model capability and task type.

- **Source:** [arXiv](https://arxiv.org/abs/2609.32459)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/FWHR5g
- **Whiteboard:** https://picx.dev/p/FWHR5g/image

## Summary

# Beyond the Model: Demystifying Harness Effects in Software Engineering Agents

## Summary (Overview)

- This paper presents a systematic empirical study of **agent harness effects** in LLM-based Software Engineering (SE) agents, treating harness design as a first-class factor alongside model choice.
- The authors evaluate **two representative harnesses** (mini-SWE-agent and OpenCode) with **10 models** from Qwen and DeepSeek families across **three benchmarks** (SWE-bench Pro, ProgramBench, GitTaskBench), yielding 60 experimental configurations.
- They construct **NanoHarness**, a lightweight modular harness, to analyze five harness components: tool registry, context compression, explicit planning, subagents, and lazy skills.
- Key findings show that harness effectiveness depends jointly on model capability and task type — complex harnesses provide diminishing returns on SWE-style issue repair as models improve, but benefit stronger models on open-ended repository-level tasks.
- NanoHarness improves over mini-SWE-agent by **7.37 and 6.21 percentage points** on Qwen3.7-Max and DeepSeek-V4-Pro respectively, recovering most gains of product-level harnesses with controlled prompt growth.

## Introduction and Theoretical Foundation

The paper addresses a critical gap in SE agent research: while much attention focuses on foundation model improvements (GPT, Claude, Gemini, DeepSeek, Qwen), the **model-external infrastructure** — the agent harness — substantially shapes performance. The harness includes context retrieval, tool use, execution environments, patch generation, and validation.

The motivation is grounded in leaderboard evidence from SWE-bench Verified showing that harness design can have an impact comparable to model choice:

- The gap between best- and worst-performing harnesses under a fixed model reaches **19.4 percentage points**
- The gap between Claude models under a fixed harness is **22.8 percentage points**
- Epoch AI estimates a well-designed harness yields roughly **20% performance improvement**

The paper identifies that existing research evaluates harnesses only as part of end-to-end agent systems, making it impossible to isolate which design choices drive gains. The authors decompose modern harnesses into 12 modules, focusing on five core components:

1. **Tools** — expose external actions (file inspection, editing, search, shell execution, testing)
2. **Context and Memory** — determine what task information remains available during the agent loop
3. **Planning** — helps decompose tasks and track intermediate goals
4. **Subagents** — delegate parts of tasks to specialized roles
5. **Skills** — encode reusable high-level workflows

## Methodology

### Study Design

The research is organized into two complementary parts:

**Study 1: Joint Effects of Models, Harnesses, and Tasks**
- 3 tasks × 2 harnesses × 10 models = 60 configurations
- Models: 5 from Qwen family, 5 from DeepSeek family (ranked by Coding Index)
- Harnesses: mini-SWE-agent (lightweight) vs. OpenCode (product-level)

**Study 2: Component-Level Harness Analysis**
- NanoHarness built on mini-SWE-agent with plug-and-play components
- Five components implemented: tool registry, context compression, explicit planning, subagents (general and task-specific), lazy skills

### Tasks and Datasets

| Dataset | Task | Language | Size |
|---------|------|----------|------|
| ProgramBench | Repository generation | Rust, Go, C/C++, Java, Haskell | 200 |
| SWE-bench Pro | Issue repair | Python, JS/TS, Go | 731 |
| GitTaskBench | Repository task solving | Python | 54 |

### Implementation Details

- Temperature set to 0.0 for all models
- SWE-bench Pro and GitTaskBench: 250 agent steps per task
- ProgramBench: 300 steps (reduced from original 1,000)
- 6-hour wall-clock limit, 3-minute timeout per command
- RQ1 uses random subsets (300 from SWE-bench Pro, 70 from ProgramBench)
- RQ2/RQ3 use full 200-instance ProgramBench set
- Context compression: 350K-token budget, 20 most recent messages preserved

## Empirical Validation / Results

### RQ1: Relationship Among Models, Harnesses, and Tasks

**RQ1.1 — Scaling with Model Capability:**

Matched-model results from SWE-bench Verified leaderboard show diminishing harness gains:

| Harness | Model (weak → strong) | mini-SWE | Gain |
|---------|----------------------|----------|------|
| SWE-agent | Claude 3.7 Sonnet | 52.80 | +9.60 |
| SWE-agent | Claude 4 Sonnet | 64.93 | +4.07 |
| OpenHands | Claude 3.7 Sonnet | 52.80 | +13.60 |
| OpenHands | Claude 4 Sonnet | 64.93 | +5.47 |
| Tools | Claude 3.7 Sonnet | 52.80 | +10.40 |
| Tools | Claude 4 Sonnet | 64.93 | +7.47 |
| Tools | Claude 4 Opus | 67.60 | +5.60 |

**Qwen family**: OpenCode advantage narrows from ~6 pp (Qwen3-Max) to 2.67 pp (Qwen3.7-Max) — smooth convergence.

**DeepSeek family**: Non-monotonic pattern — OpenCode helps intermediate models (V3, V3.1) but nearly converges with mini-SWE on DeepSeek-V4-Pro (34.3% vs. 33.7%), suggesting a threshold effect.

**RQ1.2 — Task Variation:**
- On ProgramBench and GitTaskBench, only the strongest models (Qwen3.7-Max, DeepSeek-V4-Pro) consistently leverage complex harnesses
- SWE-bench Pro benefits from relatively fixed workflows and possible task-specific model training
- Complex, open-ended repository tasks favor complex harnesses with stronger models

### RQ2: Component-Level Effect Analysis

**Individual Effects (ProgramBench, full 200 instances):**

| Component | Qwen3.7-Max | DeepSeek-V4-Pro |
|-----------|-------------|-----------------|
| Tool registry | +4.57 pp | +3.54 pp |
| Task-specific subagents | +5.91 pp | +4.46 pp |
| Planning | Small gains | Small gains |
| Lazy skills | Moderate | Larger |
| Context compression | −4.86 pp | −3.85 pp |
| General subagents | Negative | Negative |

**Combined Effect (NanoHarness):**

| Harness | Qwen3.7-Max | DeepSeek-V4-Pro |
|---------|-------------|-----------------|
| mini-SWE-agent | 42.88 | 45.14 |
| NANOHARNESS | 50.25 (+7.37%) | 51.35 (+6.21%) |
| OpenCode | 51.68 (+8.80%) | 52.48 (+7.34%) |
| ClaudeCode | 52.33 (+9.45%) | 52.76 (+7.62%) |

### RQ3: Behavior and Efficiency Effects

**Behavior patterns** (average sample-level statistics):

- Task-specific subagents increase tool calls by **115.76%** (Qwen3.7-Max) and **73.10%** (DeepSeek-V4-Pro)
- Context compression cuts prompt tokens by **54.29%** and **77.02%** but hurts performance
- Full NanoHarness: tool calls +131.18%/+83.58%, but prompt tokens only +16.37%/+7.26% — gains come from structured interaction, not more prompt budget

**Failure mode analysis** reveals two failure patterns on ProgramBench:
1. **Excessive probing**: agent performs many low-value behavioral probes, exhausts budget
2. **Insufficient probing**: agent implements after few probes (≤20), validates with only smoke tests

Case study (jqlang__jq): mini-SWE-agent issued 133 probes, passed 24.68% tests; NanoHarness reduced to 43 probes, improved to 68.17%. For insufficient probing (lfos__calcurse): mini-SWE made 3 probes (61.74% pass), NanoHarness expanded to 45 probes (75.18% pass).

## Theoretical and Practical Implications

The paper establishes that **harness design is a first-class factor** in SE-agent performance, with several important implications:

1. **Harness effectiveness is context-dependent**: The same harness can help or hinder depending on model capability and task type. This challenges one-size-fits-all harness designs.

2. **Structured tool use matters more than scaffolding**: Gains come from regulating how agents explore and act in repositories, not from adding more framework complexity.

3. **Task-specific delegation beats generic subagents**: General SE subagents introduce coordination overhead without aligned artifacts, while task-specific profiles (behavior analysis, differential testing, submission review) provide stable gains.

4. **Context compression has costs**: Aggressive compression removes requirements and debugging state needed for long-horizon repository generation tasks.

5. **Lightweight designs can compete with product-level harnesses**: A compact set of well-coordinated components recovers most of the performance of complex harnesses like OpenCode and Claude Code.

## Conclusion

The paper demonstrates that effective harnesses improve SE agents by **turning raw exploration into structured, task-aligned interaction** rather than by increasing model capability alone. Key takeaways:

- Harness gains depend jointly on model capability and task type
- Structured tools and task-specific subagents are the most valuable components
- Effective harnesses regularize exploration: reducing wasteful probing when agents are stuck, increasing behavioral coverage when agents are prematurely confident
- Future SE-agent design should jointly consider models, harnesses, tasks, and execution behavior

**Future directions** include extending evaluation to more harness components, broader model families, and additional benchmarks. The study's limitations include its focus on open-weight models (Qwen, DeepSeek) and primary experimentation on ProgramBench, though the total evaluation cost exceeded $7,000 USD with over 10B (Qwen) and 20B (DeepSeek) prompt tokens.

---

_Markdown view of https://picx.dev/p/FWHR5g, served by PicX — AI-generated visual whiteboard summaries of research papers._