Beyond the Model: Demystifying Harness Effects in Software Engineering Agents

Summary (Overview)

• This paper presents a systematic empirical study of agent harness effects in LLM-based Software Engineering (SE) agents, treating harness design as a first-class factor alongside model choice.
• The authors evaluate two representative harnesses (mini-SWE-agent and OpenCode) with 10 models from Qwen and DeepSeek families across three benchmarks (SWE-bench Pro, ProgramBench, GitTaskBench), yielding 60 experimental configurations.
• They construct NanoHarness, a lightweight modular harness, to analyze five harness components: tool registry, context compression, explicit planning, subagents, and lazy skills.
• Key findings show that harness effectiveness depends jointly on model capability and task type — complex harnesses provide diminishing returns on SWE-style issue repair as models improve, but benefit stronger models on open-ended repository-level tasks.
• NanoHarness improves over mini-SWE-agent by 7.37 and 6.21 percentage points on Qwen3.7-Max and DeepSeek-V4-Pro respectively, recovering most gains of product-level harnesses with controlled prompt growth.

Introduction and Theoretical Foundation

The paper addresses a critical gap in SE agent research: while much attention focuses on foundation model improvements (GPT, Claude, Gemini, DeepSeek, Qwen), the model-external infrastructure — the agent harness — substantially shapes performance. The harness includes context retrieval, tool use, execution environments, patch generation, and validation.

The motivation is grounded in leaderboard evidence from SWE-bench Verified showing that harness design can have an impact comparable to model choice:

• The gap between best- and worst-performing harnesses under a fixed model reaches 19.4 percentage points
• The gap between Claude models under a fixed harness is 22.8 percentage points
• Epoch AI estimates a well-designed harness yields roughly 20% performance improvement

The paper identifies that existing research evaluates harnesses only as part of end-to-end agent systems, making it impossible to isolate which design choices drive gains. The authors decompose modern harnesses into 12 modules, focusing on five core components:

1. Tools — expose external actions (file inspection, editing, search, shell execution, testing)
2. Context and Memory — determine what task information remains available during the agent loop
3. Planning — helps decompose tasks and track intermediate goals
4. Subagents — delegate parts of tasks to specialized roles
5. Skills — encode reusable high-level workflows

Methodology

Study Design

The research is organized into two complementary parts:

Study 1: Joint Effects of Models, Harnesses, and Tasks

• 3 tasks × 2 harnesses × 10 models = 60 configurations
• Models: 5 from Qwen family, 5 from DeepSeek family (ranked by Coding Index)
• Harnesses: mini-SWE-agent (lightweight) vs. OpenCode (product-level)

Study 2: Component-Level Harness Analysis

• NanoHarness built on mini-SWE-agent with plug-and-play components
• Five components implemented: tool registry, context compression, explicit planning, subagents (general and task-specific), lazy skills

Tasks and Datasets

Implementation Details

• Temperature set to 0.0 for all models
• SWE-bench Pro and GitTaskBench: 250 agent steps per task
• ProgramBench: 300 steps (reduced from original 1,000)
• 6-hour wall-clock limit, 3-minute timeout per command
• RQ1 uses random subsets (300 from SWE-bench Pro, 70 from ProgramBench)
• RQ2/RQ3 use full 200-instance ProgramBench set
• Context compression: 350K-token budget, 20 most recent messages preserved

Empirical Validation / Results

RQ1: Relationship Among Models, Harnesses, and Tasks

RQ1.1 — Scaling with Model Capability:

Matched-model results from SWE-bench Verified leaderboard show diminishing harness gains:

Qwen family: OpenCode advantage narrows from ~6 pp (Qwen3-Max) to 2.67 pp (Qwen3.7-Max) — smooth convergence.

DeepSeek family: Non-monotonic pattern — OpenCode helps intermediate models (V3, V3.1) but nearly converges with mini-SWE on DeepSeek-V4-Pro (34.3% vs. 33.7%), suggesting a threshold effect.

RQ1.2 — Task Variation:

• On ProgramBench and GitTaskBench, only the strongest models (Qwen3.7-Max, DeepSeek-V4-Pro) consistently leverage complex harnesses
• SWE-bench Pro benefits from relatively fixed workflows and possible task-specific model training
• Complex, open-ended repository tasks favor complex harnesses with stronger models

RQ2: Component-Level Effect Analysis

Individual Effects (ProgramBench, full 200 instances):

Combined Effect (NanoHarness):

RQ3: Behavior and Efficiency Effects

Behavior patterns (average sample-level statistics):

• Task-specific subagents increase tool calls by 115.76% (Qwen3.7-Max) and 73.10% (DeepSeek-V4-Pro)
• Context compression cuts prompt tokens by 54.29% and 77.02% but hurts performance
• Full NanoHarness: tool calls +131.18%/+83.58%, but prompt tokens only +16.37%/+7.26% — gains come from structured interaction, not more prompt budget

Failure mode analysis reveals two failure patterns on ProgramBench:

1. Excessive probing: agent performs many low-value behavioral probes, exhausts budget
2. Insufficient probing: agent implements after few probes (≤20), validates with only smoke tests

Case study (jqlang__jq): mini-SWE-agent issued 133 probes, passed 24.68% tests; NanoHarness reduced to 43 probes, improved to 68.17%. For insufficient probing (lfos__calcurse): mini-SWE made 3 probes (61.74% pass), NanoHarness expanded to 45 probes (75.18% pass).

Theoretical and Practical Implications

The paper establishes that harness design is a first-class factor in SE-agent performance, with several important implications:

1. Harness effectiveness is context-dependent: The same harness can help or hinder depending on model capability and task type. This challenges one-size-fits-all harness designs.

2. Structured tool use matters more than scaffolding: Gains come from regulating how agents explore and act in repositories, not from adding more framework complexity.

3. Task-specific delegation beats generic subagents: General SE subagents introduce coordination overhead without aligned artifacts, while task-specific profiles (behavior analysis, differential testing, submission review) provide stable gains.

4. Context compression has costs: Aggressive compression removes requirements and debugging state needed for long-horizon repository generation tasks.

5. Lightweight designs can compete with product-level harnesses: A compact set of well-coordinated components recovers most of the performance of complex harnesses like OpenCode and Claude Code.

Conclusion

The paper demonstrates that effective harnesses improve SE agents by turning raw exploration into structured, task-aligned interaction rather than by increasing model capability alone. Key takeaways:

• Harness gains depend jointly on model capability and task type
• Structured tools and task-specific subagents are the most valuable components
• Effective harnesses regularize exploration: reducing wasteful probing when agents are stuck, increasing behavioral coverage when agents are prematurely confident
• Future SE-agent design should jointly consider models, harnesses, tasks, and execution behavior

Future directions include extending evaluation to more harness components, broader model families, and additional benchmarks. The study's limitations include its focus on open-weight models (Qwen, DeepSeek) and primary experimentation on ProgramBench, though the total evaluation cost exceeded $7,000 USD with over 10B (Qwen) and 20B (DeepSeek) prompt tokens.