# An Empirical Study of Harness Design for Coding Agents

> Harness components are conditional tools: context management matters most under tight budgets, planning scaffolds weak models but saves costs for strong ones, and action-space effectiveness depends on model bash proficiency.

- **Source:** [arXiv](https://arxiv.org/abs/2609.20804)
- **Published:** 2026-09-19
- **Permalink:** https://picx.dev/p/Axj3q7
- **Whiteboard:** https://picx.dev/p/Axj3q7/image

## Summary

# An Empirical Study of Harness Design for Coding Agents

## Summary (Overview)

- This paper presents a **controlled, component-level empirical study** of coding harness design, systematically varying three components—**planning, action space, and context management**—while holding the execution loop fixed.
- The study evaluates **176 experimental settings** across four models (Nemotron-3 30B/120B/550B and Mistral-Medium-3.5-128B) on two benchmarks (SWE-Bench Verified and Terminal-Bench 2.1), spanning five context-management strategies and four context-window budgets (32k–128k tokens).
- **Key finding 1:** Context management is most valuable under tight context-window budgets, primarily by preventing context-overflow failures; its benefit diminishes as the window expands.
- **Key finding 2:** The staged T4 strategy (elision before LLM summarization) achieves the best accuracy-cost trade-off, while the recall mechanism (making elision reversible) is rarely used and yields no accuracy gain.
- **Key finding 3:** Planning shifts from an **accuracy scaffold for weaker models** to a **cost saver for stronger models**; predefined tools help bash-weak models, while bash-only interfaces reduce cost for bash-capable models.

## Introduction and Theoretical Foundation

### Background and Motivation

Large language models (LLMs) are increasingly deployed to resolve real software-engineering tasks autonomously, including closing GitHub issues and completing end-to-end terminal tasks. This performance is achieved through a **coding harness**—a software layer whose components intervene on different aspects of agent behavior:

- **Planning scaffold**: maintains task structure
- **Action interface**: determines how model intentions become executable operations
- **Context-management policy**: decides what interaction history remains available under a finite window

### Key Research Gap

Existing studies typically evaluate harnesses as **monolithic systems**, conflating multiple mechanisms. For example, a cross-harness evaluation by Cao et al. (2026) reports that Claude-Opus-4.5 performs best with OpenHands while Claude-Sonnet-4.5 performs best with SWE-Agent, suggesting harness preferences vary across models—but without revealing *which* component drives the difference.

### Research Question

> Are harness components generally useful across settings, or does each component's effectiveness depend on model capability, task type, and resource budget?

### Theoretical Foundation

The harness follows a **ReAct loop** (Yao et al., 2022), where each turn comprises a reasoning step, an action, and an observation. Prior systems identify planning, action space, and context management as complementary requirements of long-horizon coding agents, with:
- Planning maintaining task progress (Bairi et al., 2024)
- The action space translating model intentions into executable workspace operations (Yang et al., 2024; Wang et al., 2024)
- Context management preserving useful information as trajectories grow (Packer et al., 2023; Wu et al., 2025)

## Methodology

### Harness Architecture

The harness uses a modular ReAct loop with three independently configurable components:

#### 1. Planning (§2.1)

When enabled, a system instruction defines the protocol with a first-turn reminder requesting an initial plan before action. The model maintains the plan through the `update_plan` tool. Subsequent turns append the plan to the model input **without storing it in conversation history**.

#### 2. Action Space (§2.2)

Two conditions are compared:

| Condition | Tools Available |
|-----------|----------------|
| **Predefined tools** | `read_file`, `write_file`, `edit_file`, `list_files`, `glob_files`, `grep_text`, `web_fetch`, `bash` |
| **Bash-only** | `bash` only (plus auxiliary tools required by planning/context management) |

The predefined file tools enforce read-before-write checks, update harness file state, and trigger automatic diagnostics after supported edits.

#### 3. Context Management (§2.3)

Three composable mechanisms are defined:

- **Elision (M1)**: replaces bulky stale tool observation bodies with short stubs
- **Recall (M2)**: stores elided observations in the file system, recoverable via `recall_event` tool
- **Summarization (M3)**: folds older messages into a running natural-language summary via a separate LLM call

**Algorithm 1 (Tier 4)**:

```
Require: history H; soft and hard thresholds B₁ < B₂
1: append the new think, action, and observation to H
2: if the model invoked RECALL_EVENT(id) this turn then
3:   read observation id from the external store back into H ▷ M2
4: end if
5: keep the preamble and a budget-sized recent window verbatim; let M be the middle region
6: if tokens(H) ≥ B₁ then
7:   for each bulky tool observation in M do
8:     store the original in the external store ▷ M2
9:     replace its body with a stub ▷ M1
10:  end for
11:  if tokens(H) ≥ B₂ then
12:    summarize the oldest events in M into a running summary ▷ M3
13:  end if
14: end if
15: return H
```

**Five context-management tiers**:

| Tier | M1 (Elision) | M2 (Recall) | M3 (Summarization) |
|------|:---:|:---:|:---:|
| T0 | ✗ | ✗ | ✗ |
| T1 | ✓ | ✗ | ✗ |
| T2 | ✓ | ✓ | ✗ |
| T3 | ✗ | ✗ | ✓ |
| T4 | ✓ | ✓ | ✓ |

### Experimental Setup

- **Models**: Nemotron-3 30B/120B/550B (within-family capability axis) + Mistral-Medium-3.5-128B (cross-family comparison)
- **Benchmarks**: SWE-Bench Verified (500 GitHub issues) and Terminal-Bench 2.1 (89 end-to-end terminal tasks)
- **Metrics**: Success rate (%) and mean cost per task ($)
- **Context budgets**: 32k, 64k, 96k, 128k tokens
- **Statistical tests**: Two-sided exact McNemar tests with Benjamini-Hochberg FDR control at 0.05
- **Implementation**: SGLang serving (BF16), temperature 0, top-p 0.95, max 300 steps/task

## Empirical Validation / Results

### 1. Context Management Value Grows as Budget Shrinks

The managed–T0 success-rate gap shrinks steadily as the context window expands:

| Window | SWE-Bench Gap (pp) | Terminal-Bench Gap (pp) |
|--------|:---:|:---:|
| 32k | 35.7 | 9.5 |
| 64k | 15.9 | 7.5 |
| 96k | 5.5 | 4.8 |
| 128k | 2.7 | 2.8 |

The T0 overflow rate falls from **78.7% to 8.7%** (SWE-Bench) and **61.0% to 12.1%** (Terminal-Bench) across budgets, while all managed tiers have **zero overflow failures**.

### 2. T4 Offers Best Accuracy-Cost Trade-off

T4 achieves success rates comparable to T1–T3 with the **lowest cost in 7 of 8 model-benchmark panels**. At 32k, T1/T2 trajectories still reach approximately the full window, while T3/T4 keep peak context substantially below it. T4 invokes M1 less often than T1/T2 at 32k/64k and invokes M3 less often than T3 at every budget.

### 3. Recall (M2) is Rarely Used

Across 32 model-benchmark-window comparisons, T2 vs T1 mean difference is **-0.36 percentage points**. Among 64 T2/T4 settings, **56.3% never call `recall_event`**; the median invocation rate is zero. Mean calls per task fall from 0.540 at 32k to 0.007 at 128k. Nemotron-3 550B and Mistral-Medium-3.5-128B rarely invoke it.

### 4. Planning: Accuracy Scaffold → Cost Saver

**Key results (T4/128k, full tool set)**:

| Model | SWE-Bench SR Δ | SWE-Bench Cost Δ | Terminal-Bench SR Δ | Terminal-Bench Cost Δ |
|-------|:---:|:---:|:---:|:---:|
| Nemotron-3 30B | **+11.6 pp** | ↑ | **+4.5 pp** | ↑ |
| Nemotron-3 120B | ~0 | ↑ | ~0 | ↓ |
| Nemotron-3 550B | -2.0 pp | **-30%** | small | ↓ |
| Mistral-3.5-128B | -0.4 pp | **-32%** | small | ↓ |

For Nemotron-3 30B, disabling planning reduces median SWE-Bench trajectory from 40 to 5 turns; **68.6% of runs terminate without an edit** (vs 27.8% with planning). For stronger models, planning primarily reduces **post-edit verification** turns.

### 5. Action Space: Model-Dependent Crossover

| Model | SWE-Bench: Tools−Bash SR Δ | Terminal-Bench: Tools−Bash SR Δ | Cost Effect |
|-------|:---:|:---:|:---:|
| Nemotron-3 30B | **+15.0 pp** | **+10.1 pp** | Bash cheaper |
| Nemotron-3 120B | +1.6 pp | +4.5 pp | Similar |
| Nemotron-3 550B | **-3.6 pp** | **-5.6 pp** | Bash **-53%** cost |
| Mistral-3.5-128B | **+23.2 pp** | **-6.7 pp** | Bash cheaper |

Bash-only reduces **re-patches** (edits to already-edited files) across all models and shifts file-writing toward coarser **create-or-replace actions**:

| Model | Re-patch (Tools) | Re-patch (Bash) | Create % (Tools) | Create % (Bash) |
|-------|:---:|:---:|:---:|:---:|
| Nemotron-3 30B | 3.3 | 0.4 | 28 | 64 |
| Nemotron-3 550B | 4.6 | 1.5 | 51 | 76 |
| Mistral-3.5-128B | 3.0 | 1.3 | 28 | 57 |

### 6. Trajectory-Level Mechanisms

- **Context management**: extends execution trajectories without substantially altering agent behavior (phase ordering preserved)
- **Planning**: sustains weak-model trajectories long enough to attempt an edit; truncates strong-model post-edit verification tails
- **Action space**: changes the granularity of code-writing actions—bash allows bundling multiple low-level operations into single composite commands

## Theoretical and Practical Implications

### For Harness Designers

1. **Context management should be budget-aware**: The value of context management is inversely proportional to context-window capacity. Under tight budgets, preventing overflow is the primary benefit; under generous budgets, simpler strategies suffice.

2. **Staged compaction (T4) is cost-effective**: Applying cheap rule-based elision *before* expensive LLM summarization reduces reliance on costly summarization calls while maintaining accuracy. The recall mechanism (M2) adds machinery that models rarely use—**lossless retrieval does not translate into completed tasks**.

3. **Planning is a conditional tool**: Its role shifts from keeping weak models alive to saving costs for strong models. Harness designers should consider model capability when deciding whether to include planning scaffolds.

4. **Action-space design should match model and task**: Predefined tools scaffold models with weak bash proficiency; bash-only interfaces enable capable models to compose denser operations. The crossover depends on how shell-centric the task type is (e.g., Terminal-Bench is more shell-centric than SWE-Bench).

### For Model Developers

The findings suggest that model training should consider interface alignment: models with stronger native tool-call vocabularies benefit more from structured interfaces, while models with strong bash proficiency can operate effectively with minimal scaffolding.

## Conclusion

This study demonstrates that **harness design is a conditional systems problem** in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default. The key takeaways:

1. **Context management** extends execution trajectories without substantially altering agent behavior, most valuable under tight context budgets
2. **Planning** sustains trajectories for models that abandon tasks too early and trims repeated verification in models that verify too long
3. **Structured tools** support models with limited shell proficiency, while bash-only enables capable models to combine multiple code modifications in a single call

### Limitations

- Planning and action space are ablated only under T4/128k (not full factorial)
- Each setting is run once per task; Terminal-Bench's 89 tasks limit statistical power
- SWE-Bench Verified is Python-only; results may not generalize to other languages
- Model size is an imperfect proxy for capability—training differences, tool-interface exposure, and native shell proficiency also contribute

### Future Directions

- Full factorial studies across all component combinations
- Validation of crossover points on other model families and task types
- Dynamic context-management policies that adapt thresholds based on task progress
- Investigation of how harness components interact with model training objectives

---

_Markdown view of https://picx.dev/p/Axj3q7, served by PicX — AI-generated visual whiteboard summaries of research papers._