An Empirical Study of Harness Design for Coding Agents

Summary (Overview)

• This paper presents a controlled, component-level empirical study of coding harness design, systematically varying three components—planning, action space, and context management—while holding the execution loop fixed.
• The study evaluates 176 experimental settings across four models (Nemotron-3 30B/120B/550B and Mistral-Medium-3.5-128B) on two benchmarks (SWE-Bench Verified and Terminal-Bench 2.1), spanning five context-management strategies and four context-window budgets (32k–128k tokens).
• Key finding 1: Context management is most valuable under tight context-window budgets, primarily by preventing context-overflow failures; its benefit diminishes as the window expands.
• Key finding 2: The staged T4 strategy (elision before LLM summarization) achieves the best accuracy-cost trade-off, while the recall mechanism (making elision reversible) is rarely used and yields no accuracy gain.
• Key finding 3: Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models; predefined tools help bash-weak models, while bash-only interfaces reduce cost for bash-capable models.

Introduction and Theoretical Foundation

Background and Motivation

Large language models (LLMs) are increasingly deployed to resolve real software-engineering tasks autonomously, including closing GitHub issues and completing end-to-end terminal tasks. This performance is achieved through a coding harness—a software layer whose components intervene on different aspects of agent behavior:

• Planning scaffold: maintains task structure
• Action interface: determines how model intentions become executable operations
• Context-management policy: decides what interaction history remains available under a finite window

Key Research Gap

Existing studies typically evaluate harnesses as monolithic systems, conflating multiple mechanisms. For example, a cross-harness evaluation by Cao et al. (2026) reports that Claude-Opus-4.5 performs best with OpenHands while Claude-Sonnet-4.5 performs best with SWE-Agent, suggesting harness preferences vary across models—but without revealing which component drives the difference.

Research Question

Are harness components generally useful across settings, or does each component's effectiveness depend on model capability, task type, and resource budget?

Theoretical Foundation

The harness follows a ReAct loop (Yao et al., 2022), where each turn comprises a reasoning step, an action, and an observation. Prior systems identify planning, action space, and context management as complementary requirements of long-horizon coding agents, with:

• Planning maintaining task progress (Bairi et al., 2024)
• The action space translating model intentions into executable workspace operations (Yang et al., 2024; Wang et al., 2024)
• Context management preserving useful information as trajectories grow (Packer et al., 2023; Wu et al., 2025)

Methodology

Harness Architecture

The harness uses a modular ReAct loop with three independently configurable components:

1. Planning (§2.1)

When enabled, a system instruction defines the protocol with a first-turn reminder requesting an initial plan before action. The model maintains the plan through the update_plan tool. Subsequent turns append the plan to the model input without storing it in conversation history.

2. Action Space (§2.2)

Two conditions are compared:

The predefined file tools enforce read-before-write checks, update harness file state, and trigger automatic diagnostics after supported edits.

3. Context Management (§2.3)

Three composable mechanisms are defined:

• Elision (M1): replaces bulky stale tool observation bodies with short stubs
• Recall (M2): stores elided observations in the file system, recoverable via recall_event tool
• Summarization (M3): folds older messages into a running natural-language summary via a separate LLM call

Algorithm 1 (Tier 4):

Require: history H; soft and hard thresholds B₁ < B₂
1: append the new think, action, and observation to H
2: if the model invoked RECALL_EVENT(id) this turn then
3:   read observation id from the external store back into H ▷ M2
4: end if
5: keep the preamble and a budget-sized recent window verbatim; let M be the middle region
6: if tokens(H) ≥ B₁ then
7:   for each bulky tool observation in M do
8:     store the original in the external store ▷ M2
9:     replace its body with a stub ▷ M1
10:  end for
11:  if tokens(H) ≥ B₂ then
12:    summarize the oldest events in M into a running summary ▷ M3
13:  end if
14: end if
15: return H

Five context-management tiers:

Experimental Setup

• Models: Nemotron-3 30B/120B/550B (within-family capability axis) + Mistral-Medium-3.5-128B (cross-family comparison)
• Benchmarks: SWE-Bench Verified (500 GitHub issues) and Terminal-Bench 2.1 (89 end-to-end terminal tasks)
• Metrics: Success rate (%) and mean cost per task ($)
• Context budgets: 32k, 64k, 96k, 128k tokens
• Statistical tests: Two-sided exact McNemar tests with Benjamini-Hochberg FDR control at 0.05
• Implementation: SGLang serving (BF16), temperature 0, top-p 0.95, max 300 steps/task

Empirical Validation / Results

1. Context Management Value Grows as Budget Shrinks

The managed–T0 success-rate gap shrinks steadily as the context window expands:

The T0 overflow rate falls from 78.7% to 8.7% (SWE-Bench) and 61.0% to 12.1% (Terminal-Bench) across budgets, while all managed tiers have zero overflow failures.

2. T4 Offers Best Accuracy-Cost Trade-off

T4 achieves success rates comparable to T1–T3 with the lowest cost in 7 of 8 model-benchmark panels. At 32k, T1/T2 trajectories still reach approximately the full window, while T3/T4 keep peak context substantially below it. T4 invokes M1 less often than T1/T2 at 32k/64k and invokes M3 less often than T3 at every budget.

3. Recall (M2) is Rarely Used

Across 32 model-benchmark-window comparisons, T2 vs T1 mean difference is -0.36 percentage points. Among 64 T2/T4 settings, 56.3% never call recall_event; the median invocation rate is zero. Mean calls per task fall from 0.540 at 32k to 0.007 at 128k. Nemotron-3 550B and Mistral-Medium-3.5-128B rarely invoke it.

4. Planning: Accuracy Scaffold → Cost Saver

Key results (T4/128k, full tool set):

For Nemotron-3 30B, disabling planning reduces median SWE-Bench trajectory from 40 to 5 turns; 68.6% of runs terminate without an edit (vs 27.8% with planning). For stronger models, planning primarily reduces post-edit verification turns.

5. Action Space: Model-Dependent Crossover

Bash-only reduces re-patches (edits to already-edited files) across all models and shifts file-writing toward coarser create-or-replace actions:

6. Trajectory-Level Mechanisms

• Context management: extends execution trajectories without substantially altering agent behavior (phase ordering preserved)
• Planning: sustains weak-model trajectories long enough to attempt an edit; truncates strong-model post-edit verification tails
• Action space: changes the granularity of code-writing actions—bash allows bundling multiple low-level operations into single composite commands

Theoretical and Practical Implications

For Harness Designers

1. Context management should be budget-aware: The value of context management is inversely proportional to context-window capacity. Under tight budgets, preventing overflow is the primary benefit; under generous budgets, simpler strategies suffice.

2. Staged compaction (T4) is cost-effective: Applying cheap rule-based elision before expensive LLM summarization reduces reliance on costly summarization calls while maintaining accuracy. The recall mechanism (M2) adds machinery that models rarely use—lossless retrieval does not translate into completed tasks.

3. Planning is a conditional tool: Its role shifts from keeping weak models alive to saving costs for strong models. Harness designers should consider model capability when deciding whether to include planning scaffolds.

4. Action-space design should match model and task: Predefined tools scaffold models with weak bash proficiency; bash-only interfaces enable capable models to compose denser operations. The crossover depends on how shell-centric the task type is (e.g., Terminal-Bench is more shell-centric than SWE-Bench).

For Model Developers

The findings suggest that model training should consider interface alignment: models with stronger native tool-call vocabularies benefit more from structured interfaces, while models with strong bash proficiency can operate effectively with minimal scaffolding.

Conclusion

This study demonstrates that harness design is a conditional systems problem in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default. The key takeaways:

1. Context management extends execution trajectories without substantially altering agent behavior, most valuable under tight context budgets
2. Planning sustains trajectories for models that abandon tasks too early and trims repeated verification in models that verify too long
3. Structured tools support models with limited shell proficiency, while bash-only enables capable models to combine multiple code modifications in a single call

Limitations

• Planning and action space are ablated only under T4/128k (not full factorial)
• Each setting is run once per task; Terminal-Bench's 89 tasks limit statistical power
• SWE-Bench Verified is Python-only; results may not generalize to other languages
• Model size is an imperfect proxy for capability—training differences, tool-interface exposure, and native shell proficiency also contribute

Future Directions

• Full factorial studies across all component combinations
• Validation of crossover points on other model families and task types
• Dynamic context-management policies that adapt thresholds based on task progress
• Investigation of how harness components interact with model training objectives