Same Model, Different Harness: Different Coding-Agent Results

Summary (Overview)

• Core finding: Changing the harness configuration while keeping the model weights, tasks, and context capacity fixed significantly changes coding-agent performance, demonstrating that the model and harness together constitute the tested solver.

• Key results: Under context pressure, the treatment (closed-loop harness) raised mean per-task fail-to-pass fraction (F2PF) on all three benchmarks (SWE-bench Verified: 28%→49%, Pro: 15%→33%, FeatureBench: 11%→20%) and increased complete solutions on Verified (43→72) and Pro (31→72).

• Cross-model transfer: The frozen treatment package improved outcomes for all four tested models (Qwen3.6, Devstral, Nemotron, Qwen3.8) on the same 169-task Verified cohort at a 20,480-token window, with Devstral showing the largest proportional gains (2.1× F2PF, 2.4× solutions).

• Context regime dependence: At effectively unconstrained context (262,144 tokens), outcomes were close between arms on Verified and Pro, while FeatureBench retained a significant F2PF gain in the primary task-paired test.

• Mechanism: The treatment separates the complete in-memory conversation record from the model's working view, applying mechanical shortening of older tool results and responding to detected stall patterns.

Introduction and Theoretical Foundation

Background

Coding agents combine a model with a harness—the system that decides what the model sees, which tools it can use, and how work continues. On large codebases, accumulated history (files, search results, commands, test logs) competes for the same finite context window, making context pressure a significant challenge.

Motivation

The paper asks whether changing the harness changes results when the model and task stay fixed. The control configuration presents the conversation as a growing chronological transcript and stops when it fills the context window. The treatment (called Yuj) keeps a durable record but shows the model a smaller, changing view, and responds mechanically to detected stall patterns.

Theoretical Foundation

The authors argue that harness choices—view construction, tool behavior, run control—are part of the solver, not background plumbing. The key insight is that the model's working view need not grow identically to the permanent record of its work. The paper distinguishes between:

• The in-memory conversation: Complete record of everything that happened
• The model's working view: What the model actually sees at each step

The F2PF metric captures partial progress below the all-or-nothing resolution threshold:

$$\mathrm{F2PF} = \frac{\mathrm{F2P\ tests\ made\ to\ pass}}{\mathrm{all\ F2P\ tests}}$$

Methodology

Experimental Design

The study uses a paired comparison design: each task runs once under control and once under treatment, with both arms sharing the same model weights, tasks, context capacity, tool interface, serving setup, evaluator, and harness code. Only one checked-in configuration package differs.

The Treatment Package

Three components constitute the treatment:

1. Working-view rule (half-life policy): Shortens older tool-result bodies in stages as the context fills. The newest four tool results remain full; older results keep their beginning and end around an omission marker. As age doubles across tiers, the character cap halves. Activation begins when the estimated full prompt reaches half the configured window.

2. Detector: A rule-based checker that identifies fixed patterns in the execution record (e.g., repeated failing commands, repeated reading without edits) without consuming model tokens.

3. Command safeguards: Fixed protections that normalize test commands, stop forbidden commands, and prevent oversized setup output.

Benchmarks and Models

• Primary model: Qwen3.6-35B-A3B (DeltaNet/attention MoE), served with llama-server at four-bit quantization
• Benchmarks: SWE-bench Verified (169 tasks), SWE-bench Pro (316 tasks), FeatureBench (183 tasks)
• Transfer models: Devstral (dense transformer), Nemotron (Mamba-2/attention MoE), Qwen3.8 (dense DeltaNet/attention)

Statistical Tests

• Resolution (binary): Exact two-sided McNemar test
• F2PF (continuous): Exact two-sided sign test on paired differences, with repository-level sensitivity analysis using bootstrap resampling

Empirical Validation / Results

Primary Pressure Comparisons

Table 2: Outcomes for the three primary Qwen3.6 pressure comparisons

All three F2PF increases were significant at $p < 0.0001$ (exact paired sign tests), as were the Verified and Pro resolution increases (McNemar tests).

Context Regime Analysis

Table 6: Completed-run paired differences on the same 169-task Verified cohort

The treatment advantage is largest under tight context and nearly disappears when context is effectively unconstrained on Verified and Pro. FeatureBench retained a significant gain (F2PF: 23.9%→30.7%, $p = 0.00022$).

Cross-Model Transfer

Table 3: Paired outcomes on the 169-task Verified cohort at 20,480 tokens

Reading Boundary Analysis

The treatment moved the fitted reading boundary (where half the runs stop before the first source change) to more than twice the control location on every benchmark:

Resource Use

Treatment consumed more model work in every comparison (e.g., Verified: 3,280 vs. 6,517 model turns; 37.8 vs. 80.7 million prompt tokens), often remaining active after control reached its context limit.

Theoretical and Practical Implications

The Tested Solver is the Model and Harness Together

The central implication: a model name alone does not identify the solver behind measured coding-agent performance. Since changing the harness changed what unchanged model weights accomplished, evaluations must report the complete system—model, harness, tool interface, context policy, and run controls.

Harness Design as a Design Lever

The results show harness design is a legitimate optimization axis:

• The treatment produced large gains when context bound, while observed performance remained close when context was ample
• This pattern supports enabling the treatment from the start, whether or not context later binds
• The mechanical approach (no model summaries, no retrieval) works without additional model calls

For Builders and Evaluators

• Builders should treat view construction, tool behavior, and run control as parts of solver design, versioning and testing these choices as carefully as the model setup
• Evaluations should report: model, harness, tool interface, model-visible context, context limit and policy, time limits, and the rule for choosing the final result

Scope Limitations

• The comparison estimates the treatment-package effect as run, including differences in realized model work after control reached its context limit
• One greedy trajectory per arm per task—does not measure run-to-run variation
• Evidence covers four locally served open-weight checkpoints at four-bit precision
• Human patch-quality judgment was not an endpoint

Conclusion

The same model produced different coding-agent results under different configurations of the same harness. Under context pressure, treatment repaired more required behavior on all three benchmarks and completed more tasks on Verified and Pro. When context was effectively unconstrained, outcomes were close on Verified and Pro, while FeatureBench retained a significant mean per-task F2PF gain. The treatment raised both endpoints for every tested model on the frozen 169-task Verified cohort, though gains differed by model.

Future Directions

1. Component study: Isolate what the detector response adds to the fixed package
2. Time variation: Hold all else fixed while varying available time to separate time-limited from context-limited tasks
3. Adaptive harness: Let the harness adjust to run state (e.g., context fill rate, whether editing has begun), including caching repeated read-only results
4. Model-design interaction: Directly study how model design interacts with context pressure across severe, moderate, and unconstrained windows, starting with Devstral given its >2× gains

The paper concludes that long repository tasks collect files, searches, commands, and test output, but a harness can preserve that record while managing the model's working view and responding to the run as it develops. The paired results support treating the model and harness as one solver.