# Don’t Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality

> Rapid agent harness evolution (10–18 releases/week) yields no quality gains but doubles token consumption, making harness versions critical for reproducible coding agent evaluation.

- **Source:** [arXiv](https://arxiv.org/abs/2607.03691)
- **Published:** 2026-08-15
- **Permalink:** https://picx.dev/p/nyL7PI
- **Whiteboard:** https://picx.dev/p/nyL7PI/image

## Summary

, Summary: "Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality"

## Summary (Overview)

- This paper investigates how the evolution of **agent harnesses** (the middleware layer between developers and LLMs that orchestrates prompts, tools, and reasoning loops) affects coding agent quality over time, while holding the LLM constant.
- The authors characterize **"hyper-churn"** in five major open-source coding agent harnesses (Gemini CLI, Codex, OpenCode, OpenHands, Qwen Code), finding release velocities of **10–18 releases per week**—13–28× higher than traditional projects like VS Code and GitHub CLI.
- Through a controlled longitudinal study of **35 sequential Qwen Code CLI releases** evaluated on **50 stratified SWE-bench Verified tasks** with a fixed LLM (Qwen3-Next-80B-A3B-Instruct), the study finds **no statistically significant improvement in resolve rates** (mean ≈ 30.5%) despite substantial codebase growth.
- Later releases consume **nearly double the tokens** (up from ~391K to ~668K per task, +70%) and more tool calls without corresponding quality gains, driven by 8% larger system prompts and 18% more reasoning turns.
- The paper identifies **LLM Provider and Context Management layers as high-risk architectural components** for regressions, and calls for **"Non-functional Agentic Quality Assurance"** —automated regression testing of agent effectiveness and efficiency metrics.

---

## Introduction and Theoretical Foundation

### Background and Motivation

The paper addresses a critical gap in understanding AI-assisted software development. While large language models (LLMs) have received substantial research attention, the **agent harness**—the software layer that mediates between the LLM and execution environment—remains largely uninvestigated. The authors note that practitioners regularly report quality regressions after agent harness updates but consistently attribute these to the underlying model rather than the harness itself.

> "The agent harness surrounding the LLM is not stable but is instead a continuously evolving software system. This raises the question of whether this rapid evolution translates into actual quality improvements."

### The Agent Harness Layer

The agent harness is defined as the middleware that orchestrates:
- **System prompts** and instructions specifying the agent's goals and constraints
- **Tool definitions** and execution sandboxing
- **Context management** including history updating and prompt assembly
- **Iterative reasoning loops** (e.g., ReAct pattern)

### Research Questions

- **RQ0:** What are the scale and evolutionary characteristics of coding agent harness projects?
- **RQ1:** How does agent quality evolve across successive agent harness releases?
- **RQ2:** What release-level development patterns are associated with quality shifts?
- **RQ3:** Which architectural components of agent harnesses exhibit greater sensitivity to change?

---

## Methodology

### Study Design

The study employs a **two-phase design**:

**Phase 1 (RQ0): Landscape Analysis** — Mining GitHub repositories of five coding agent harnesses and two baseline projects (VS Code, GitHub CLI) across four dimensions: release velocity, development activity, maintenance burden, and issue tracking.

**Phase 2 (RQ1–RQ3): Controlled Longitudinal Evaluation** — Evaluating 35 sequential Qwen Code CLI releases (v0.0.10–v0.10.3) on 50 SWE-bench Verified tasks while holding the LLM constant.

### Experimental Setup

| Component | Specification |
|-----------|--------------|
| **Agent Harness** | Qwen Code CLI (35 releases) |
| **LLM** | Qwen3-Next-80B-A3B-Instruct (self-hosted via vLLM) |
| **Tasks** | 50 stratified SWE-bench Verified tasks |
| **Runs** | 2 per task (3,500 total executions) |
| **Timeout** | 600 seconds per task |
| **Metrics** | Resolve rate, token consumption, tool calls |

### Task Stratification

**Table 1: Stratified task sampling from SWE-bench Verified**

| Difficulty | Fix Time | SWE-bench % | Sampled |
|------------|----------|-------------|---------|
| Easy | < 15 min | 38.8% | 20 (40%) |
| Medium | 15 min–1 hr | 52.2% | 25 (50%) |
| Hard | 1–4 hrs | 8.4% | 5 (10%) |
| Very Hard | > 4 hrs | 0.6% | 0 (0%) |
| **Total** | | | **50** |

### Normalization Approach

To remove task-difficulty bias, token consumption and tool calls were normalized per task:

$$\frac{m_{t,v} - \bar{m}_t}{\bar{m}_t} \times 100$$

where $m_{t,v}$ is the metric value for task $t$ in version $v$, and $\bar{m}_t$ is that task's mean across all versions and runs.

### Statistical Methods

- **Wilcoxon signed-rank test** for run consistency
- **Spearman rank correlation** for monotonic trends
- **Mann-Whitney U test** with **Benjamini-Hochberg correction** for tier comparisons
- **Cliff's delta** for effect sizes
- **Partial Spearman correlations** controlling for codebase churn

---

## Empirical Validation / Results

### RQ0: Hyper-Churn in Agent Harness Development

**Finding 1: Agent harnesses release 13–28× more frequently than traditional open-source projects.**

**Table 3: Release velocity metrics**

| Harness | #Releases (lifetime) | Release frequency (per week) | Median Interval |
|---------|----------------------|------------------------------|-----------------|
| Gemini CLI | 330 | 10.3 | 0.20 days |
| Codex | 518 | 12.4 | 0.19 days |
| OpenCode | 716 | 18.0 | 0.12 days |
| OpenHands CLI | 19 | 1.5 | 5.06 days |
| Qwen Code | 288 | 10.0 | 0.59 days |
| **VS Code** | **200** | **0.8** | **6.99 days** |
| **GitHub CLI** | **191** | **0.6** | **11.31 days** |

**Finding 2:** Agent harnesses sustain **13–34 merged commits per day** with median PR review times under 4 hours (vs. 0.6–0.8 releases/week for baselines).

**Finding 3:** Issue backlogs grow rapidly—OpenCode accumulated 3,965 open issues with only a 54.0% close rate.

### RQ1: Quality Evolution Across Releases

**Finding 4: Quality metrics are consistent across independent runs** (87.7% binary agreement; Wilcoxon p-values > 0.43).

**Finding 5: No significant resolve rate trend across releases** (Spearman ρ = 0.208, p = 0.231), fluctuating around a mean of 30.5% with peaks at 39% in early versions never exceeded.

**Finding 6: Token consumption shows significant increasing trend** (ρ = 0.743, p < 0.0001), with later releases consuming up to 19.6% more tokens per task relative to baseline, while earlier versions operated 24.2% below baseline.

**Finding 7: Failed tasks consume 2.7× more tokens and 1.8× more tool calls** than successful ones (697.7K vs. 258.7K tokens; 12.95 vs. 7.2 tool calls).

**Finding 8: Token inflation driven by 8% larger system prompts and 18% more LLM turns**, with near-perfect correlation between tokens and turns (ρ = 0.941, p < 0.0001).

### RQ2: Release-Level Development Patterns

**Finding 9: Development intensity correlates with cost, not effectiveness.** Total commits and unique authors show large negative effect sizes for token consumption (d = −0.601 and d = −0.482 respectively).

**Finding 10: Feature and issue activity positively associate with resolve rate but at efficiency cost:**
- `bug_to_feature_ratio`: ρ = 0.546, p = 0.0006
- `feat_churn_ratio`: ρ = 0.438, p = 0.0075
- `feat_churn_ratio` also correlates with higher tokens (d = −0.573) and tool calls (d = −0.900)

**Finding 11: Refactoring shows no quality benefit** (ρ = −0.372, p = 0.025 for resolve rate).

**Finding 12–13 (Delta analysis):** Larger PR sizes correlate with immediate cost savings (ρ = −0.484 for tool calls), and code deletions associate with lower token consumption (ρ = −0.351), while fix-heavy transitions correlate with higher costs (d = −0.833).

### RQ3: Architectural Sensitivity

**Finding 14: LLM Provider and Extensibility changes associate with better token efficiency** (AE1: ρ = +0.490, partial ρ = +0.492).

**Finding 15: Context Management expansion associates with lower token efficiency** (AE4: ρ = −0.346, partial ρ = −0.392).

**Finding 17: Security fixes correlate with improved efficiency** (DE4: ρ = +0.346; DE6: ρ = +0.341).

**Finding 18: LLM Provider transitions carry high regression risk** (DE2: ρ = −0.336), with the v0.4.1→v0.5.0 transition causing resolve rate to drop from 39.4% to 32.5%.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Challenges the "more is better" assumption**: The study empirically demonstrates that continuous agent harness development does not guarantee quality improvement, extending Lehman's laws of software evolution to agentic systems.

2. **Introduces "hyper-churn" concept**: A new development regime characterized by extreme release velocity that challenges traditional software engineering quality assurance models.

3. **Emergent quality dimensions**: Agentic quality extends beyond functional correctness to include task effectiveness and resource efficiency, which can evolve independently.

### Practical Implications

1. **For practitioners**: Pin agent harness versions for reproducibility; monitor token consumption and tool call overhead as first-class quality metrics.

2. **For harness developers**: Implement non-functional agentic regression testing—evaluate resolve rate, token consumption, and tool calls in CI/CD pipelines. The LLM Provider and Context Management layers require extra scrutiny.

3. **For researchers**: Report and control for agent harness versions, not just LLM versions, when evaluating coding agents. Consider efficiency metrics alongside effectiveness.

4. **For organizations**: The near-doubling of token consumption translates directly to increased operational costs when deploying coding agents at scale.

---

## Conclusion

### Key Takeaways

The paper provides the first controlled, longitudinal evidence that agent harness evolution—not just LLM capability—fundamentally shapes coding agent quality. Despite unprecedented development intensity (10–18 releases/week), later harness versions consume nearly double the computational resources without quality gains.

### Main Contributions

1. Characterized "hyper-churn" across five coding agent harnesses
2. First controlled longitudinal evaluation of 35 agent harness releases
3. Project-level and architecture-level explanations of quality shifts
4. Public replication package with all 35 Qwen Code CLI versions and evaluation scripts

### Future Directions

- **Multi-harness longitudinal studies**: Replicate across other harnesses (Codex, OpenCode) with broader model families
- **Partial-credit metrics**: Move beyond binary SWE-bench resolution to capture graded quality
- **Automated regression testing**: Develop cost-effective agentic evaluation pipelines suitable for CI/CD integration
- **Causal analysis**: Move from correlational to causal identification of quality drivers
- **Mitigation strategies**: Develop tools and practices to control agent harness complexity while preserving innovation velocity

> "More Code Does Not Mean Better Agents." — The central lesson: without explicit quality assurance for agentic behavior, rapid development produces complexity without capability gains.

---

_Markdown view of https://picx.dev/p/nyL7PI, served by PicX — AI-generated visual whiteboard summaries of research papers._