# When Agents Fail: Model or Harness—Who's to Blame?

> Agent benchmark scores measure model-harness pairs jointly, not model capability alone; harness choice can reverse model rankings.

- **Source:** [arXiv](https://arxiv.org/abs/2610.32483)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/z9M51I
- **Whiteboard:** https://picx.dev/p/z9M51I/image

## Summary

# When Agents Fail: Model or Harness—Who's to Blame?

## Summary (Overview)

- **Core contribution**: This paper introduces a framework for evaluating LLM agents by separating model capability from harness (execution layer) effects, arguing that agentic benchmark performance reflects the *joint* behavior of model–harness systems rather than model capability alone.
- **Five complementary metrics proposed**: Valid Completion Rate (VCR), Model–Harness Fit, Cross-Model Fit, Interaction Residuals, and Token Load—designed to distinguish task failure from execution or handoff failure.
- **Large-scale empirical study**: 111 evaluation settings across 6 benchmarks yielding 52,168 trajectories, crossing 4 general-purpose harnesses (DeepAgents, OpenCode, Pi CLI, Codex CLI) with 4 core models plus 2 fixed native reference systems.
- **Key findings**: Harness rankings vary substantially across models and benchmarks; task performance and valid completion can favor different pairings; similar task scores can accompany distinct execution profiles. OpenCode ranks first in cross-model performance, while DeepSeek V4 Flash Preview maintains consistently high Mean VCR across harnesses.
- **Implications**: The authors advocate for evaluating model–harness pairs, training transferable harness-use capabilities in models, and designing harnesses adapted to specific model–task combinations.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Classical views describe intelligent agents as goal-oriented systems that perceive, reason, and act in environments. An LLM agent combines a language model with an **execution harness**—the layer connecting the model to the environment, providing observation processing, context management, tool interfaces, workflow control, state management, execution, evaluation, and error handling.

The central problem: **end-to-end benchmark performance reflects joint model–harness behavior**, yet most benchmarks evaluate fixed configurations by task success, treating the harness as part of the setup. This conflates:
- Insufficient model capability
- Ineffective harness use by the model
- Harness limitations (observation processing, action interfaces, state management, recovery)

### Theoretical Foundation

The paper formalizes the model–harness system as follows. For a model $M_m$ and harness $H_h$, the executable agent is:

$$\mathsf{Agent}_{m,h} = H_h[M_m]$$

The harness is defined as a stateful runtime with six interfaces:

$$H_h = \langle \text{Observe}_h, \text{Context}_h, \text{Control}_h, \text{Act}_h, \text{Update}_h, \text{Govern}_h \rangle$$

The runtime loop is formalized as:

$$o_t = \text{Observe}_h(x_t, \mu_t), \qquad c_t = \text{Context}_h(g, o_{\leq t}, \mu_t), \qquad y_t \sim M_m(\cdot | c_t)$$

where $o_t$ is the observation, $c_t$ the context, $y_t$ the model's output proposal, $x_t$ the raw environmental signal, and $\mu_t$ the harness's runtime memory.

**Two key dimensions** are distinguished:
1. **Task performance**: how well the final state satisfies the task's success criterion
2. **Operational validity**: whether execution terminates normally with an evaluator-consumable state, regardless of correctness

---

## Methodology

### Benchmark Suite

Six benchmarks were selected requiring closed-loop action, matched replay, and complementary harness-sensitive demands:

| Benchmark | Tasks | Key Interaction Demands |
|---|---|---|
| SWE-bench-45 | 45 | Repository editing, testing, patch handoff |
| Terminal-Bench 2 | 89 | Stateful terminal interaction |
| VitaBench | 400 | Multi-turn services, domain APIs |
| LOCA-Bench | 75 | Long-context planning, exact-state completion |
| ClawMark | 100 | Multi-day workflows, multimodal inputs |
| AgentSkillOS | 30 | Skill selection, artifact handoff |

### Models and Harnesses

- **Harnesses**: DeepAgents, OpenCode, Pi CLI, Codex CLI
- **Core models**: DeepSeek V4 Flash Preview, MiniMax M3, Kimi K2.7, Qwen3.6-35B-A3B
- **Fixed native references**: GPT-5.5 xhigh + Codex CLI, DeepSeek V4 Pro 0813 + DeepSeek Harness

### Key Metrics

**Valid Completion Rate (VCR)** — the fraction of attempts that terminate normally with valid evaluator handoff:

$$\mathrm{VCR} = \frac{1}{|\mathcal{T}| R} \sum_{i \in \mathcal{T}} \sum_{r=1}^{R} U_{i,r} \tag{1}$$

where $U_{i,r} = N_{i,r} V_{i,r} = 1$ indicates a clean run (normal termination AND valid handoff).

**Model–Harness Fit** — task scores earned through valid completion, normalized to the best observed harness:

$$J_{m,h,b} = \frac{1}{|\mathcal{T}_b| R} \sum_{i \in \mathcal{T}_b} \sum_{r=1}^{R} U_{i,r}^{m,h,b} S_{i,r}^{m,h,b} \tag{2}$$

$$F_{m,h} = \frac{1}{|\mathcal{B}|} \sum_{b \in \mathcal{B}} \frac{J_{m,h,b}}{J_{m,b}^{\star}}, \quad \text{Cross-Model Fit}(h) = 100 \left(\prod_{m \in \mathcal{M}} F_{m,h}\right)^{1/|\mathcal{M}|} \tag{3}$$

**Interaction Residuals** — VCR deviations beyond additive model and harness averages:

$$\Delta_{m,h} = \upsilon_{m,h} - \upsilon_{m,\cdot} - \upsilon_{\cdot,h} + \upsilon_{\cdot,\cdot} \tag{4}$$

**Token Load** — mean token consumption over all scheduled attempts, including failures.

---

## Empirical Validation / Results

### RQ1: Model–Harness Interaction

**Key finding: Harness changes can reverse model rankings within a benchmark.** On SWE-bench-45, DeepSeek V4 Flash Preview outperforms MiniMax with Codex CLI, while MiniMax leads with OpenCode.

**Consistent valid completion**: DeepSeek V4 Flash Preview maintains high Mean VCR (91.30%–93.98%) across all four harnesses on the shared four-benchmark panel—a range of only 2.68 percentage points.

**Pairing-specific deviations**: MiniMax–OpenCode and Kimi–Pi exceed additive VCR baselines with statistically significant positive residuals. Deviations vary by benchmark—e.g., Kimi–Codex is below baseline on SWE-bench-45 but above it on LOCA-Bench.

### RQ2: Cross-Model Fit

**OpenCode ranks first** in Cross-Model Fit with similar Fit across all three models:

| Harness | MiniMax M3 | Kimi K2.7 | Qwen3.6 | Cross-model |
|---|---|---|---|---|
| **OpenCode** | 86.86 | 87.06 | 88.55 | **87.48** |
| Pi CLI | 73.73 | 76.16 | 81.70 | 77.13 |
| Codex CLI | 72.37 | 60.45 | 91.21 | 73.62 |
| DeepAgents | 60.56 | 61.29 | 71.29 | 64.20 |

Codex CLI leads for Qwen but its lower Kimi Fit reduces its aggregate ranking.

### Joint Outcomes and Execution Diagnostics

**Clean/non-clean divergence**: Non-clean attempts can earn task credit, while clean attempts can leave tasks unsolved. On ClawMark, Kimi–DeepAgents exceeds fully clean Kimi–Codex by 46.7 Mean Score points, with nearly a third of its score from non-clean attempts.

**Execution-stage decomposition**: DeepAgents loses nearly ten times as many runs at pre-normal termination than post-normal handoff; Codex CLI's losses are roughly balanced between stages.

**Action-realization case study** (Kimi K2.7 on ClawMark):

| Harness | Score Mean | VCR | Affected attempts % |
|---|---|---|---|
| Codex CLI | 11.96 | 100.00 | 6.75 |
| DeepAgents | 58.65 | 53.25 | 0.00 |
| OpenCode | 70.36 | 88.00 | 0.00 |
| Pi CLI | 11.93 | 99.25 | 40.50 |

High VCR can coexist with action-realization gaps—Pi CLI has 40.5% of attempts affected by strict gaps (intent without executable action), yet 99.25% VCR.

### RQ3: Metric Validation and Robustness

- Joint and official scores coincide in 37 of 72 benchmark-level cells; counting only valid completions reduces scores in the other 35.
- Omitting any single model or benchmark leaves OpenCode first, but intermediate ranks vary.
- Leave-one-model-out selection: all metrics select OpenCode in all three folds; regret is 0.00 for MiniMax and Kimi, 2.66 for Qwen.

### Resource Use

- Pi CLI leads token-use win rate (55.56%); OpenCode leads outcome-preserving win rate (35.19%) and Pareto-frontier share (61.11%).
- Token Load is strongly right-skewed: means exceed medians in 89 of 90 fully covered primary cells; the three most skewed cells have means roughly 7–12× their medians.
- The highest-token 5% of Qwen–Pi SWE runs consume ~83% of that configuration's tokens.

---

## Theoretical and Practical Implications

### For Evaluation
- Ranking reversals motivate **crossed evaluation on matched tasks**—varying model and harness independently while holding tasks fixed.
- Use model-conditioned outcomes, VCR, and execution diagnostics to distinguish task failure from execution or handoff failure.
- Cross-Model Fit supports shared harness selection; benchmark profiles and uncertainty reveal pairing differences.

### For Model Training
- Native- and mixed-harness RL provides trajectories from diverse execution interfaces.
- Ranking reversals motivate pairing the same training tasks with multiple harnesses to develop **transferable action realization, feedback use, recovery, and stopping** capabilities.
- Non-clean successes motivate combining task rewards with execution validity feedback.

### For Harness Design
- Use model-specific Fit and VCR profiles to guide interface, recovery, and context-management choices.
- Feedback should distinguish rejected from executed operations; compression should retain task-relevant state.
- Adapt resource limits to the model and task, bound failed retries, and detect stagnation and repeated actions.

---

## Conclusion

The paper demonstrates that **interpreting agent failures requires examining both the model and the harness, plus their particular pairing**. Task failure must be distinguished from execution or handoff failure: valid completion can leave a task unsolved, while abnormal termination can coexist with task credit.

The proposed framework—combining VCR, Model–Harness Fit, Cross-Model Fit, Interaction Residuals, and Token Load—provides a systematic approach to this joint assessment. The findings support:
1. Evaluating model–harness pairs rather than fixed configurations
2. Designing harnesses for reliable, efficient operation across models
3. Treating **harness-use capability as a training objective**, with the goal of sustaining strong task performance and valid completion across diverse execution interfaces, including harnesses unseen during training

**Future directions** include component-level ablations of harness mechanisms, extending evaluation to additional harnesses and models, and developing training methods that explicitly optimize for both task performance and operational validity across diverse execution interfaces.

---

_Markdown view of https://picx.dev/p/z9M51I, served by PicX — AI-generated visual whiteboard summaries of research papers._