When Agents Fail: Model or Harness—Who's to Blame?

Summary (Overview)

• Core contribution: This paper introduces a framework for evaluating LLM agents by separating model capability from harness (execution layer) effects, arguing that agentic benchmark performance reflects the joint behavior of model–harness systems rather than model capability alone.
• Five complementary metrics proposed: Valid Completion Rate (VCR), Model–Harness Fit, Cross-Model Fit, Interaction Residuals, and Token Load—designed to distinguish task failure from execution or handoff failure.
• Large-scale empirical study: 111 evaluation settings across 6 benchmarks yielding 52,168 trajectories, crossing 4 general-purpose harnesses (DeepAgents, OpenCode, Pi CLI, Codex CLI) with 4 core models plus 2 fixed native reference systems.
• Key findings: Harness rankings vary substantially across models and benchmarks; task performance and valid completion can favor different pairings; similar task scores can accompany distinct execution profiles. OpenCode ranks first in cross-model performance, while DeepSeek V4 Flash Preview maintains consistently high Mean VCR across harnesses.
• Implications: The authors advocate for evaluating model–harness pairs, training transferable harness-use capabilities in models, and designing harnesses adapted to specific model–task combinations.

---

Introduction and Theoretical Foundation

Background and Motivation

Classical views describe intelligent agents as goal-oriented systems that perceive, reason, and act in environments. An LLM agent combines a language model with an execution harness—the layer connecting the model to the environment, providing observation processing, context management, tool interfaces, workflow control, state management, execution, evaluation, and error handling.

The central problem: end-to-end benchmark performance reflects joint model–harness behavior, yet most benchmarks evaluate fixed configurations by task success, treating the harness as part of the setup. This conflates:

• Insufficient model capability
• Ineffective harness use by the model
• Harness limitations (observation processing, action interfaces, state management, recovery)

Theoretical Foundation

The paper formalizes the model–harness system as follows. For a model $M_m$ and harness $H_h$, the executable agent is:

$$\mathsf{Agent}_{m,h} = H_h[M_m]$$

The harness is defined as a stateful runtime with six interfaces:

$$H_h = \langle \text{Observe}_h, \text{Context}_h, \text{Control}_h, \text{Act}_h, \text{Update}_h, \text{Govern}_h \rangle$$

The runtime loop is formalized as:

$$o_t = \text{Observe}_h(x_t, \mu_t), \qquad c_t = \text{Context}_h(g, o_{\leq t}, \mu_t), \qquad y_t \sim M_m(\cdot | c_t)$$

where $o_t$ is the observation, $c_t$ the context, $y_t$ the model's output proposal, $x_t$ the raw environmental signal, and $\mu_t$ the harness's runtime memory.

Two key dimensions are distinguished:

1. Task performance: how well the final state satisfies the task's success criterion
2. Operational validity: whether execution terminates normally with an evaluator-consumable state, regardless of correctness

---

Methodology

Benchmark Suite

Six benchmarks were selected requiring closed-loop action, matched replay, and complementary harness-sensitive demands:

Models and Harnesses

• Harnesses: DeepAgents, OpenCode, Pi CLI, Codex CLI
• Core models: DeepSeek V4 Flash Preview, MiniMax M3, Kimi K2.7, Qwen3.6-35B-A3B
• Fixed native references: GPT-5.5 xhigh + Codex CLI, DeepSeek V4 Pro 0813 + DeepSeek Harness

Key Metrics

Valid Completion Rate (VCR) — the fraction of attempts that terminate normally with valid evaluator handoff:

$$\mathrm{VCR} = \frac{1}{|\mathcal{T}| R} \sum_{i \in \mathcal{T}} \sum_{r=1}^{R} U_{i,r} \tag{1}$$

where $U_{i,r} = N_{i,r} V_{i,r} = 1$ indicates a clean run (normal termination AND valid handoff).

Model–Harness Fit — task scores earned through valid completion, normalized to the best observed harness:

$$J_{m,h,b} = \frac{1}{|\mathcal{T}_b| R} \sum_{i \in \mathcal{T}_b} \sum_{r=1}^{R} U_{i,r}^{m,h,b} S_{i,r}^{m,h,b} \tag{2}$$ $$F_{m,h} = \frac{1}{|\mathcal{B}|} \sum_{b \in \mathcal{B}} \frac{J_{m,h,b}}{J_{m,b}^{\star}}, \quad \text{Cross-Model Fit}(h) = 100 \left(\prod_{m \in \mathcal{M}} F_{m,h}\right)^{1/|\mathcal{M}|} \tag{3}$$

Interaction Residuals — VCR deviations beyond additive model and harness averages:

$$\Delta_{m,h} = \upsilon_{m,h} - \upsilon_{m,\cdot} - \upsilon_{\cdot,h} + \upsilon_{\cdot,\cdot} \tag{4}$$

Token Load — mean token consumption over all scheduled attempts, including failures.

---

Empirical Validation / Results

RQ1: Model–Harness Interaction

Key finding: Harness changes can reverse model rankings within a benchmark. On SWE-bench-45, DeepSeek V4 Flash Preview outperforms MiniMax with Codex CLI, while MiniMax leads with OpenCode.

Consistent valid completion: DeepSeek V4 Flash Preview maintains high Mean VCR (91.30%–93.98%) across all four harnesses on the shared four-benchmark panel—a range of only 2.68 percentage points.

Pairing-specific deviations: MiniMax–OpenCode and Kimi–Pi exceed additive VCR baselines with statistically significant positive residuals. Deviations vary by benchmark—e.g., Kimi–Codex is below baseline on SWE-bench-45 but above it on LOCA-Bench.

RQ2: Cross-Model Fit

OpenCode ranks first in Cross-Model Fit with similar Fit across all three models:

Codex CLI leads for Qwen but its lower Kimi Fit reduces its aggregate ranking.

Joint Outcomes and Execution Diagnostics

Clean/non-clean divergence: Non-clean attempts can earn task credit, while clean attempts can leave tasks unsolved. On ClawMark, Kimi–DeepAgents exceeds fully clean Kimi–Codex by 46.7 Mean Score points, with nearly a third of its score from non-clean attempts.

Execution-stage decomposition: DeepAgents loses nearly ten times as many runs at pre-normal termination than post-normal handoff; Codex CLI's losses are roughly balanced between stages.

Action-realization case study (Kimi K2.7 on ClawMark):

High VCR can coexist with action-realization gaps—Pi CLI has 40.5% of attempts affected by strict gaps (intent without executable action), yet 99.25% VCR.

RQ3: Metric Validation and Robustness

• Joint and official scores coincide in 37 of 72 benchmark-level cells; counting only valid completions reduces scores in the other 35.
• Omitting any single model or benchmark leaves OpenCode first, but intermediate ranks vary.
• Leave-one-model-out selection: all metrics select OpenCode in all three folds; regret is 0.00 for MiniMax and Kimi, 2.66 for Qwen.

Resource Use

• Pi CLI leads token-use win rate (55.56%); OpenCode leads outcome-preserving win rate (35.19%) and Pareto-frontier share (61.11%).
• Token Load is strongly right-skewed: means exceed medians in 89 of 90 fully covered primary cells; the three most skewed cells have means roughly 7–12× their medians.
• The highest-token 5% of Qwen–Pi SWE runs consume ~83% of that configuration's tokens.

---

Theoretical and Practical Implications

For Evaluation

• Ranking reversals motivate crossed evaluation on matched tasks—varying model and harness independently while holding tasks fixed.
• Use model-conditioned outcomes, VCR, and execution diagnostics to distinguish task failure from execution or handoff failure.
• Cross-Model Fit supports shared harness selection; benchmark profiles and uncertainty reveal pairing differences.

For Model Training

• Native- and mixed-harness RL provides trajectories from diverse execution interfaces.
• Ranking reversals motivate pairing the same training tasks with multiple harnesses to develop transferable action realization, feedback use, recovery, and stopping capabilities.
• Non-clean successes motivate combining task rewards with execution validity feedback.

For Harness Design

• Use model-specific Fit and VCR profiles to guide interface, recovery, and context-management choices.
• Feedback should distinguish rejected from executed operations; compression should retain task-relevant state.
• Adapt resource limits to the model and task, bound failed retries, and detect stagnation and repeated actions.

---

Conclusion

The paper demonstrates that interpreting agent failures requires examining both the model and the harness, plus their particular pairing. Task failure must be distinguished from execution or handoff failure: valid completion can leave a task unsolved, while abnormal termination can coexist with task credit.

The proposed framework—combining VCR, Model–Harness Fit, Cross-Model Fit, Interaction Residuals, and Token Load—provides a systematic approach to this joint assessment. The findings support:

1. Evaluating model–harness pairs rather than fixed configurations
2. Designing harnesses for reliable, efficient operation across models
3. Treating harness-use capability as a training objective, with the goal of sustaining strong task performance and valid completion across diverse execution interfaces, including harnesses unseen during training

Future directions include component-level ablations of harness mechanisms, extending evaluation to additional harnesses and models, and developing training methods that explicitly optimize for both task performance and operational validity across diverse execution interfaces.