# Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

> TraceTrial-Bench reveals that 31.1% of AI agent runs succeed despite harness failures, proving task outcomes alone cannot characterize harness reliability.

- **Source:** [arXiv](https://arxiv.org/abs/2610.32285)
- **Published:** 2026-10-28
- **Permalink:** https://picx.dev/p/suVvr3
- **Whiteboard:** https://picx.dev/p/suVvr3/image

## Summary

## Summary (Overview)

- **TraceTrial-Bench** is a novel benchmark for evaluating AI agent **harnesses** (the system layer connecting LLMs to tools, context, and execution environments) as distinct from the underlying models, addressing a gap in existing benchmarks that focus primarily on model capabilities and overall task outcomes.
- The benchmark organizes harness reliability into **seven components** (Context, Execution, Governance, Lifecycle, Observability, Tools, Verification) across **95 expert-level, mechanism-oriented tasks** with executable or state-based ground truth.
- A **two-layer evaluation framework** separately assesses Agent task performance and Harness reliability using a Trajectory Tracing (TT) module for execution traces and a Multi-source Evidence Acquisition (MEA) module for evidence-based LLM judging.
- Evaluation across **4 harnesses, 9 models, and 3,420 runs** reveals that harness scores range from 59.8% to 72.0% (macro-averaged), and **31.1% of runs achieve task success despite failing at least one harness criterion**—demonstrating that outcome-only evaluation is insufficient.
- A **Failure Attribution (FA) module** uses explicit mechanism chains to support run-level failure attribution, distinguishing candidate links from validated failure propagation through evidence review.

## Introduction and Theoretical Foundation

### Background

Large language models have evolved from text generators into the foundation of AI agents capable of complex, multi-step tasks involving tools, command execution, file modification, and external environment interaction. However, an AI agent's behavior depends not only on the underlying model but also on the **harness**—the system layer that manages:

- System instructions and context management
- Available tools and their interfaces
- Execution environments and persistent state
- Lifecycle controls and safety mechanisms

The paper formalizes this relationship as:

$$
\text { Agent } = \text { Model } + \text { Harness }.\tag{1}
$$

### The Gap in Existing Evaluation

Most existing agent benchmarks evaluate task completion through final answers, test results, or environment states. These measures capture the *combined* effects of model and harness but provide limited insight into whether specific harness mechanisms function reliably. Key observations:

- **Task success may conceal** lost context, misleading tool results, or lifecycle violations
- **Task failure can occur** despite faithful harness execution
- Outcome-level comparisons alone **cannot identify which harness behaviors** account for observed differences

### Theoretical Foundation

The paper establishes a clear **system boundary**: the harness includes orchestration and control functions (prompts, workflows, action strategies, runtime processes), while external tool implementations, task environments, output artifacts, evaluators, and measurement apparatus remain outside that boundary.

**Evidence boundaries** are critical: an external result, a model-facing tool return, a native session record, and a normalized event are related observations but need not contain the same information. The paper distinguishes **four levels of claim**:

1. **Criterion verdict** — satisfaction under selected evidence
2. **Confirmed Harness defect** — requires reconciliation with authoritative sources
3. **Activated candidate link** — failed endpoints with matching predeclared condition
4. **Validated propagation** — requires observable intermediate distortion and supported event order

## Methodology

### Benchmark Architecture

The system under test (SUT) is the **native harness**. The model, task, environment, and resource budgets are controlled inputs. The evaluator and capture machinery are measurement apparatus.

For harness $h$, model $m$, task $t$, and execution instance $r$, one run is defined as:

$$
R _ {h, m, t, r} = (\tau , \Delta W, G, V _ {H}, V _ {A}, L _ {\mathrm{act}}),\tag{2}
$$

where $\tau$ is the normalized event sequence, $\Delta W$ is the workspace change, $G$ is the apparatus-admission state, $V_H$ and $V_A$ are criterion verdicts for Harness and Agent layers, and $L_{\mathrm{act}}$ is the set of activated candidate mechanism links.

### Harness Reliability Taxonomy

The seven dimensions and task coverage:

| Dimension | Evaluation focus | Tasks |
|---|---|---|
| Context (C) | Information preservation and disclosure across boundaries | 15 |
| Execution (E) | Faithful process semantics, state, and result delivery | 27 |
| Governance (G) | Approvals, guardrails, secret redaction, and trust boundaries | 9 |
| Lifecycle (L) | Correct state transitions, recovery, and idempotent retries | 19 |
| Observability (O) | Model-visible diagnostics, replayable logs, and provenance | 7 |
| Tools (T) | Valid tool interfaces, faithful transport, and discovery | 17 |
| Verification (V) | Consistency of completion claims with authoritative outcomes | 1 |
| **Total** | | **95** |

### Trajectory Tracing (TT) and Evidence Acquisition (MEA)

**TT Module**: Harness-specific adapters collect native execution traces and normalize them into a shared **Trace Intermediate Representation (Trace IR)** with a canonical event schema (lifecycle, model surface, action/observation, stateful control, accounting/archive event families). Call–result correlation uses native call identifiers; ambiguous observations remain explicitly unlinked rather than heuristically paired.

**MEA Module**: Combines trace records with workspace artifacts and authoritative environment state to assemble criterion-specific evidence packets. Evidence sources include:

- Simulator ledger and final state (runner-owned private service)
- Workspace baseline and changed artifacts
- Native normalized trajectory
- Provider-request evidence (structured tool declarations)
- Execution/security probes (post-tool observers)
- Agent claims (final messages and artifacts)

### Two-Layer Evaluation

The scoring formula for layer $X \in \{H, A\}$ with $n_X$ finalized items and $p_X$ passed items:

$$
S = 5 0 \frac {p _ {H}}{n _ {H}} + 5 0 \frac {p _ {A}}{n _ {A}}.\tag{3}
$$

**Judge states** include: 0 (Unsupported/fail), 1 (Supported/pass), 2 (Not exercised—resolved from Agent outcome), and 3 (Malfunction—finalized as Harness fail).

### Failure Attribution

Candidate mechanism activation is defined as:

$$
\ell = (h _ {\ell}, a _ {\ell}, \mu_ {\ell}),\tag{5}
$$

where $h_\ell$ and $a_\ell$ are criterion endpoints and $\mu_\ell$ describes the intervening distortion mechanism. Activation:

$$
L _ {\mathrm{act}} (R) = \{\ell \in \mathcal {C} _ {t} ^ {\mathrm{run}} \mid h _ {\ell} \in F _ {H}, a _ {\ell} \in Z _ {A}, q _ {\ell} (R) = 1 \}.\tag{6}
$$

The Harness-side failure set is:

$$
F _ {H} = \mathsf {f a i l} (V _ {H}), \qquad Z _ {H} = F _ {H} \setminus B, \qquad Z _ {A} = \mathsf {f a i l} (V _ {A}).\tag{4}
$$

where $B$ is the set of Harness criteria affected by observation bypass (e.g., a wrapper command replacing exit status).

### Aggregate Metrics

Agent-level task success rate:

$$
M _ {A} = \frac {1}{| \mathcal {R} _ {A} |} \sum_ {R \in \mathcal {R} _ {A}} \mathbb {I} [ Z _ {A} (R) = \emptyset ],\tag{8}
$$

Harness-level micro- and macro-averaged criterion pass rates:

$$
M _ {H} ^ {\text { micro }} = \frac {\sum_ {R \in \mathcal {R} _ {H}} p _ {H} (R)}{\sum_ {R \in \mathcal {R} _ {H}} n _ {H} (R)}, \qquad M _ {H} ^ {\text { macro }} = \frac {1}{| \mathcal {R} _ {H} |} \sum_ {R \in \mathcal {R} _ {H}} \frac {p _ {H} (R)}{n _ {H} (R)}.\tag{9}
$$

## Empirical Validation / Results

### Experimental Setup

- **4 harnesses**: Claude Code, Codex, DeepSeek Harness, OpenClaw
- **9 models**: DeepSeek-V4-Pro, Gemini 3.7 Flash, GLM-5.3, GPT-5.6-Sol, Grok-4.6, HY3, Kimi K3, Opus 5, Qwen3.8-Max
- **Full factorial matrix**: $4 \times 9 \times 95 \times 1 = 3{,}420$ cells
- **Judge**: deepseek-v4-flash (fixed identity across all evaluations)

### Aggregate Results

**Table 6: Aggregate results (%)**

| Harness | Agent Pass | Agent Score | Harness Micro | Harness Macro | Clean | Overall |
|---|---|---|---|---|---|---|
| Claude Code | 44.0 | 75.9 | 71.5 | 71.2 | 9.1 | 73.5 |
| Codex | 35.8 | 70.7 | 60.2 | 59.8 | 2.8 | 65.2 |
| DeepSeek Harness | 40.0 | 73.3 | 69.5 | 68.8 | 9.8 | 71.0 |
| OpenClaw | 38.2 | 71.7 | 71.9 | 72.0 | 11.7 | 71.8 |
| **All harnesses** | **39.5** | **72.9** | **68.3** | **67.9** | **8.4** | **70.4** |

**Key findings**:
- Claude Code has the highest Agent task success (44.0%) and combined score (73.5%)
- OpenClaw has the highest Harness macro rate (72.0%) and jointly clean rate (11.7%)
- **The harness with the highest task success is NOT the one with the highest harness reliability**
- Harness macro range: 12.2 points (Codex 59.8% to OpenClaw 72.0%)
- Only 8.4% of runs pass every criterion in both layers

### Reliability by Taxonomy Dimension

**Table 9: Reliability by taxonomy dimension**

| Dimension | Tasks | Claude Code | Codex | DeepSeek | OpenClaw | Pooled H Micro | Agent Pass |
|---|---|---|---|---|---|---|---|
| Context | 15 | 74.5 | 62.0 | 64.3 | 61.6 | 65.6 | 45.2 |
| Execution | 27 | 64.7 | 51.5 | 65.3 | **77.1** | 64.6 | 49.2 |
| Governance | 9 | 72.3 | 74.2 | 68.8 | **80.5** | 74.0 | 27.2 |
| Lifecycle | 19 | 71.3 | 53.0 | 71.7 | 65.9 | 65.5 | 43.6 |
| Observability | 7 | **88.7** | 76.5 | 86.8 | 85.0 | 84.3 | 43.3 |
| Tools | 17 | 71.5 | 65.6 | 71.5 | 70.2 | 69.7 | 21.4 |
| Verification | 1 | 93.7 | **95.2** | 82.5 | 88.9 | 90.1 | 8.3 |

Execution has the lowest pooled Harness micro rate (64.6%) with the largest cross-harness spread (Codex 51.5% vs. OpenClaw 77.1%). The layer gap changes direction across dimensions—Execution has Agent score 78.0% vs. Harness 64.7%, while Tools reverses this (53.2% Agent vs. 66.8% Harness).

### Failure States and Candidate Attribution

**Table 10: Failure states and candidate-link activation**

| Group | Runs | Clean | H-only | A-only | Both | Linked | Link/Both (%) |
|---|---|---|---|---|---|---|---|
| All Harnesses | 3420 | 286 | 1065 | 369 | 1700 | 842 | 49.5 |
| Claude Code | 855 | 78 | 298 | 97 | 382 | 194 | 50.8 |
| Codex | 855 | 24 | 282 | 58 | 491 | 240 | 48.9 |
| DeepSeek Harness | 855 | 84 | 258 | 71 | 442 | 186 | 42.1 |
| OpenClaw | 855 | 100 | 227 | 143 | 385 | 222 | 57.7 |

**Critical finding**: **31.1% of all runs (1,065 of 3,420) achieve Agent success despite at least one failed Harness criterion** (the "H-only" state). This demonstrates that outcome-only evaluation cannot characterize harness reliability.

### Model–Harness Interactions

- Within-model Harness macro ranges span 10.0–16.2 percentage points
- Grok-4.6 Agent success ranges from 33.7% (OpenClaw) to 55.8% (Claude Code)—a 22.1-point difference
- Opus 5 ranges from 30.5% (Codex) to 53.7% (Claude Code)—a 23.2-point difference
- Codex has the lowest Harness macro rate in **every** model group

### Resource Usage

Mean estimated Agent costs per run: $0.436 (Claude Code), $0.595 (Codex), $0.446 (DeepSeek Harness), $0.633 (OpenClaw). Mean total tokens: 578.8k, 617.2k, 372.7k, 680.0k respectively.

## Theoretical and Practical Implications

### Theoretical Implications

1. **Agent performance is a system property**: The results empirically validate the $\text{Agent} = \text{Model} + \text{Harness}$ abstraction—the same model exhibits substantially different task success rates across harnesses (up to 23.2-point differences).

2. **Outcome quality and harness reliability are orthogonal**: Task success and harness-criterion satisfaction yield different harness rankings, and 31.1% of runs achieve task success despite harness failures. This challenges the assumption that task outcomes proxy for system reliability.

3. **Failure attribution requires mechanism-level evidence**: Co-occurrence of Harness and Agent failures does not imply propagation—fewer than half of jointly failing runs contain a stored candidate link, and 85.7% of Agent-failure runs retain at least one failed criterion not reached by any link.

### Practical Implications

1. **Harness selection should be task-aware**: Different harnesses excel in different dimensions (e.g., OpenClaw leads in Execution, Governance; Claude Code leads in Context, Observability). Harness choice should reflect the reliability requirements of target tasks.

2. **Diagnostic value of two-layer evaluation**: The benchmark reveals mechanism-level shortfalls even when Agent task-success rates are high (e.g., cross-turn fact retention has 48.4% Harness satisfaction but 95.8% Agent success), enabling targeted improvement.

3. **Trace-guided improvement loops**: The benchmark supports identifying mechanism failures, revising harness behavior, and evaluating changes on held-out tasks—a foundation for harness evolution and recursive self-improvement.

## Conclusion

TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior. Key takeaways:

- **95 mechanism-oriented tasks** across seven reliability components with executable or state-based ground truth
- **Two-layer evaluation** separately assesses Agent performance and Harness reliability using trajectory tracing and multi-source evidence
- **Failure attribution** through explicit mechanism chains distinguishes candidate links from validated propagation
- **Empirical results** show harness macro scores from 59.8% to 72.0%, with only 8.4% of runs clean across both layers, and 31.1% of runs achieving task success despite harness failures

**Future directions** include harness evolution through trace-guided improvement loops, evaluation for recursive self-improvement (combining task outcomes, harness behavior, and evidence quality for actionable feedback), and extending evaluation to held-out tasks and unfamiliar harnesses to assess transfer of improvements.

---

_Markdown view of https://picx.dev/p/suVvr3, served by PicX — AI-generated visual whiteboard summaries of research papers._