Summary (Overview)

• TraceTrial-Bench is a novel benchmark for evaluating AI agent harnesses (the system layer connecting LLMs to tools, context, and execution environments) as distinct from the underlying models, addressing a gap in existing benchmarks that focus primarily on model capabilities and overall task outcomes.
• The benchmark organizes harness reliability into seven components (Context, Execution, Governance, Lifecycle, Observability, Tools, Verification) across 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth.
• A two-layer evaluation framework separately assesses Agent task performance and Harness reliability using a Trajectory Tracing (TT) module for execution traces and a Multi-source Evidence Acquisition (MEA) module for evidence-based LLM judging.
• Evaluation across 4 harnesses, 9 models, and 3,420 runs reveals that harness scores range from 59.8% to 72.0% (macro-averaged), and 31.1% of runs achieve task success despite failing at least one harness criterion—demonstrating that outcome-only evaluation is insufficient.
• A Failure Attribution (FA) module uses explicit mechanism chains to support run-level failure attribution, distinguishing candidate links from validated failure propagation through evidence review.

Introduction and Theoretical Foundation

Background

Large language models have evolved from text generators into the foundation of AI agents capable of complex, multi-step tasks involving tools, command execution, file modification, and external environment interaction. However, an AI agent's behavior depends not only on the underlying model but also on the harness—the system layer that manages:

• System instructions and context management
• Available tools and their interfaces
• Execution environments and persistent state
• Lifecycle controls and safety mechanisms

The paper formalizes this relationship as:

$$\text { Agent } = \text { Model } + \text { Harness }.\tag{1}$$

The Gap in Existing Evaluation

Most existing agent benchmarks evaluate task completion through final answers, test results, or environment states. These measures capture the combined effects of model and harness but provide limited insight into whether specific harness mechanisms function reliably. Key observations:

• Task success may conceal lost context, misleading tool results, or lifecycle violations
• Task failure can occur despite faithful harness execution
• Outcome-level comparisons alone cannot identify which harness behaviors account for observed differences

Theoretical Foundation

The paper establishes a clear system boundary: the harness includes orchestration and control functions (prompts, workflows, action strategies, runtime processes), while external tool implementations, task environments, output artifacts, evaluators, and measurement apparatus remain outside that boundary.

Evidence boundaries are critical: an external result, a model-facing tool return, a native session record, and a normalized event are related observations but need not contain the same information. The paper distinguishes four levels of claim:

1. Criterion verdict — satisfaction under selected evidence
2. Confirmed Harness defect — requires reconciliation with authoritative sources
3. Activated candidate link — failed endpoints with matching predeclared condition
4. Validated propagation — requires observable intermediate distortion and supported event order

Methodology

Benchmark Architecture

The system under test (SUT) is the native harness. The model, task, environment, and resource budgets are controlled inputs. The evaluator and capture machinery are measurement apparatus.

For harness $h$, model $m$, task $t$, and execution instance $r$, one run is defined as:

$$R _ {h, m, t, r} = (\tau , \Delta W, G, V _ {H}, V _ {A}, L _ {\mathrm{act}}),\tag{2}$$

where $\tau$ is the normalized event sequence, $\Delta W$ is the workspace change, $G$ is the apparatus-admission state, $V_H$ and $V_A$ are criterion verdicts for Harness and Agent layers, and $L_{\mathrm{act}}$ is the set of activated candidate mechanism links.

Harness Reliability Taxonomy

The seven dimensions and task coverage:

Trajectory Tracing (TT) and Evidence Acquisition (MEA)

TT Module: Harness-specific adapters collect native execution traces and normalize them into a shared Trace Intermediate Representation (Trace IR) with a canonical event schema (lifecycle, model surface, action/observation, stateful control, accounting/archive event families). Call–result correlation uses native call identifiers; ambiguous observations remain explicitly unlinked rather than heuristically paired.

MEA Module: Combines trace records with workspace artifacts and authoritative environment state to assemble criterion-specific evidence packets. Evidence sources include:

• Simulator ledger and final state (runner-owned private service)
• Workspace baseline and changed artifacts
• Native normalized trajectory
• Provider-request evidence (structured tool declarations)
• Execution/security probes (post-tool observers)
• Agent claims (final messages and artifacts)

Two-Layer Evaluation

The scoring formula for layer $X \in \{H, A\}$ with $n_X$ finalized items and $p_X$ passed items:

$$S = 5 0 \frac {p _ {H}}{n _ {H}} + 5 0 \frac {p _ {A}}{n _ {A}}.\tag{3}$$

Judge states include: 0 (Unsupported/fail), 1 (Supported/pass), 2 (Not exercised—resolved from Agent outcome), and 3 (Malfunction—finalized as Harness fail).

Failure Attribution

Candidate mechanism activation is defined as:

$$\ell = (h _ {\ell}, a _ {\ell}, \mu_ {\ell}),\tag{5}$$

where $h_\ell$ and $a_\ell$ are criterion endpoints and $\mu_\ell$ describes the intervening distortion mechanism. Activation:

$$L _ {\mathrm{act}} (R) = \{\ell \in \mathcal {C} _ {t} ^ {\mathrm{run}} \mid h _ {\ell} \in F _ {H}, a _ {\ell} \in Z _ {A}, q _ {\ell} (R) = 1 \}.\tag{6}$$

The Harness-side failure set is:

$$F _ {H} = \mathsf {f a i l} (V _ {H}), \qquad Z _ {H} = F _ {H} \setminus B, \qquad Z _ {A} = \mathsf {f a i l} (V _ {A}).\tag{4}$$

where $B$ is the set of Harness criteria affected by observation bypass (e.g., a wrapper command replacing exit status).

Aggregate Metrics

Agent-level task success rate:

$$M _ {A} = \frac {1}{| \mathcal {R} _ {A} |} \sum_ {R \in \mathcal {R} _ {A}} \mathbb {I} [ Z _ {A} (R) = \emptyset ],\tag{8}$$

Harness-level micro- and macro-averaged criterion pass rates:

$$M _ {H} ^ {\text { micro }} = \frac {\sum_ {R \in \mathcal {R} _ {H}} p _ {H} (R)}{\sum_ {R \in \mathcal {R} _ {H}} n _ {H} (R)}, \qquad M _ {H} ^ {\text { macro }} = \frac {1}{| \mathcal {R} _ {H} |} \sum_ {R \in \mathcal {R} _ {H}} \frac {p _ {H} (R)}{n _ {H} (R)}.\tag{9}$$

Empirical Validation / Results

Experimental Setup

• 4 harnesses: Claude Code, Codex, DeepSeek Harness, OpenClaw
• 9 models: DeepSeek-V4-Pro, Gemini 3.7 Flash, GLM-5.3, GPT-5.6-Sol, Grok-4.6, HY3, Kimi K3, Opus 5, Qwen3.8-Max
• Full factorial matrix: $4 \times 9 \times 95 \times 1 = 3{,}420$ cells
• Judge: deepseek-v4-flash (fixed identity across all evaluations)

Aggregate Results

Table 6: Aggregate results (%)

Key findings:

• Claude Code has the highest Agent task success (44.0%) and combined score (73.5%)
• OpenClaw has the highest Harness macro rate (72.0%) and jointly clean rate (11.7%)
• The harness with the highest task success is NOT the one with the highest harness reliability
• Harness macro range: 12.2 points (Codex 59.8% to OpenClaw 72.0%)
• Only 8.4% of runs pass every criterion in both layers

Reliability by Taxonomy Dimension

Table 9: Reliability by taxonomy dimension

Execution has the lowest pooled Harness micro rate (64.6%) with the largest cross-harness spread (Codex 51.5% vs. OpenClaw 77.1%). The layer gap changes direction across dimensions—Execution has Agent score 78.0% vs. Harness 64.7%, while Tools reverses this (53.2% Agent vs. 66.8% Harness).

Failure States and Candidate Attribution

Table 10: Failure states and candidate-link activation

Critical finding: 31.1% of all runs (1,065 of 3,420) achieve Agent success despite at least one failed Harness criterion (the "H-only" state). This demonstrates that outcome-only evaluation cannot characterize harness reliability.

Model–Harness Interactions

• Within-model Harness macro ranges span 10.0–16.2 percentage points
• Grok-4.6 Agent success ranges from 33.7% (OpenClaw) to 55.8% (Claude Code)—a 22.1-point difference
• Opus 5 ranges from 30.5% (Codex) to 53.7% (Claude Code)—a 23.2-point difference
• Codex has the lowest Harness macro rate in every model group

Resource Usage

Mean estimated Agent costs per run: $0.436 (Claude Code),$0.595 (Codex), $0.446 (DeepSeek Harness),$0.633 (OpenClaw). Mean total tokens: 578.8k, 617.2k, 372.7k, 680.0k respectively.

Theoretical and Practical Implications

Theoretical Implications

1. Agent performance is a system property: The results empirically validate the $\text{Agent} = \text{Model} + \text{Harness}$ abstraction—the same model exhibits substantially different task success rates across harnesses (up to 23.2-point differences).

2. Outcome quality and harness reliability are orthogonal: Task success and harness-criterion satisfaction yield different harness rankings, and 31.1% of runs achieve task success despite harness failures. This challenges the assumption that task outcomes proxy for system reliability.

3. Failure attribution requires mechanism-level evidence: Co-occurrence of Harness and Agent failures does not imply propagation—fewer than half of jointly failing runs contain a stored candidate link, and 85.7% of Agent-failure runs retain at least one failed criterion not reached by any link.

Practical Implications

1. Harness selection should be task-aware: Different harnesses excel in different dimensions (e.g., OpenClaw leads in Execution, Governance; Claude Code leads in Context, Observability). Harness choice should reflect the reliability requirements of target tasks.

2. Diagnostic value of two-layer evaluation: The benchmark reveals mechanism-level shortfalls even when Agent task-success rates are high (e.g., cross-turn fact retention has 48.4% Harness satisfaction but 95.8% Agent success), enabling targeted improvement.

3. Trace-guided improvement loops: The benchmark supports identifying mechanism failures, revising harness behavior, and evaluating changes on held-out tasks—a foundation for harness evolution and recursive self-improvement.

Conclusion

TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior. Key takeaways:

• 95 mechanism-oriented tasks across seven reliability components with executable or state-based ground truth
• Two-layer evaluation separately assesses Agent performance and Harness reliability using trajectory tracing and multi-source evidence
• Failure attribution through explicit mechanism chains distinguishes candidate links from validated propagation
• Empirical results show harness macro scores from 59.8% to 72.0%, with only 8.4% of runs clean across both layers, and 31.1% of runs achieving task success despite harness failures

Future directions include harness evolution through trace-guided improvement loops, evaluation for recursive self-improvement (combining task outcomes, harness behavior, and evidence quality for actionable feedback), and extending evaluation to held-out tasks and unfamiliar harnesses to assess transfer of improvements.