CoTrace: Data Recipes for Training Terminal Agents with Harness–Model Co-Evolution

Summary (Overview)

• Core contribution: CoTrace is a harness-aware data recipe that explicitly governs how execution trajectories flow between policy training and harness search during model–harness co-evolution, addressing the overlooked issue that a trajectory's value depends on the harness under which it was generated.

• Three key mechanisms: The framework implements Route (separating failure evidence for harness search from verified successes for policy training), Ratchet (component-wise promotion decisions that isolate and preserve gains), and Refresh (curriculum progression that retires solved-and-harvested tasks).

• Key results: On the 102-task Tmax promotion split, CoTrace advances Qwen3.5-9B from 78 to 88 solved tasks under supervised fine-tuning, while an online reinforcement variant (CoTrace-RL) reaches 90. A compact harness-matched corpus (30–50 trajectories/iteration) produces steady gains at substantially lower compute than much larger pooled corpora (149–308 trajectories/iteration) which yield zero accepted model updates.

• Transfer findings: Out-of-distribution transfer on Terminal-Bench 2.1 and SWE-bench Lite depends fundamentally on harness compatibility—deploying a learned checkpoint within its co-evolved runtime dramatically suppresses early execution faults and no-patch failures, whereas checkpoint-only gains largely disappear under foreign runtimes.

Introduction and Theoretical Foundation

Background and Motivation

Terminal agents combine two inseparable components:

• A language model acting as the action-selection policy (θ)
• A runtime harness (H) that formats prompts, binds tools, manages context, and recovers from execution errors

Prior research has largely optimized either component in isolation:

• Verifier-grounded policy post-training under a fixed runtime
• Automated harness search with frozen model weights

Although recent systems interleave harness adaptation with policy training, they leave the shared data interface between the two optimizers largely implicit. Because each update changes the conditions under which subsequent trajectories are collected, alternating the two processes without an explicit data recipe creates subtle failure modes.

Key Insight

"An execution trajectory has different value for the harness and the policy. Execution failures can expose missing recovery logic and thereby guide harness mutation, but they are poor targets for policy imitation."

The paper demonstrates a concrete failure mode: in one experiment, harness updates add nine solved tasks, while the promotion procedure rejects every model update trained on pooled candidate successes.

Problem Formulation

An autonomous terminal agent is a pair of a parameterized policy θ and an execution harness H. A task $x = (q_x, s_x, v_x) \sim \mathcal{T}$ specifies an instruction $q_x$, an initial sandbox state $s_x$, and a deterministic verifier $v_x$ of the terminal state. The harness formats prompt context, binds tools, parses feedback, and handles error recovery, so executing the pair induces a rollout distribution $\tau \sim \pi(\cdot \mid \theta, H, x)$.

Model–harness co-evolution maximizes:

$$\max_{\theta, H} J(\theta, H) = \mathbb{E}_{x \sim \mathcal{T}, \tau \sim \pi(\cdot | \theta, H, x)}[v_x(\tau)], \tag{1}$$

which is optimized alternately over iterations $t$: $H_{t+1} \approx \arg\max_H J(\theta_t, H)$ and $\theta_{t+1} \approx \arg\max_\theta J(\theta, H_{t+1})$.

Methodology

CoTrace Architecture

CoTrace governs the data interface within an alternating co-evolution loop over a versioned trajectory bank $B_{0:t}$. Instead of pooling all generated traces into an undifferentiated replay buffer, CoTrace coordinates information exchange through three operations:

1. Route: Separate Evidence for Each Optimizer

• Failure view $\mathcal{D}_t^H = R_H(B_{0:t})$: keeps executions with runtime faults (not environment/grader errors), clusters them by earliest unrecovered failure, and uses the evidence to propose harness changes.
• Model view $\mathcal{D}_t^M = R_M(B_{0:t})$: keeps verified, well-formed demonstrations outside the promotion split V.

The SFT corpus is assembled by solving:

$$\max_{\mathcal{S}_t} \sum_{\tau \in \mathcal{S}_t} Q(\tau) \quad \text{s.t.} \quad |\mathcal{S}_t| \leq C, \quad |\{\tau \in \mathcal{S}_t: \operatorname{task}(\tau) = x\}| \leq c_r \, \forall x, \tag{2}$$

where $Q$ scores trajectory quality, $C$ bounds the corpus, and $c_r$ caps trajectories per task.

Provenance is recorded per trajectory as a fingerprint $\phi(\tau)$ hashing the prompt template, tool bindings, and processors it ran under—not merely a harness identifier.

Two training modes:

• CoTrace-SFT: Retains demonstrations whose fingerprint matches the adopted harness, prioritizes current successes over bounded historical replay, limits repeated examples per task, and adds fresh rollouts under the adopted harness when coverage is insufficient.
• CoTrace-RL: Uses fresh on-policy interactions and rewards collected under the adopted harness; runtime alignment holds by construction.

2. Ratchet: Isolate and Preserve Component Gains

Every candidate c is evaluated on the frozen promotion split V against the incumbent's task-by-task record. Letting $w$ and $\ell$ denote tasks newly solved and newly broken on V:

$$\operatorname{Promote}(c) \iff w > \ell, \tag{3}$$

A candidate harness is scored with $\theta_t$ fixed, while a candidate policy is scored with $H_{t+1}$ fixed. This coordinate promotion protocol attributes each accepted increment to the isolated component that changed.

3. Refresh: Track the Learning Frontier

A task is retired from the evolve set $\mathcal{E}_t$ once it is both solved by the incumbent and harvested (selected into a built SFT corpus). The evolve set is then replenished from the reservoir $\mathcal{T}_{pool}$. Requiring both conditions prevents a task from leaving before its success becomes usable training data.

Algorithm Overview

Algorithm 1: CoTrace Alternating Co-Evolution Loop (Single Iteration)

Phase 1: Failure-Guided Harness Search
  - Cluster recurring runtime faults from trajectory bank
  - Explore harness candidates with θ_t fixed
  - Adopt if w_H > ℓ_H on V (Ratchet)

Phase 2: Runtime-Matched Policy Learning
  - SFT mode: match fingerprint φ(H_{t+1}), top up with fresh rollouts
  - RL mode: sample rollouts and rewards online under H_{t+1}
  - Adopt if w_M > ℓ_M on V (Ratchet)

Phase 3: Moving Curriculum Frontier
  - Retire tasks that are solved AND harvested
  - Replenish evolve set from reservoir with domain stratification

Experimental Setup

• Primary testbed: Tmax benchmark (2,200 executable terminal tasks with containerized environments and programmatic verifiers)
• Splits: Promotion split V (102 tasks), rotating evolve set $\mathcal{E}_t$ (50 tasks), unvisited reservoir $\mathcal{T}_{pool}$, plus 100 tasks for RL
• OOD evaluation: Terminal-Bench 2.1 (89 tasks), SWE-bench Lite (300 instances)
• Models: Qwen3.5-9B and Qwen3.5-4B
• Harness search: Typed configuration space of HARNESSX via two-generation tournament (5+5)
• Policy training: LoRA for SFT, DPPO for RL

Empirical Validation / Results

Main Co-Evolution Results (Table 1)

Key observations:

• Mixed siblings reaches 86 but all +9 gains come from harness; model updates contribute zero
• CoTrace-SFT achieves +6 through the model channel with only 30–50 trajectories per iteration
• CoTrace-RL reaches 90 with +7 from reinforcement updates
• At 4B, SFT yields no model-side gain, but RL contributes +12 entirely through model updates

Data Recipe Comparison (Table 2)

Critical finding: "Pooling 149–308 successful trajectories per iteration from exploratory sibling harnesses yields zero accepted model updates, while our harness-matched recipe utilizes merely 30–50 trajectories to deliver consistent gains, achieving superior performance at the lowest per-iteration training cost of 47 GPU-hours."

Cross-Evaluation (Table 3)

Completed cross-evaluation for CoTrace-SFT, iterations 1–2:

The interaction term $I_t = \Delta_M(H_{t+1}) - \Delta_M(H_t) = 0$, showing the harness and model gains add cleanly rather than interact.

RQ1: Channel Contributions

• Both channels contribute at different stages; a pair-level total conceals which optimizer is active
• Only 9 of 17 search-winning harness candidates improved the promotion split
• As the pair improves, the evolve set hardens (e.g., 34→26→17 of 50 solved) while the promotion score rises (75→81→83 of 102)
• Late candidates in the reinforcement chain (trained after 40 of 50 evolve tasks retired) give tasks back (85 vs. incumbent 90)

RQ2: Trajectory Attributes

• Volume alone does not explain model-side gain: mixed siblings (149–308 trajectories) yields ΣΔ_M = 0, while winner-only + SFT-gen (30–50) yields +6
• Runtime matching directly explains why the pooled corpus is dominated by rollouts from harnesses the promotion procedure later rejects
• At 4B, winner-only filter kept only 2–3 matched trajectories per iteration (below the top-up floor), and all updates were rejected; online RL still obtains relative reward signal from the same policy

RQ3: Cross-Domain Transfer

• Under foreign runtime (TB2.1, baseline harness): Supervised checkpoint ≈ base model (18.3±2.5 vs. 18.7±1.5 tasks of 89); RL checkpoint ≈ one task above base—both within trial-to-trial spread despite 10–12 task gains on Tmax
• Under matched runtime (SWE-bench Lite):
  • mini-swe-agent (foreign scaffold): SFT resolves 24.7%, base 34.7%, RL 36.7%
  • Co-evolved harness: SFT improves to 35.0% (no-patch instances drop from 155 to 64), RL reaches 41.0%

"Deploying the learned checkpoint within its co-evolved runtime dramatically suppresses early execution faults and no-patch failures, effectively unlocking the policy's underlying problem-solving capability across foreign domains without task-specific tuning."

Theoretical and Practical Implications

Theoretical Implications

1. Trajectory provenance as a design variable: The paper establishes that a trajectory's value for model training depends fundamentally on the harness under which it was generated—a dimension overlooked by prior co-evolution approaches that treat search trajectories as an undifferentiated replay buffer.

2. Runtime compatibility governs transfer: The finding that checkpoint-only gains largely disappear under foreign runtimes has deep implications for how we think about agent capabilities—they are properties of the model–harness pair, not the policy in isolation.

3. On-policy vs. off-policy in co-evolution: The results connect to the broader RL literature (Tajwar et al., 2024) by showing that a rollout that is on-policy for the weights can become off-distribution once the harness changes its prompts, tools, or observation processing.

Practical Implications

1. Compute efficiency: CoTrace-SFT achieves superior results at 47 GPU-hours/iteration vs. 54 for mixed siblings, using 30–50 trajectories instead of 149–308—a substantial cost saving with better outcomes.

2. Curriculum design: The Refresh mechanism provides a principled way to track the learning frontier, retiring tasks only after they are both solved and incorporated into training data.

3. Deployment guidance: The cross-domain results suggest that when deploying terminal agents, the co-evolved harness should be deployed alongside the checkpoint; checkpoint-only transfer may substantially underperform.

Conclusion

CoTrace introduces a harness-aware data recipe for co-evolving terminal-agent policies and execution harnesses, operating through three coordinated mechanisms:

• Route: Directs failures to harness search, trains policies on verified runtime-matched experience
• Ratchet: Promotes each component independently, cleanly attributing gains
• Refresh: Advances the curriculum frontier

Key takeaways:

1. Compact matched corpora yielded model gains where much larger sibling-pooled corpora did not
2. Online reinforcement remained effective when clean supervised successes were scarce
3. Runtime choice materially shapes checkpoint performance in cross-domain settings
4. Trajectory provenance and model–harness compatibility are central design variables for agent post-training

Future directions (implied): The observed saturation of late-stage curriculum suggests the need for more sophisticated frontier management; the strong interaction between harness and policy transfer suggests that co-evolved harnesses should be treated as part of the deployable artifact, not just a training-time convenience.