# What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents

> Cross-harness credit assignment adds no detectable portability over within-harness grouping when exposure is held fixed, differing by only 0.25 pp on a held-out harness.

- **Source:** [arXiv](https://arxiv.org/abs/2609.04518)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/KSIROM
- **Whiteboard:** https://picx.dev/p/KSIROM/image

## Summary

# What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents

## Summary (Overview)

- This paper isolates the effect of **cross-harness credit assignment** in agent reinforcement learning (RL) by holding exposure to multiple harnesses fixed while varying whether rewards are compared across harnesses within GRPO groups.
- Using four production coding harnesses (Aider, OpenHands, Qwen Code, SWE-agent) on a shared SWE-Gym task pool, the authors replay identical frozen trajectory data from a Qwen3-8B supervised warm start under two credit rules: **Within** (per task–harness groups) and **Cross** (harnesses pooled within tasks).
- On a **held-out minimal harness**, Cross minus Within yields only +0.25 percentage points (pp) [95% CI: −0.48, +1.02] at eight attempts per task, and +0.16 pp [−0.41, +0.72] pooled over three seeds—statistically indistinguishable from zero.
- Cross-harness credit pooling leaves **recoverable harness identity** in the advantage signal (+4.48 pp above shuffled-label null), while Within leaves none; yet both rules produce the same held-out scores and action distributions inside each harness.
- The paper concludes that multi-harness exposure and cross-harness credit assignment are **distinct interventions**, and recommends that multi-harness RL reports state the grouping boundary and test portability under an unseen interface.

## Introduction and Theoretical Foundation

### Background

Coding agents operate through production harnesses that control prompts, tools, observations, context management, retries, and control flow. Recent systems train policies through multiple harnesses simultaneously, but the phrase "multi-harness RL" does not specify how harnesses interact during credit assignment.

### The Core Question

The central question is whether outcomes from different harnesses should be compared directly (pooled into one GRPO group) or normalized separately (per task–harness pair). Two existing production systems embody these choices:

- **HarnessX**: Pools same-task traces across harness versions, making traces compete inside one GRPO group.
- **ClawGym II**: Treats each task–harness pair as the optimization unit, sharing policy and minibatch but keeping reward normalization separate.

Because these systems differ in exposure, rollout generation, training regime, and evaluation, their end-to-end results do not isolate the grouping boundary.

### Theoretical Framework

The paper formalizes the setting with an episode indexed by task $x$, harness $h$, and rollout $i$, with sealed-oracle reward $r_{x,h,i} \in \{0, 1\}$. Both primary arms train on the same frozen manifest of records, tokens, rewards, and loss masks. The advantage is a standardized reward within a group of sampled trajectories, following the GRPO family of baseline-subtracting policy-gradient methods rather than learned-critic RLHF objectives.

The key theoretical insight is that a harness-heterogeneous group changes the composition of the comparison group: under Cross, rewards from contexts that are not exchangeable in the usual same-prompt GRPO sense compete for credit. The paper isolates this boundary experimentally rather than assuming it away.

## Methodology

### Experimental Design

The study uses a **controlled recipe-by-harness matrix** with a single sealed oracle:

1. **Four production harnesses** collect trajectories and rewards from the same SWE-Gym task pool.
2. **Frozen manifest**: Records, tokens, and rewards are fixed across primary arms.
3. **Two primary RL arms** (Within and Cross) replay identical records from the same Qwen3-8B SFT warm start, with the same update budget.
4. **Within**: GRPO groups formed inside each task–harness pair.
5. **Cross**: Harnesses pooled within each task for GRPO grouping.

### Training Details

- **Base model**: Qwen3-8B supervised fine-tuned warm start
- **Update budget**: 81,216 updates per arm
- **Evaluation**: 24,000 sealed-oracle evaluations (500 SWE-bench Verified tasks × 4 source harnesses × 2 attempts × 6 checkpoints)
- **Seeds**: Three training seeds (17, 29, 41) plus one on-policy re-collection arm

### Additional Arms

- **Residualized arm**: Subtracts a per-harness baseline before pooled standardization, with each task's own outcomes excluded from the baseline applied to it.
- **On-policy re-collection**: One Cross arm replays frozen records for the first half, then re-runs training tasks with the half-trained policy to collect the second half.

### Held-Out Evaluation

A **minimal harness** (weak-ReAct interface) is held out of training. This separates source adaptation from portable capability.

## Empirical Validation / Results

### Main Results on Held-Out Harness

At eight attempts per task on the held-out weak-ReAct interface:

| Metric | Cross − Within (pp) | 95% CI |
|--------|---------------------|--------|
| Eight attempts | +0.25 | [−0.48, +1.02] |
| Pooled over 3 seeds (k=4) | +0.16 | [−0.41, +0.72] |

Relative to SFT warm start:
- Within: −0.03 pp [−0.64, +0.59]
- Cross: +0.13 pp [−0.65, +0.92]

### Source Harness Results

- **Cross** is +0.77 pp above SFT on training harnesses, with non-negative point estimates in all four columns.
- **Within** is +0.48 pp above SFT on training harnesses, with largest gain on OpenHands (+1.09 pp).
- Both arms concentrate gains on source configurations—a property of RL on this corpus, not of the grouping rule.

### Harness Identity in Advantages

An out-of-fold classifier recovers the generating harness from:
- **Cross's advantage**: +4.48 pp above shuffled-label baseline
- **Within's advantage**: +0.02 pp above null (as construction requires)

### Seed Variability

Cross minus Within on 499 common tasks at k=4:
- Seed 17: +0.62 pp
- Seed 29: +0.05 pp  
- Seed 41: −0.20 pp

### Action Distribution Analysis

Moving the same checkpoint between harnesses reorders action distributions: search/editing dominate under weak-ReAct, inspection under OpenHands, editing/planning under Qwen Code, search/script execution under SWE-agent. The credit rule does not change the action repertoire.

### Source Harness Heterogeneity

Harness strength varies by model: Seed-Coder-8B scores alike with Qwen3-8B under minimal harness, yet Aider and SWE-agent each cost it ~4 pp, while two further harnesses cannot run. Harness strength is not a model-independent scalar.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Credit assignment resolution**: The paper varies the comparison-group boundary while fixing reward resolution at one binary outcome per episode—isolating a dimension orthogonal to per-step value estimation or hierarchical turn-level objectives.

2. **Contamination measurement**: Cross-harness pooling introduces measurable harness identity into advantages. The residualized arm (removing per-harness baselines) still recovers 90% of harness identity, suggesting the contamination is fundamental to cross-harness comparison rather than a simple offset.

3. **Exposure vs. credit**: Multi-harness exposure and cross-harness credit assignment are distinct interventions. Holding exposure fixed while varying credit alters the learning signal but not downstream portability.

### Practical Implications

1. **Reporting standards**: Multi-harness RL reports should state which harnesses generated experience, whether advantage normalization crosses harness boundaries, source-harness deltas, and deltas under at least one unseen harness.

2. **Portability testing**: The held-out harness measurement separates source adaptation from portable capability—essential for understanding what multi-harness training actually learns.

3. **Action distribution**: The harness sets the action repertoire; the credit rule does not change it. This suggests portability interventions should focus on harness design rather than credit assignment.

## Conclusion

The paper demonstrates that with exposure held fixed, comparing rewards across harnesses adds no detectable portability: Cross and Within are indistinguishable on a held-out harness, with or without on-policy data re-collection. Both rules place their largest gains on the same source harness, and the action distribution inside each harness is set by the harness, not the credit rule.

The authors recommend that multi-harness RL studies report the grouping boundary and test portability under an unseen interface. They also acknowledge limitations: one model family (Qwen3-8B), one budget, four source harnesses, one on-policy re-collection round, and Python-only tasks. The six held-out contrasts were fixed before checkpoints existed but were not registered externally.

**Future directions** include extending to other model families (e.g., Seed-Coder-8B), larger update budgets, additional harnesses, and multi-round on-policy training. The finding that harness strength is model-dependent (factor of 4.3 solve-rate variation) suggests harness evaluation must be model-specific.

---

_Markdown view of https://picx.dev/p/KSIROM, served by PicX — AI-generated visual whiteboard summaries of research papers._