Cross-Benchmark Transfer from RL on Agentic Coding Tasks

Summary (Overview)

• Single RL run, broad transfer: One epoch of reinforcement learning (GSPO) on just 1,700 expert-built coding tasks improved pass@1 on all six external benchmarks evaluated, spanning three different agent harnesses, with gains of +4.7 to +20.0 percentage points.
• Out-of-distribution generalization: Improvements held on four benchmarks released after training data collection (p = 0.004), under two harnesses never used in training (Terminus 2, Claude Code), and on multi-hour tasks with no horizon counterpart in training.
• Efficiency gains: The trained model also solved tasks in fewer agent steps—median trajectories fell 24–35% on DeepSWE and Terminal-Bench 3.
• Identified failure modes: Paired trajectory analysis revealed the base model's failures cluster into four "last-mile" modes (lost requirements, narrow testing, silent regressions, weak ground truth), all of which the trained model avoids.
• What transfers is behavior, not knowledge: The evidence suggests RL taught general engineering completion habits, not domain knowledge the base model already possessed.

Introduction and Theoretical Foundation

Background and Motivation

Coding agents frequently fail in the "last mile" of a task: they build most of a feature but drop a subtle requirement, test only cases their own implementation handles, break existing behavior, or validate against unchecked assumptions. The authors pose two questions:

1. What does RL teach? Domain knowledge or general task-completion behavior?
2. Does improvement transfer? Across benchmarks, harnesses, and horizons?

Standard RLVR (reinforcement learning with verifiable rewards) evidence typically shows gains only on benchmarks whose format matches training data. This paper investigates whether a small, expert-built training set can induce transferable engineering behavior.

Theoretical Foundations

The work builds on several established threads:

• RLVR for coding agents, established in mathematics and short-form code (DeepSeek-R1, GRPO), extended to long-horizon agentic software engineering (SWE-RL, SWE-Gym, DeepSWE-Preview).
• Partial-credit rewards proportional to fraction of tests passed, with the novel addition of a hard regression gate.
• Transfer from post-training: Prior work shows RL generalizes better than SFT to unseen variants, and can transfer across task distributions (e.g., office-work training improving SWE-Bench Pro).
• GSPO (Group Sequence Policy Optimization) for Mixture-of-Experts models, addressing token-level ratio fluctuations in GRPO.

Methodology

Training Data (1,700 Tasks)

Two task families from Surge AI's expert-built collections:

• Repository tasks (1,000): Real repos at pinned commits with change requests. Hidden fail-to-pass tests grade the requested behavior; pass-to-pass tests guard existing behavior. Specifications enumerate concrete, individually gradable requirements.
• Terminal tasks (700): Containerized environments with deliverables graded by expert-written hidden verifiers.

Four task properties shape the training signal:

• (P1) Gradable requirements: Each requirement maps to checks → partial credit possible.
• (P2) Hidden graders: Agent sees spec but never the tests → must decide its own checks.
• (P3) Protected behavior: Pass-to-pass tests enforce "what must not change"; a single regression zeroes the reward.
• (P4) No oracle: No reference solution provided → evidence of correctness must be self-produced.

All training tasks were checked against all six benchmarks—no overlap or derivation.

Reward Function

Let $P_i \in \{0,1\}$ ($i \leq N_P$) be pass-to-pass test outcomes and $F_j \in [0,1]$ ($j \leq N_F$) be fail-to-pass/verifier outcomes. The reward is:

$$R = \mathbf{1}\left[\sum_{i=1}^{N_P} P_i = N_P\right] \cdot \frac{1}{N_F} \sum_{j=1}^{N_F} F_j. \tag{1}$$

• Partial credit: 3 of 12 checks passed scores 0.25.
• Hard gate: 12/12 new checks but 1/100 pass-to-pass failure scores 0.

Why partial credit matters: Under binary rewards with $G=8$ rollouts per group, a group is silent (no gradient signal) with probability $q^G + (1-q)^G$—43% at $q=0.1$, 66% at $q=0.05$. Partial credit keeps ranking signal even when no rollout fully succeeds, while the gate prevents exploiting a trade between new and existing behavior.

Policy Optimization

GSPO with rank-32 LoRA adapter ($\alpha=32$) on attention and MLP projections. For prompt $x$ with rollouts $y_1,\ldots,y_G$ and rewards $R_1,\ldots,R_G$:

$$s_i(\theta) = \left(\frac{\pi_\theta(y_i \mid x)}{\pi_{\theta_{\text{old}}}(y_i \mid x)}\right)^{1/|y_i|}, \qquad \hat{A}_i = \frac{R_i - \text{mean}_j R_j}{\text{std}_j R_j}, \tag{2}$$

maximizing:

$$\mathcal{J}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G} \min\left(s_i(\theta)\hat{A}_i,\ \text{clip}(s_i(\theta), 1-\varepsilon_{\text{low}}, 1+\varepsilon_{\text{high}})\hat{A}_i\right)\right]. \tag{3}$$

Key details: No KL penalty, no entropy bonus, no supervised warm-up. Truncated importance sampling corrects rollout/trainer numerical mismatch. 108 rollout steps, 216 optimizer updates total. Training on 64 NVIDIA H200 GPUs using open-source stack (Miles, SGLang, Megatron-LM, FlashAttention).

Evaluation Protocol

Six benchmarks, one harness each (fixed for both models):

*Released after training data collection. Terminal-Bench 4 revises Terminal-Bench 3 (8 removed, 19 fixed), so they form one family.

Empirical Validation / Results

Main Results (Table 2)

†Publicly reported baseline. ‡8 of 74 runs failed for harness reasons (not scored); 8 of remaining 66 pass.

Statistical Significance

• Pooled across 5 independent task sets (TB 3/4 counted once via TB 4): Cochran–Mantel–Haenszel test rejects equal pass rates at p < 0.001.
• Post-release benchmarks only (3 independent sets): improvement significant at p = 0.004.
• Individual significance (Boschloo's exact test, no multiple-comparison correction): TB 2.1, 3, 4, and DeepSWE vs. in-house base.

Efficiency Gains

• DeepSWE: median agent steps fell from 150 → 98 (−35%).
• Terminal-Bench 3: median steps fell from 102 → 78 (−24%).

Near-Miss Analysis (DeepSWE Base Failures)

Of 83 failed in-house base runs:

• 84% preserved every pass-to-pass test.
• 59% passed ≥80% of target tests.
• Median failed run passed 86% of target tests.

Paired Trajectory Analysis: Four Failure Modes

Illustrative examples:

• FastAPI task: Base passed 121/137 tests but defined methods at module level instead of on middleware object; trained model made them instance methods, passed all 137.
• Spectral solver: Base assumed 1-D problem and validated against data from same assumption; trained model used method of manufactured solutions with an analytic 2-D solution.
• Zstandard decompressor: Trained model wrote a compressor first to generate test inputs for round-trip verification.

Theoretical and Practical Implications

What Transfers Is Behavior, Not Knowledge

The base model already possessed the domain knowledge required (it was a frontier 1T-parameter model). The training did not teach new facts about FastAPI, PDE solvers, or compression—it taught a completion discipline:

1. Treat the specification as a contract to be met in full, not approximately.
2. Test what was asked, not what was built.
3. Track what must remain unchanged alongside what must change.
4. Produce independent evidence of correctness when no oracle exists.

Implications for RL Training Design

• Small, expert-built datasets can suffice: 1,700 tasks with rich, enumerable requirements outperformed the implicit assumption that scale of synthetic data is necessary.
• Partial credit + regression gate is a powerful reward shape: It maintains gradient signal on hard tasks where all rollouts fail, while eliminating the incentive to trade new behavior for regressions.
• Task properties determine learned behaviors: The four training properties (P1–P4) map directly onto the four behavioral changes observed, suggesting deliberate task design can target specific failure modes.

Practical Significance

• Gains on post-release benchmarks (TB 3, TB 4, SWE-Marathon) demonstrate robustness against benchmark contamination concerns.
• Improvement under unseen harnesses (Terminus 2, Claude Code) suggests the learned behavior is harness-agnostic.
• Multi-hour horizon transfer (SWE-Marathon: 5% → 25%) indicates the training generalizes beyond its own temporal scope.

Conclusion

One epoch of GSPO RL on 1,700 expert-built coding tasks, with a rank-32 LoRA adapter and a partial-credit reward gated on regressions, improved Kimi K2.7 Code on all six external benchmarks evaluated (+4.7 to +20.0 pp). The gains held on benchmarks released after training data collection, under harnesses absent from training, and on tasks with horizons far beyond anything in the training set.

The paired trajectory analysis shows the base model's failures were predominantly near-misses—most of the feature built, one or two requirements dropped, existing behavior mostly preserved. The trained model avoids these failures by implementing the full specification, testing requirements rather than implementations, protecting existing behavior, and constructing independent ground truth.

The central conclusion: What transferred was not domain knowledge—which the base model already had—but a way of finishing work. The training tasks and reward function favored specific engineering habits, and those habits generalized across benchmarks, harnesses, and task horizons.

Future directions suggested by this work:

• Exploring whether even smaller, more targeted task sets can induce specific behavioral changes.
• Investigating which task properties (P1–P4) are necessary vs. sufficient for transfer.
• Extending the approach to other agentic domains beyond coding (e.g., data analysis, research tasks).