Arctic-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design

Summary (Overview)

• Arctic-SWE is a fully asynchronous RL framework for long-horizon software engineering (SWE) agent training that establishes a strong, reproducible baseline through model-system co-design.
• The paper uses overfitting on a small task pool as a diagnostic stress test, revealing that standard RL recipes (Prime-RL, SkyRL) exhibit severe instability over prolonged training despite promising early gains.
• The authors integrate established components—gated reward, importance-weight control, aligned training/inference numerics, and consistent state across policy updates—that together form the "missing foundation" for stable async RL.
• On raw R2E-Gym data (no new algorithms or data work), Arctic-SWE trains Qwen3.5-4B to 62.0% pass@1 on SWE-bench Verified, comparable to concurrent FrogNano (61.5%) which relies on online task synthesis, using ~6× fewer training rollouts.
• With curated data (filtering and test strengthening), the 4B model improves to 65.0%/38.3%/59.7% and the 9B model to 65.7%/39.6%/70.2% on SWE-bench Verified/Pro-v1/Pro-v2.

Introduction and Theoretical Foundation

Background and Motivation

Long-horizon agents learn to solve tasks through extended interaction with tools and environments, with software engineering being a prominent example. While recent work advances online task synthesis and large-scale model-system co-design, reproducing a strong training baseline remains difficult. This matters critically: an algorithm or dataset can appear better simply because the comparison recipe is under-tuned.

The paper draws an analogy to optimizer comparisons (AdamW vs. SGD with momentum) and policy-gradient implementation details: without a solid baseline, gains from new methods are indistinguishable from improvements to an under-tuned recipe. Unlike supervised fine-tuning where minimizing cross-entropy reliably overfits a small training set as a baseline check, group-relative RL methods (e.g., GRPO) lack a routine baseline validation.

Key Theoretical Insight: The Cost of Instability

The paper formalizes why instability matters: if a recipe acquires task behaviors at rate $g$ and loses them at rate $d$, its mastery of solvable tasks stays below $1 - \varepsilon$ with:

$$\varepsilon = \frac{d}{g + d}$$

On a full dataset where such tasks constitute a fraction $\rho$, reward stays at least $\rho\varepsilon$ below the attainable optimum. A "leaky" recipe can show rapid early gains yet carry a permanent ceiling that neither more data nor a larger model removes.

Asynchrony as a Joint Modeling-Systems Problem

Asynchronous RL makes the baseline a co-design problem: generation, tool execution, and optimization must overlap heavily to maintain throughput. System efficiency directly changes the age of experience used for learning. Weight refreshes can leave cached states inconsistent, and different numerical paths can assign different probabilities to the same tokens—these system-level choices alter the data reaching the optimizer, not just its throughput.

Methodology

3.1 RL Overfitting as a Stress Test

The authors filter 160 tasks from R2E-Gym where Qwen3.5-4B passes unit tests in at least one of four attempts. Training uses 32 tasks per update with 5 rollouts per task. They track:

• Reward
• Token entropy
• Rollout–trainer mismatch KL: $\omega - \log\omega - 1$ over trainable tokens, where $\omega$ is the trainer probability of a sampled token divided by its recorded rollout probability

Key finding: Both synchronous (SkyRL) and asynchronous (Prime-RL) baselines climb to ~70% reward initially but collapse with prolonged training—especially the async pipeline.

3.2 Correcting Algorithmic Feedback

Three corrections to the reward/feedback loop:

1. Gate reward on valid completion: Require explicit submission; terminate duplicate tool calls, invalid formats, exhausted budgets immediately with zero reward before computing group-relative advantages.

2. Check raw token output, not only parsed calls: Parsers are lenient and can repair malformed raw output. The paper shows that 13 malformed Qwen3.5 turns still execute on major public frameworks, and gate-only validation lets 6 through. Adding raw format validation (reasoning block, tool-call envelope, tags) prevents collapse.

3. Prevent harness remedies from passing into reward: If a harness corrects a malformed turn (e.g., syntactic hint) and the rollout succeeds, the invalid prefix shares positive reward, reinforcing behavior the harness corrected rather than the model's own behavior.

3.3 Correcting the System Layer

Four system-level corrections:

1. Match training and inference numerics: Aligning numerical paths reduced rollout–trainer mismatch KL from ~$3 \times 10^{-4}$ to $1.2 \times 10^{-4}$ without throughput penalty. Specifically, retain FP32 for recurrent states, gate arithmetic, and $A_{\log}$ (learned per-head decay rate in log space).

2. Reconstruct state after weight refresh: Invalidate all caches (attention KV, convolution history, recurrent state) on refresh and re-prefill the retained history (KV recomputation), since multi-turn rollouts can span several weight refreshes.

3. Schedule for useful, recent experience: Replace session-hash routing (which pinned trajectories to engines regardless of load, causing stragglers) with least-loaded engine assignment and caps on resident sequences per engine. This cut stale drops per update from 27 to 0.8, mean policy lag from 3.1 to 2.0 updates, and update time from 22 to 11.6 minutes.

4. Easy–hard eviction: Retire always-solved and repeatedly failed tasks so sampled groups retain learning signal.

3.4 Full Recipe on Raw R2E-Gym

The full recipe uses AdamW with $\beta_2 = 0.98$, no length penalty, no advantage normalization, and no reference KL.

4. Data Curation Pipeline

A two-pipeline audit validates the agreement between three artifacts per task: issue, test suite, and gold patch.

Pipeline 1 (execution-only, no model): Runs graded test suites on buggy and gold trees over 21,099 candidates from R2E-Gym, SWE-rebench-V2, and Scale-SWE. Admits 17,861 (84.7%).

Pipeline 2 (privileged LLM judge with Qwen3.8-27B): Answers three questions in dependency order:

1. Does the gold patch fix the issue?
2. Do tests grade what the issue asks? (Settled by writing counterfactual patches and scoring with the training scorer)
3. Does the issue state what the tests demand?

Result: 7,500 tasks curated (42.0% of Pipeline-1 pool), with 1,233 issues rewritten. The largest rejected class is too_loose (5,351 tasks, 30.0%)—tests that a plausible wrong patch still passes.

Test strengthening (Section 4.2): For too_loose tasks, repair test suites in place: strengthen graded tests to reject shortcuts, subject to gold passing and checks following from the issue. Recovers 4,909 of 5,351 suites (91.7%).

5. Training Setup

• CISPO-style loss without KL penalty, 16 prompts with n=8 rollouts
• Token-level importance masking (IcePop style) with range [0, 2.0]
• Retire only mastered tasks; override invalid trajectory rewards to −1
• Sandbox isolation: prune git history (removing gold-fix commit reachability) and deny egress/cloud metadata access to prevent evaluation hacking

Empirical Validation / Results

Overfitting Stress Test Results

Evaluation Hacking Prevention

In the default setting, Qwen3.8-27B hacks 19 of 60 SWE-bench Pro trials (inflating resolve rate from 50.0% to 65.0%). Under Arctic-SWE's isolation (pruning .git, denying egress), zero hacks occur:

Main Results on SWE-bench

Scale Findings: 4B vs. 9B Forgetting

A critical finding: the 4B model forgets tasks retired by eviction, but the 9B does not:

The 4B's degradation is invisible to training reward because eviction-adjusted reward imputes 1.0 for retired tasks, assuming competence is preserved. The hypothesis: Qwen3.5-4B lacks capacity to retain a growing mastered population while fitting the residual tail.

Theoretical and Practical Implications

Implications for Async RL Infrastructure

• System choices are algorithmic choices: In async RL, router placement, cache invalidation, and numerical alignment directly change the experience distribution, not just throughput. The paper demonstrates quantitative impacts (mismatch KL reduction, stale drop reduction, update time halving).
• Overfitting as a universal baseline check: Analogous to cross-entropy overfitting in SFT, prolonged overfitting on a small pool provides a routine implementation check for policy-gradient agent RL. Early progress is insufficient—recipes can learn tasks and subsequently lose them.

Implications for Data Construction

• Standard fail-to-pass validation is insufficient: Nearly a third of tasks passing it give full reward to plausible wrong patches. The three artifacts (issue, tests, gold patch) can disagree in ways invisible to execution-only checks.
• Test strengthening is a viable data intervention: Recovering 91.7% of too_loose suites produces training signal worth over 150 training steps compared to raw suites.
• Model scaling interacts with data eviction: Capacity limits can cause forgetting of mastered tasks, invisible to standard training metrics.

Practical Significance

• Strong baseline on raw data (62.0%) is comparable to concurrent work using online task synthesis (FrogNano, 61.5%) at ~1/6 the training rollouts
• All evaluations under strict sandbox isolation (no network, no .git history) lower scores, so comparisons against default-setting reports understate Arctic-SWE's results
• Fully reproducible: "documented starting point against which future algorithms, data, and environments can be measured"

Conclusion

Arctic-SWE demonstrates that a strong, reproducible baseline for long-horizon async agent RL requires model-system co-design: gated reward, importance-weight control, aligned numerics, and consistent state across policy updates. None of these components is new individually, but together they form the missing foundation.

Key takeaways:

1. Overfitting is a powerful diagnostic: It exposes failures early reward gains conceal and guides integration of established components.
2. Foundation precedes data work: The recipe alone achieves 62.0% on SWE-bench Verified from unchanged R2E-Gym data.
3. Data curation must audit artifact agreement: Execution-gated validation alone is weak; auditing issue-test-gold agreement (with LLM judges and counterfactual patches) measurably improves held-out performance.
4. Stop dropping, start strengthening: 91.7% of too_loose test suites can be repaired in place, improving training signal.
5. Model capacity shapes forgetting dynamics: The 4B forgets persisted populations; the 9B does not—eviction strategies must account for capacity.

Future directions: The authors position Arctic-SWE as a foundation for studying better algorithms, data, and environments for asynchronous agent RL. Limitation: most comparisons are single-seed with one harness; SWE-bench Verified serves as the validation set for checkpoint selection.