Summary (Overview)

• Open midtraining recipe: This paper presents a fully documented, open midtraining stage for agentic coding models, positioned between pretraining and SFT (supervised fine-tuning), developed and validated on two NVIDIA Nemotron models.
• Key components of the recipe: The blend consists primarily of natural software-engineering data (Issue-PR pairs, dependency-ordered repositories, diff data, build/CI logs) mixed with a bounded 10-15% share of agentic traces, with tool responses masked from the loss.
• Significant performance gains: On N-3.5 Lightning, 500B midtraining tokens improve SWE-bench Verified from 60.7 to 64.5, SWE-bench Multilingual from 55.8 to 62.8, and SWE-bench Pro from 32.0 to 35.5 after identical full SFT.
• Scaling to larger models: On N-3.5 Super, 280B midtraining tokens improve DeepSWE (a hard long-horizon benchmark) from 11.7 to 15.7 and SWE-bench Verified from 72.9 to 74.3.
• Robustness and practical insights: Midtraining improves performance across all agent harnesses tested and serves as an effective intermediate long-context stage (256K context), while base-model proxy metrics (LoCoDiff, BPB) did not reliably predict post-training results.

Introduction and Theoretical Foundation

The paper addresses a critical gap in modern LLM training pipelines. Frontier coding models (Qwen3-Coder-Next, GLM-5, Kimi-Dev, Composer 2) now include a "midtraining" stage between pretraining and post-training, but technical reports rarely detail how this stage is built. The motivation stems from several structural mismatches:

1. Pretraining context limitation: Base models are pretrained at short sequence lengths (typically 4K–8K tokens), while software-engineering data (whole repositories, commit histories, PRs) often runs to tens or hundreds of thousands of tokens.

2. SFT data syntheticity and RL trainability: SFT data is mostly synthetic chat data, and recent work documents that very long SFT stages can reduce the model's ability to benefit from RL (Liu et al., 2026b; Kang et al., 2025; Zhang et al., 2026).

3. Absence of agent interactions: Pretraining corpora contain almost no agent interactions, and adding agent traces to pretraining is problematic since pretraining applies no loss masking.

Midtraining fills this gap by training at 256K context for hundreds of billions of tokens. The recipe is theoretically grounded in the idea that much of the "agentic learning" can move out of SFT, allowing SFT to stay short and preserve RL trainability. The paper builds on prior work like Kimi-Dev (which uses ~150B tokens of diff patches, commit sequences, and synthetic agent interactions) but provides the ablation-level detail missing from existing technical reports.

Methodology

Data Sources and Blend

The midtraining blend (Figure 1, Table 1) is designed for diversity:

Key data construction details:

• Issue–PR: 12.1 million merged PRs from public GitHub repositories with LLM-judge validation for issue-PR matching.
• Repositories: Only repositories where ≥20% of code files import another file; serialized with dependency ordering (import-graph-based), README first.
• Diff data: Multi-file edits from merged PRs and code-evolution documents from full git histories.
• Agentic traces: ~39B tokens, 1M+ trajectories, 30M tool calls from 10+ scaffolds (OpenHands, SWE-agent, Codex, Claude Code, Terminus, etc.).

Training Setup

• Context length: 256K tokens
• Learning rate: 5e-5 (N-3.5 Lightning), 1e-5 (N-3.5 Super)
• Replay share: 20% from pretraining data
• Tool responses masked from loss (only assistant turns trained)
• 500B tokens max midtraining on N-3.5 Lightning, 280B on N-3.5 Super

Evaluation Protocol

The paper emphasizes a holistic evaluation:

• Short SFT (33B tokens) for quick signal, full SFT (100B tokens) for main claims
• Benchmarks: SWE-bench Verified, Multilingual, Pro; DeepSWE; Terminal-Bench; BrowseComp
• Base-model post-trainability metrics (LoCoDiff, RepoBench, bits-per-byte) tracked but found unreliable as proxies

Empirical Validation / Results

Scaling on N-3.5 Lightning (500B tokens after full SFT)

Agentic Traces Essential

The blend validation showed that natural data alone gives much smaller gains. The no-traces model gained less than 1 point on SWE-bench Verified after full SFT, while the full blend improved by 3.0, 2.5, and 6.6 points on SWE-bench Verified, Multilingual, and Terminal-Bench respectively.

Scaling to N-3.5 Super

The clearest gain is on DeepSWE (hardest benchmark), improved from 11.7 to 15.7 after 280B tokens.

Harness Robustness

Midtraining improves pass@1 on every harness tested. On N-3.5 Super after short SFT, the average gain is 8.6 points on SWE-bench Verified and 6.8 on Terminal-Bench 2.1. On N-3.5 Lightning, gains range from +3.8 to +19.0 points across six harnesses.

Key Ablations

• Tool-response masking: Masking tool responses (vs. training on them) improves SWE-bench Verified from 27.5 to 37.8 pass@1 (10-point gain).
• Learning rate: Higher LRs (3e-5, 5e-5) hurt general capabilities, especially MATH-500 (79.7 → 61.7 at 5e-5).
• Replay: 20% replay maintains general capabilities (no regression >1.2 points on any benchmark group).
• Long-context effect: Midtraining at 256K brings 8K-pretrained model within ~5 points of long-context model on RULER up to 128K.

Theoretical and Practical Implications

The paper makes several significant contributions to the field:

1. Open recipe: Provides the most complete midtraining recipe to date, including data construction, filtering, weighting, and ablations—filling a documentation gap in frontier model technical reports.

2. Data diversity principle: Shows that a diverse blend of natural software-engineering data (repositories, PRs, diffs, logs) combined with a bounded share of agentic traces is the optimal configuration. Natural data provides long-context and file-state modeling capabilities; traces provide interaction patterns.

3. Evaluation methodology: Challenges the utility of base-model proxy metrics (LoCoDiff, BPB, RepoBench) for predicting post-training results, advocating instead for short fixed SFT runs as the primary evaluation signal.

4. Practical training insights: Established best practices including tool-response masking, moderate learning rates with pretraining replay, and positioning midtraining before the final long-context stage.

5. Scale robustness: Demonstrates the recipe transfers across model sizes (30B and 120B) and midtraining token budgets (up to 500B), with the largest gains on the hardest benchmarks.

Conclusion

The paper presents an open, fully documented midtraining recipe for agentic coding that delivers clear, consistent gains across models, benchmarks, and agent harnesses. Key takeaways:

• Agentic traces are essential: They drive most of the gains after SFT, though they should be bounded to 10-15% of the blend to preserve RL trainability.
• Short SFT is the best evaluation signal: Base-model proxy metrics did not reliably track post-training results.
• Midtraining doubles as a long-context stage: Training at 256K naturally extends the context, and gains persist through the final long-context phase.

Limitations: Benchmarks are noisy (single-run comparisons); no comparison against spending the same tokens on prolonged pretraining or SFT; recipe tested only on Nemotron models and mainly for agentic coding; RL results are limited (short single-seed run).

Future directions: Scaling trace data with more diversity (already expanded corpus to ~90B tokens), extending agentic data to more capabilities, and moving more post-training data (reasoning, chat) into midtraining.