Pretraining DataIssue 11Oct 17 – 24, 2026

Agentic mid-training recipe first disclosed: post-SFT performance as criterion

Highlights

This issue's clearest signal is the progression of mid-training from 'corpus construction' to 'complete recipe + post-training ceiling criterion'. Priming Base Models for Agentic Post-Training for the first time discloses an open recipe for agentic mid-training: the corpus is dominated by natural software engineering data (packed repositories, pull requests, commit histories, build logs), agentic traces account for only 10-15% with tool-response masking, supplemented by 20% pretraining replay and 256K context, achieving significant improvements on SWE-bench/DeepSWE after identical SFT on Nemotron 3.5 Lightning (500B tokens) and Super (280B tokens) (Priming Base Models for Agentic Post-Training). Its evaluation protocol itself is noteworthy—the authors explicitly report that base-model proxy metrics do not correlate with post-training results, so they instead use short fixed SFT as the criterion, directly echoing this direction's shift toward 'post-training ceiling'.

In data-constrained repetition, Selecting Repetition Counts for the first time provides evidence of ranking inversion of optimal repetition count with model scale: at 50M, r=16 outperforms r=8, but at 80M and 520M, the order reverses, and it proposes candidate-set pruning based on small-model loss curves as an alternative paradigm to exact point prediction (Selecting Repetition Counts Across Model Scales). In MoE multilingual direction, Scaling Multilingual Mixture-of-Experts jointly models language coverage K and expert count E, showing that 32 experts reduce the loss penalty from 12→200 languages by 83%, and broader coverage extends the computational advantage regime of expert-heavy configurations (Scaling Multilingual Mixture-of-Experts).

Post-training data selection this issue features two new signal axes. GapFT uses the 'gap between Pass@K and Pass@1' itself as the selection criterion, selecting only validation responses that fail on single samples but are recoverable via search (Learning from the Gap Between Pass@K and Pass@1); Pick the Mixture advances data selection from sample-level to the combination level of 'domain mixtures', explicitly modeling complementary and substitutive interactions between samples (Pick the Mixture, Not the Samples). In document extraction quality axis, Xiaomi-OCR-0 provides evidence that the transfer direction of understanding supervision to parsing reverses with parsing capability maturity (Xiaomi-OCR-0 Technical Report).

Community and Dynamics

This issue presents direct evidence that changes mixture ratio decisions regarding data-constrained repetition and filtering trade-offs. Datasets, documents, and repetitions challenges the default assumption that 'more unique data always beats repetition' at 1B–12.6B scale: with regularization (weight decay adjusted upward with repetition rounds), aggressively filtered data repeated up to 10 rounds can outperform a 10-times larger single-pass superset, and document-level repetition outperforms whole-dataset multi-epoch—this corroborates the conclusion from issue 10, Scaling Repeated-Data Pretraining, that regularization strength needs to be adjusted upward under data constraints.

In proxy→target extrapolation, Data mixture optimization: a multi-fidelity multi-scale Bayesian framework for the first time models data mixture optimization as multi-fidelity multi-scale Bayesian optimization, explicitly quantifying extrapolation uncertainty across 20M–1B scales and mixture dimensions, rather than assuming deterministic scaling laws are reliable; TANDEM uses twin networks to formalize mixture optimization as a bilevel problem, explicitly upweighting domains that benefit most from additional data, and notes that data-constrained and SFT scenarios are more worth optimizing than traditional pretraining. In mid-training evaluation protocols, Evaluating mid-training argues that the quality of mid-training recipes should be judged by downstream post-training metrics (pass@k/RL trainability), and warns that micro-anneal comparisons often fall within run-to-run noise. In document extraction quality axis, OmniParseBench provides an auditable OCR benchmark across 92 languages, serving as an open yardstick for parser quality in this data quality axis.

Open Questions

  1. Does the ranking inversion from Selecting Repetition Counts (at 50M, r=16 outperforms r=8; at 520M, order reverses) hold at ≥1B scale with post-training ceiling as criterion? Can candidate-set pruning as a substitute for exact point prediction transfer across corpora and mixtures?
  2. To what extent can Priming Base Models' agentic mid-training recipe (traces 10-15%, replay 20%, tool-response masking) transfer to non-Nemotron, non-code domains? Does its conclusion that 'base proxy metrics do not correlate with post-training' hold more broadly?
  3. Does GapFT's 'search-reachable failures outperform repeating solved problems' hold in RL trainability? Can its selection signal under SFT/RFT serve as a prior criterion for RL data mixtures?
  4. Does Xiaomi-OCR-0's 'parse first, understand later' curriculum constraint hold at ≥1B scale and on more document tasks? Does the negative→positive transfer reversal of understanding supervision to parsing depend on specific data mixtures and training order?

Papers in this issue

  1. Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe

    An open midtraining recipe blending natural software-engineering data with 10-15% masked agentic traces boosts agentic coding performance by up to 6 points on SWE-bench and 4 on DeepSWE.

    Editor's note

    First disclosure of a complete open recipe for agentic mid-training, advancing mid-training from 'corpus construction' to 'complete recipe + post-training ceiling criterion'. Achieves significant improvements on SWE-bench/DeepSWE after identical SFT on Nemotron 3.5 Lightning (500B tokens) and Super (280B tokens), explicitly distinguishing the ratio of natural software engineering data to agentic traces (traces only 10-15%, tool-response masking), and systematically ablating replay ratio and learning rate. Relative to issue 6's MiST synthetic mid-training corpus design, its increment is a reproducible complete recipe, and the authors explicitly report that base-model proxy metrics do not correlate with post-training results, instead using short fixed SFT as criterion, directly responding to this direction's shift toward 'post-training ceiling'. Limitation: data and code are not disclosed, RL evidence is thin, but as an open recipe its methodological contribution is worth reading.

  2. Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining

    Optimal repetition count for scarce data decreases with model size, so smaller-model sweeps must prune candidates rather than pick a single count for larger-scale training.

    Editor's note

    First to systematically show that under fixed target corpus mixed with generic data, the optimal repetition count r undergoes ranking inversion with model scale (at 50M, r=16 outperforms r=8; at 80M and 520M, order reverses), and proposes candidate-set pruning based on small-model loss curves as an alternative to exact point prediction. Relative to issue 9's 'What Is a Repeated Token Worth?' single-round geometric framework, its increment is advancing 'repetition value varies with scale' into explicit scale-dependent ranking inversion and an operational pruning procedure; relative to issue 3's 'Scaling Laws for Mixture Pretraining Under Data Constraints' r-h coupling law, its increment is validating the inversion on independent, complete configurations and providing candidate retention. For mixture decision-makers relying on small-model scans to determine repetition rates, this is key evidence that changes decisions.

  3. Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation

    Scaling experts from 1 to 32 cuts the multilingual loss penalty of expanding language coverage from 12 to 200 languages by 83%, with broader coverage favoring expert-heavy models.

    Editor's note

    First to jointly model MoE laws with multilingual coverage laws, explicitly characterizing the interaction between expert count E and language coverage K, proving that 32 experts reduce the loss penalty from 12→200 languages by 83%, and revealing that broader coverage extends the computational advantage regime of expert-heavy configurations. Relative to issue 10's 'Scaling Laws for Multimodal Data Mixtures' unified three-modality mixture scaling law, its increment is making language coverage K an interaction design axis with expert count E, providing a repetition-aware joint law. Grid of 26,300 loss observations and 14 multilingual benchmarks align with preferences; limitation: max 1.15B, criterion still base loss.

  4. Learning from the Gap Between Pass@K and Pass@1

    GapFT selects verified responses from problems the source policy fails at first but solves via search, outperforming uniform rejection sampling on logic benchmarks with far fewer training examples.

    Editor's note

    First to use the 'gap between Pass@K and Pass@1' itself as explicit criterion for post-training data selection: classifies problems into three states based on source-policy single-sample success/failure, and under fixed budget selects only validation responses that fail on single samples but are recoverable via search. Relative to issue 9's 'What pass@k Cannot Measure' focusing on pass@k failure as diversity proxy, its increment is converting the pass@k vs pass@1 gap into an operational data allocation principle, and proving search reachability itself is a selection signal (gold-label failures are actually worse). 7B scale, three model families, matched budgets and exact decomposition align with preferences; limitation: covers only SFT/RFT, gain on Qwen insignificant.

  5. Pick the Mixture, Not the Samples: Combinatorial Data Selection for LLMs

    Combinatorial data selection that optimizes domain mixture before ranking samples outperforms per-sample scoring by 20-54%, making domain allocation the key lever in LLM fine-tuning.

    Editor's note

    First to formalize data selection as a cross-domain combinatorial allocation problem, decoupling sample scoring from subset selection, explicitly modeling complementary and substitutive interactions between samples, proposing a two-stage Mixture-then-Select algorithm (first determine per-domain sample counts, then rank within domains). Relative to sample-level scoring or domain mixture optimization like RegMix/DoReMi/LESS, its increment is making the 'domain mixture' itself the object of combinatorial selection, providing spectral norm bounds and data pricing applications. Controlled ablations at 8B scale, multiple models and pool sizes show domain mixture is more critical than sample scoring; limitation: criterion mainly held-out loss, not RL post-training ceiling.

  6. Xiaomi-OCR-0 Technical Report

    Xiaomi-OCR-0, a 0.8B VLM, achieves state-of-the-art document parsing and strong OCR understanding by showing understanding supervision shifts from harmful to beneficial as parsing matures.

    Editor's note

    First to systematically characterize how 'parsing capability maturity' determines the transfer direction of understanding supervision to parsing: before parsing matures, adding understanding supervision causes negative transfer (-0.330); after parsing foundation established, the same supervision turns positive (+0.639), providing three-stage controlled evidence. Relative to issue 6's WeVisDoc residual diagnostics and issue 4's 'Closing Cost-Quality Gap' difficulty-aware curation, its increment is making 'training order of parsing and understanding tasks' an explicit research object, providing a 'parse first, understand later' curriculum constraint for multi-task mid-training. Limitation: 0.8B scale does not meet ≥1B preference threshold, and overall it leans toward a model technical report, but its data-ordering conclusions add value to the document extraction quality axis.