Pretraining DataIssue 14Nov 7 – 14, 2026

The value of data curation is explicitly priced: the distinction between advancement and acceleration

Highlights

The clearest signal this issue is that "the value of data curation is explicitly priced"—gains are decomposed into two separable effects: advancement (endpoint improvement) and acceleration (faster convergence). Advancement or Acceleration? provides a unified theoretical framework (two-teacher perceptron + replica method): when baseline training and target are aligned, any curation strategy cannot improve the endpoint, only accelerate; advancement requires a correctable train–target gap, and based on this, a Δ-test diagnostic that can be computed in advance from a single uncurated baseline run is provided (Advancement or Acceleration? How Data Curation Improves LLM Training). This unifies previously contradictory findings such as "unified mixtures are competitive" and "random selection is competitive" into a gap-dependent picture: in the no-gap setting, optimizing mixture only accelerates without improving the endpoint; under exogenous/endogenous gaps, curation gains persist with budget. For mixture decision-makers, "run Δ-test first, then decide whether to invest in curation compute" is a directly usable judgment tool.

Parallel negative evidence falls on post-training data strategies. DataFlex-RL, with 12 matched seeds, 13 configurations, and 591 controlled runs, proves that three data strategy types—selection, reweighting, and adaptive mixing—under a unified GRPO recipe cannot yield reproducible gains (all paired 95% confidence intervals contain zero), and quantifies the impact of evaluation summaries on method rankings—math-heavy summaries and 12-benchmark summaries are negatively correlated in ranking (DataFlex-RL: An Evaluation Platform for RLVR Data Policies). This mirrors last issue's "evaluation criteria shift toward post-training ceilings": last issue provided a positive path for CPT with failure signals, while this issue offers a strong negative baseline that any study claiming RLVR data policy gains must compare against.

New levers for mid-training data sources and mixture ratios have emerged. SPT introduces public multi-file skill packages (human-written reusable tool semantics and workflows) as a new mid-training corpus source, proposes the Reference Insert citation-aware serialization strategy, and identifies a sweet spot of 20–50% skill ratio (30% optimal; both pure-skill and pure-general endpoints are worse than intermediate mixtures) (SPT: Skills as Pre-Training Data for Agentic Language Models); Beyond Data Schedule, under fixed single exposure, proves that loss reweighting can serve as a control variable for mixture ratios independently of data scheduling (stage trajectories), with a single checkpoint outperforming both stage trajectories (Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting). Time-Incremental CPT treats "temporal increment" as an explicit research object, showing that CPT gains correlate with the pre-training saturation degree (tokens-per-parameter) of the trained checkpoint rather than model size, and that "data quality dominates quantity" (6B curated outperforms 40B broad) (Time-Incremental Continued Pretraining of LLMs). Data recipes in the annealing/continued pre-training phase are beginning to show richer design axes than just "mixture ratios."

Open Questions

  1. Can Δ-test determine in advance at real scale whether "curation only accelerates but does not improve the endpoint"? Can this unified framework incorporate the per-domain optimal bands from Issue 8's Everything in Moderation and the pass@1/pass@K gaps from Issue 11's Learning from the Gap into the same gap-dependent picture for prediction, thereby making "whether it is worth investing curation compute" a computable decision?
  2. To what extent does DataFlex-RL's no-gain conclusion hold—would data strategies beyond σ-control, RL algorithms other than GRPO, and base-loss criteria allow some data strategies to produce reproducible gains again? How much impact does its evaluation sensitivity analysis (summary selection changing rankings) have on the reported scores of existing RLVR data strategy papers?
  3. What are the gain boundaries of SPT's skill packages as a mid-training corpus source? Does the Reference Insert citation serialization strategy hold under larger token budgets and other agentic tasks, and how can its natural software engineering data and agentic traces from Issue 11's Priming Base Models be jointly optimized in a unified recipe with the ratio?
  4. Does Beyond Data Schedule's loss reweighting, as an independent control axis, remain orthogonal under the constraint of "data scheduling coupled with LR scheduling" proposed in Issue 13? Does its joint optimum with LR scheduling yield new ranking reversals, or does it share the same optimal interval as data scheduling?

Papers in this issue

  1. Advancement or Acceleration? How Data Curation Improves LLM Training

    Data curation improves LLM endpoints only when a correctable train-target gap exists; otherwise it merely accelerates training, as diagnosed by the lightweight Delta-test.

    Editor's note

    For the first time, decomposes data curation gains into two separable effects—advancement (endpoint improvement) and acceleration (faster convergence)—and proves that advancement requires a correctable train–target gap, deriving the Δ-test diagnostic computable in advance from a single uncurated baseline run. Relative to Issue 8's Everything in Moderation and Issue 11's Learning from the Gap, which focus on coverage and pass@k gaps respectively, its increment is unifying contradictory findings like "unified mixtures are competitive" and "random selection is competitive" into a gap-dependent picture, with controlled no-gap and with-gap comparisons on 160M/1.1B. For mixture decision-makers, "run Δ-test first then decide whether to invest curation compute" is a directly usable judgment tool.

  2. DataFlex-RL: An Evaluation Platform for RLVR Data Policies

    Under controlled GRPO training, none of 8 selection/reweighting or 3 adaptive-mixture data policies reliably beat uniform sampling, though GRPO itself boosts accuracy by 7.76 points.

    Editor's note

    For the first time, under the RLVR post-training setting, with 12 matched seeds and 13 configurations in systematic comparison, proves that three data strategy types—selection, reweighting, and adaptive mixing—under a unified GRPO recipe cannot yield reproducible gains (all paired 95% confidence intervals contain zero), and quantifies the impact of evaluation summaries on method rankings (math-heavy summaries and 12-benchmark summaries are negatively correlated at ρ=-0.33). Relative to Issue 9's What pass@k Cannot Measure, which focuses on post-training evaluation blind spots, its increment advances data strategy criteria from single benchmark scores to paired confidence intervals and evaluation sensitivity analysis. It is a strong negative baseline that any study claiming RLVR data strategy gains must compare against.

  3. SPT: Skills as Pre-Training Data for Agentic Language Models

    Skill pre-training on public multi-file skill packages improves agentic performance by up to 25 points over direct SFT while preserving general capabilities.

    Editor's note

    For the first time, introduces public multi-file skill packages (human-written reusable tool semantics and workflows) as a mid-training corpus source, and proposes the Reference Insert citation-aware serialization strategy, outperforming general corpora and agent trajectory corpora across six backbone×SFT settings from 1.6B–7B. Relative to Issue 11's Priming Base Models for agentic mid-training recipes, its increment shifts data sources from trajectories/synthetic calls to human skill packages that do not require environment execution, and provides ratio evidence for a 20–50% skill ratio sweet spot (30% optimal; both endpoints worse than intermediate mixtures). Limitations include a single data source (ClawHub) and criteria based mainly on agentic benchmarks, but as a new data source and ratio sweet spot, it is worth reading.

  4. Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting

    Time-incremental continued pretraining on web-scale post-cutoff data reliably teaches new facts across LLM families without catastrophic forgetting, though acquisition tracks pretraining saturation.

    Editor's note

    For the first time, studies knowledge acquisition and catastrophic forgetting in the realistic setting of time-incremental CPT, explicitly challenging the "disjoint data stream" assumption of continual learning—between web-scale snapshots, only 30–40% of URLs are new, making strict disjointness operationally infeasible. Across three families and six models (1B–8B), provides controlled evidence: CPT gains correlate with the pre-training saturation degree (tokens-per-parameter) of the trained checkpoint rather than model size, and "data quality dominates quantity" (6B curated outperforms 40B broad, 7 rounds yield no additional gains). Relative to Issue 13's Data Schedules, which couples data/LR scheduling, its increment treats "temporal increment" as an explicit research object and uses Daily Oracle temporal knowledge as criteria, adding new constraints to data recipes in the annealing/continued pre-training phase.

  5. Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting

    Separating domain exposure from loss reweighting lets pretraining expand the capability frontier, with Lare's adaptive inverse-square-root weights boosting all benchmarks by up to +17 points.

    Editor's note

    For the first time, under fixed data exposure (each sample appears only once), proves that loss reweighting can serve as a control variable for mixture ratios independently of data scheduling (stage trajectories), with a single checkpoint outperforming both stage trajectories, revealing an independent control dimension that mixture ratios alone cannot capture. Relative to Issue 13's Data Schedules, which couples data scheduling with LR scheduling, its increment treats loss reweighting as an independent lever orthogonal to scheduling, giving +7.84 pp average gains at 1.5B/240B scale. Limitations include only a summary without public recipes or ablation details, and criteria based on downstream benchmarks rather than post-training ceilings, but as a new axis for ratio control, it offers methodological increments.