Pretraining DataIssue 15Nov 14 – 21, 2026

Repetition Mismatch: Why Small-Scale Mixture Experiments Don't Transfer

Highlights of This Issue

The clearest signal this issue is that "why small-scale mixture experiments don't transfer" finally has an actionable answer. Repetition Mismatch separates repetition mismatch from the mixture ratio itself: because high-quality corpora are small in scale, their repetition rate changes as training budget grows, placing proxy and target runs in fundamentally different repetition regimes—even with the same mixture ratio, the optimal ratio drifts as budget grows. The authors propose a subsampling procedure matched to the target repetition rate, which locks in a near-optimal mixture with a single small-scale run at 1/16 token budget (mixture error on 1.17B drops from 0.850 to 0.100, using only about 6% of the target budget) (Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them). This advances the repetition/mixture scaling work from Issue 3's Scaling Laws for Mixture Pretraining Under Data Constraints and Issue 11's Selecting Repetition Counts into a diagnosable confounding mechanism, a must-check fix for any study using small-scale scans to set mixture ratios.

Reasoning data mixture ratios are beginning to yield analyzable theory. Understanding the Role of Reasoning Data is the first to explicitly model the "atomic facts vs. reasoning traces" ratio as a mixture law: the optimal atomic data weight increases with reasoning chain length and decreases with reasoning coverage (held-out ratio), yielding a closed-form solution under linear attention, validated on FineMath–MathLake mixtures with a 10–20% reasoning data sweet spot and "the earlier introduced, the better" (Understanding the Role of Reasoning Data in Pretraining). Parallel mechanistic studies treat corpus statistics themselves as causal variables: learning on entity comparison tasks is decomposed into three phases—frequency → position → correct algorithm—where entity frequency distribution (Zipfian vs. uniform) and synthetic data density causally determine the presence of each phase, with 0.1% synthetic density as the critical threshold for the correct algorithm to emerge (Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models).

Post-training data selection sees a theoretical response to Issue 14's strong negative conclusion. GIST starts from the Gibbs-optimal objective of KL-regularized RLVR, decomposes offline data selection into response-level and token-level mismatches via bidirectional KL divergence, and provides temperature pairing and token masking rules, achieving gains with 66.6% token reduction on 0.6B–2B models (GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR); DeGMatch, in target-aware selection of long reasoning traces, identifies that candidate gradients are dominated by group-level principal directions (top-50 principal components explain 86.7–89.4% variance), using deconfounded gradient matching to improve target discrimination (DeGMatch: Target-Aware Reasoning Data Selection via Deconfounded Gradient Matching). On the multimodal CPT side, Scaling Laws for Vision-Language Fusion unifies fusion timing into a single scaling law, proving that late fusion's visual advantage decays with model size as N^{-0.5} while text penalty is size-independent, thus giving the quantitative conclusion that early fusion becomes better as scale grows under fixed budget (Scaling Laws for Vision-Language Fusion).

Community and Dynamics

Two community-level new perspectives emerge on the data-constrained repetition axis. One blog points out that Chinchilla-optimal compute allocation is miscalibrated in the data-constrained regime, proposing a token half-life threshold (about a dozen epochs) as the switch point for compute shifting from epochs to capacity (The Chinchilla scaling law is miscalibrated for the regime almost everyone now trains in); another identifies vocabulary size as a previously unexamined axis in the repetition trade-off, noting that low-frequency vocabulary rows are the main driver of multi-epoch overfitting regret, and proposes fixes such as shuffling, weight averaging, and rare-row regularization (Larger Tokenizers Suffer More Multi-Epoch Regret). Both are community reviews without controlled experiments, best treated as hypotheses to validate on one's own ≥1B runs. Additionally, a news release claims that selecting high-perplexity ("surprising") documents during training can mitigate synthetic data model collapse (AI model collapse: 'surprise' could be the antidote), but lacks experimental details and scale, so it is not yet sufficient to change any mixture decisions.

Open Questions

  1. Over what scale range does Repetition Mismatch's subsampling fix hold—above its applicability lower bound (fails at 30M), does repetition control still maintain "single small-scale run locks near-optimal mixture" on larger models and more data sources? How does it interact with the "data scheduling coupled with LR scheduling" constraint from Issue 13?
  2. Can the closed-form mixture law of Understanding the Role of Reasoning Data extrapolate to ≥1B real scale? Can its prediction of "reasoning chain length and coverage as independent mixture axes" be unified with the rewriting ratio scaling law from Issue 13's Rewriting Scaling Laws for Synthetic Data into the same mixture decision framework?
  3. Can GIST's theory-driven selection principles reproduce its gains at larger scale and across broader RL algorithms, and be directly compared against Issue 14's DataFlex-RL strong negative baseline? Is its masking rule of "keep low-probability tokens for correct responses, keep high-probability tokens for incorrect responses" mechanistically equivalent to Issue 9's LESSER gradient feature selection?
  4. Can the entity frequency distribution as a causal data statistics axis, with its "0.1% synthetic density critical threshold" and "Zipfian→uniform flattening accelerates learning" conclusions, be translated into actionable mixture prescriptions for mid-training, and explain Issue 8's Everything in Moderation per-domain coverage optimal band?

Papers in this issue

  1. Optimal pretraining mixtures of atomic facts and reasoning traces favor more atomic data as reasoning chains lengthen, with interior mixtures outperforming pure extremes.

    Editor's note

    First to explicitly model the "atomic facts vs. reasoning traces" ratio as an analyzable mixture law in controlled settings: proves the optimal atomic data weight increases with reasoning chain length and decreases with reasoning coverage (held-out ratio), gives a closed-form solution under linear attention β(z)=1/(2-z+√(z²-z+1)), and validates on FineMath–MathLake mixtures (10–20% reasoning data sweet spot, earlier introduction better). Relative to Issue 13's *Rewriting Scaling Laws for Synthetic Data*, which focuses on synthetic rewriting ratios, its increment is treating reasoning chain length and reasoning trace coverage as two independent mixture axes with theoretical predictions. Limitation: scale is only 150M, criterion is GSM8K solution CE rather than post-training ceiling, but as a controlled theory+empirical combination for reasoning data mixture, it is worth reading.

  2. GIST derives bidirectional KL selection principles showing correct responses need low-probability tokens and incorrect ones need high-probability tokens, improving RLVR accuracy while supervising 66.6% fewer tokens.

    Editor's note

    First to start from the Gibbs-optimal objective of KL-regularized RLVR, decompose offline RLVR data selection into response-level and token-level mismatches via bidirectional KL divergence, and provide actionable temperature pairing (on-policy for correct responses, near-mode for incorrect) and token masking rules (keep low-probability tokens for correct responses, keep high-probability tokens for incorrect), achieving gains with 66.6% token reduction on 0.6B–2B and outperforming seven selection baselines. Relative to Issue 14's DataFlex-RL strong negative conclusion of "no gain from data strategies," its increment is providing theory-driven selection principles with positive gains, directly responding to "evaluation criteria shifting to post-training ceiling." Limitation: scale is small, criterion is benchmark accuracy, and no direct comparison with DataFlex-RL baseline.

  3. Pretraining data statistics causally determine three sequential learning phases—frequency heuristic, positional bias, then correct algorithm—with critical data density thresholds governing whether models ever escape heuristics.

    Editor's note

    First to decompose training dynamics on entity comparison tasks into three phases—frequency → position → correct algorithm—and prove that entity frequency distribution (Zipfian vs. uniform) and task-related synthetic data density causally determine the presence and order of each phase; validates this phase structure on OLMo-2 1B–32B, with 1B/7B unable to escape positional heuristics. Provides 0.1% synthetic density critical threshold and evidence that Zipfian→uniform flattening shortens the heuristic phase. Relative to Issue 13's *Where RL Breaks*, which uses failure signals for CPT selection, its increment is treating entity frequency distribution as an explicit data statistics axis. Limitation: controlled experiments at 163M scale, OLMo analysis is observational evidence, criterion is task accuracy rather than post-training ceiling.

  4. DeGMatch improves target-aware reasoning data selection by removing population-level gradient directions that obscure target-specific variation, outperforming all baselines across math, science, and code benchmarks.

    Editor's note

    First to identify in target-aware data selection of long reasoning traces that candidate gradients are dominated by a few group-level principal directions (top-50 principal components explain 86.7–89.4% variance, and the dominant subspace remains aligned across targets), proposing deconfounding operations of subtracting group mean and removing dominant principal components before gradient matching, improving target discrimination (five-way target identification from 56% to 71%). Relative to gradient feature selection methods like Issue 9's LESSER and Issue 7's Data-DPO, its increment is treating group-level gradient structure as an explicit confounder for debiasing. 7B main student, cross three domains, multi-budget preference alignment; limitation is criterion remains downstream benchmark rather than RL trainability.

  5. A unified scaling law shows early fusion preserves text performance while late fusion's vision advantage decays with model size, making early fusion preferable at scale.

    Editor's note

    First to unify early/late fusion timing in VLMs into a single scaling law: decomposes validation loss into a base learning trajectory (cross-modal exchange rate w_m characterizes contribution rates of different data types) and a distribution shift term driven by modal competition (fractional competition form ρ(R)=kR/(kR+1-R)), and proves late fusion's visual advantage decays with model size as N^{-0.5} while text penalty is size-independent, thus giving the quantitative conclusion that early fusion becomes better as scale grows under fixed budget. Relative to Issue 10's *Scaling Laws for Multimodal Data Mixtures* unified three-modal mixture scaling law, its increment is treating fusion timing (coupling of PT duration and CPT mixture ratio R) as an explicit research object. Limitation: scale stops at 3.5B, criterion mainly validation loss, and no public code or data recipe.