The benefit boundary of synthetic data begins to be priced by compute
Highlights of this issue
The clearest signal this issue is that the benefit boundary of synthetic data has shifted from a 'fixed ratio' to 'pricing based on compute and data constraints'. Rewriting Scaling Laws for the first time explicitly incorporates the generation FLOPs of synthetic rewriting into the scaling law, jointly predicting loss with rewriting ratio R and natural data repetition penalty, and providing an extrapolable optimal rewriting ratio—under compute-limited settings, it rises from less than 10% at 3×10^18 FLOPs to 32% at 10^22 FLOPs; extrapolating to 33× FLOPs to train a 22B model yields an error of only 1.1%, and this model surpasses its rewriter on 10 out of 16 Paloma sources (Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints). The sweet spot of rewriting ratio becomes predictable for the first time and monotonically increases with compute, directly addressing the core concern of 'benefit boundary of synthetic data' in this direction; additionally, in the data-limited setting, a same-size rewriter rewriting its own pretraining data outperforms pure natural data at every model scale.
In parallel, the 'pre-pretraining' stage of synthetic data faces a compute-matched cold shower. Rethinking Pre-pretraining systematically examines synthetic pre-pretraining (PPT) under three training regimes (data-limited, data-unlimited, over-Chinchilla, up to 881M), finding that allocating compute to PPT never outperforms pure natural language training; previously reported NCA gains are highly sensitive to weight initialization (the original scheme reproduces a 5.4% perplexity gain, but after changing initialization, the baseline surpasses it by 9.0%), and what PPT learns is forgotten when transitioning to natural language; interleaved mixing can prevent forgetting but does not improve natural language performance (Rethinking Pre-pretraining). This is a strong warning for ratio decisions that rely on PPT.
New constraints emerge in the design of data schedules for mid-training: Data Schedules and LR Schedules prove that the optimality of data schedules (the temporal allocation of PT/MT mixing ratios) and learning rate schedules are interdependent—the same data schedule shows reversed rankings under full LR decay versus constant LR, and the optimal design is 'LR decay starts after MT data enters', validated across 250+ controlled runs spanning dense/MoE models (Data Schedules and Learning Rate Schedules Should Be Designed Together). In the direction of post-training upper bounds, Where RL Breaks uses failing n-grams from RL rollouts as retrieval signals, and with a continued pretraining corpus of only 35M tokens, it repairs low-value states that on-policy RL cannot self-correct; gains are largest when inserted before mid-training, and the corpus transfers across model families (Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining).
Community and dynamics
The benefit boundary of synthetic data is also a focus in community discussions. AI Model Collapse: Why Synthetic Training Data Hits a Hard Wall and The Data Wall: Why Synthetic Data Buys Time but Not Accuracy both emphasize that the utility of synthetic data concentrates in post-training rather than pretraining, and the optimal ratio depends on real data supply and validator quality rather than a fixed percentage—this complements the conclusion of this issue's Rewriting Scaling Laws that the ratio rises with compute and the rewriter's capability is not an upper bound; validator quality is becoming a moderating variable alongside compute. Regarding checkpoint selection, Good Pretraining, Bad SFT further refines: even when the aggregate score of the long-context stage is high, checkpoints from Constant/Cooldown are misranked after SFT, and using 'solution density' (the probability of maintaining score under Gaussian perturbation) as a selection signal is more promising. Mid-training as a corpus evaluation protocol becomes practical: Mid-training: the stage between pretraining and fine-tuning treats the annealing stage as a cheap 'candidate mixture evaluation' protocol, citing Llama-3's technique of covering 40B tokens with 30% weight to test corpus fit before full continued pretraining. At the corpus statistics level, the official version of 'Counting Documents Is Not Counting Text' (WAC) further reframes PDF extraction quality as 'token yield' rather than a cleanliness variable (Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics).
Open questions
- Does the optimal rewriting ratio from Rewriting Scaling Laws hold under the 'post-training upper bound' criterion (pass@k, RL trainability) rather than base loss? Can the patterns under the FineWeb-Edu/FinePhrase setting transfer to other natural corpora, rewriting styles, and rewriter scales (e.g., same-size rewriters in the data-limited setting)?
- Do the compute-matched negative conclusions of Rethinking Pre-pretraining (PPT never outperforms natural language, initialization sensitivity, PPT→PT transition forgetting) hold at ≥1B scale and with more synthetic pre-curricula—could PPT benefits show scale-dependent ranking reversals like repetition epochs?
- Does the 'data schedule and LR schedule ranking reversal' from Data Schedules remain when the criterion is post-training upper bound? How far can the joint guideline 'LR decay starts after MT data enters' transfer across corpus ratios and multi-head budgets?
- The previous issue proposed 'search-reachable failures are better than repeating solved problems'; this issue sees a counterpart on the pretraining side: Where RL Breaks uses failing n-grams as retrieval signals for CPT. Is this selection signal stable across more domains, larger models, and non-public corpora, and is its repair effect robust to RL algorithms other than GRPO and the construction of matched controls?
Papers in this issue
A rewriting scaling law predicts optimal synthetic data fractions within 1.1% of held-out loss, and models trained on rewrites can surpass their rewriter's capabilities.
Editor's noteFor the first time, explicitly incorporates the generation FLOPs of synthetic rewriting into the scaling law, connecting model size, training tokens, rewriting ratio R, and natural data repetition penalty, and proving that the optimal rewriting ratio is extrapolable: under compute-limited settings, it rises from <10% to 32% (3×10^18→10^22 FLOPs); extrapolating to 33× FLOPs to train a 22B model yields an error of only 1.1%, and this model surpasses its rewriter on 10 of 16 Paloma sources—the rewriter's capability is not an upper bound on gains. Compared to Scaling Repeated-Data (issue 10) and Scaling Laws for Mixture Pretraining (issue 3), which treat repetition/mixing as scaling variables, the increment is bringing 'rewriting generation cost' into the shared budget and turning the synthetic ratio into a decision variable predictable by compute. The criterion remains base loss (Paloma BPB), but scale, extrapolation validation, and the data-limited same-size rewriter comparison suffice to change ratio decisions.
Synthetic pre-pretraining never beats compute-matched natural-language training; reported gains stem from suboptimal baselines and are forgotten across the phase transition.
Editor's noteFor the first time, systematically examines synthetic pre-pretraining (PPT) under compute-matched settings: across three training regimes (data-limited, unlimited, over-Chinchilla, up to 881M) and four curricula, allocating compute to PPT never outperforms equal-compute pure natural language training; previously reported NCA gains are highly sensitive to weight initialization (the original scheme reproduces a 5.4% gain, but after changing initialization, the baseline surpasses it by 9.0%), and what is learned during PPT is forgotten when transitioning to natural language; interleaved mixing only prevents forgetting but does not improve natural language performance. Relative to studies like Priming Base Models (issue 11) that treat mid-training as a ratio lever, the increment is treating the 'synthetic pre-pretraining stage' as an independent object of study and first making initialization a key ablation axis, directly refuting ratio hypotheses that rely on PPT warm-up. Limitations are the maximum 881M not reaching the ≥1B threshold and the criterion being validation BPB and downstream tasks, but it is strong negative evidence for the synthetic data benefit boundary.
Data and learning-rate schedules must be designed jointly: PT replay helps under LR decay but hurts under constant LR, with optimal decay starting after MT entry.
Editor's noteFor the first time, systematically proves that the optimality of data schedules (temporal allocation of PT/MT mixing ratios) in mid-training and learning rate schedules are interdependent: the same data schedule shows reversed final MT loss rankings under full LR decay (FDR) versus constant LR; the optimal joint design is 'LR decay starts after MT data enters', with the optimal relative LR decay duration stably in 10–40%; TS and ZL data schedules prefer different decay durations. Relative to Priming Base Models (issue 11), which treats mid-training ratios as a corpus design variable, the increment is explicitly making LR schedule a design axis coupled with data schedule and providing a bias-variance theoretical explanation. With 630M/2.99B, 250+ controlled runs across dense/MoE and multiple budgets, it imposes a strong design constraint on mid-training ratio evaluation: 'data schedules cannot be evaluated independently of LR schedules'.
RL failures in language model reasoning are localized to characteristic n-grams, and repairing them via targeted continued pretraining on just 35M tokens outperforms matched controls in 10 of 11 settings.
Editor's noteFor the first time, directly uses failing n-grams from RL rollouts as retrieval signals for continued pretraining corpora: low-accuracy phrases mark on-policy, KL-anchored RL's unrecoverable low-value states; using infini-gram retrieval of surrounding 2k token windows with only 35M tokens (3-4 orders of magnitude smaller than typical mid-training), gains are largest when inserted before mid-training (GSM8K three subsets +2.0), and the corpus transfers across model families (Qwen3-4B→1.7B leads by +2.6 over no-CPT). Relative to the previous issue's 'search-reachable failures are better than repeating solved problems', it pushes failure signals to the pretraining side and provides an actionable pipeline using 'the model's own failures' as CPT selection signals. Limitations are small absolute gains, mostly single-seed, and reliance on public corpora, but it directly hits the 'criterion shift to post-training upper bound' and echoes the failure-selection thread of GapFT (issue 11).
RVL-CDIP is compromised by label errors, test-train leakage, and spurious Bates codes, making published accuracy gains smaller than OCR-engine choice effects.
Editor's noteFor the first time, conducts a multi-dimensional systematic audit of RVL-CDIP and quantifies defect interactions: about 10.5% of document labels are erroneous (letter class up to 31.26%), 35.2% of the test set duplicates or near-duplicates training samples, page ID codes alone achieve 69% classification (a -11 point drop when SOTA models are absent), and OCR engine choice affects accuracy (4.1–8.3 points) more than the compared architectural differences. Relative to ReScraper (issue 8) and WeVisDoc (issue 6), which treat parsers as a data quality axis, the increment is explicitly quantifying 'benchmark defects', pointing out that RVL-CDIP scores are a composite of classification ability and learning benchmark defects, and releasing repaired corpora, annotations, and OOD sets. This is a strong warning for any research using such document classification benchmarks to judge corpora/parsers, and the repaired corpus can be directly used to improve data recipes.
Benchmark hinting converts benchmarks into continuous difficulty ladders, yielding accurate scaling laws that forecast large-model accuracy within 3 points where standard methods fail by 13.
Editor's noteFor the first time, proposes 'hinted scaling laws': using privileged information (reference solution truncation/masking, strong model logit mixing) to turn a benchmark into a continuous variant with hint level H, allowing small models with near-zero scores on the original benchmark to still have extrapolable signals, and fitting a joint law L(C,H) to predict large model raw accuracy (on the Delphi ladder, GSM8K prediction error from 10^20→10^23 FLOPs drops from 13 points to within 3 points). This provides a downstream accuracy extrapolation path that does not rely on base loss for proxy→target extrapolation in this direction (small model zero scores prevent extrapolation), serving as a complementary tool for the 'post-training upper bound' criterion. Limitations are that it does not address corpus ratios itself and its Delphi data recipe is not public; worth reading as extrapolation methodology.
