Summary (Overview)

• This paper presents the first systematic, controlled study of synthetic pretraining data generation across three key design axes: rephrasing strategy (prompt design), generator model (scale and architecture), and source/mix-in data composition.
• The authors generated over one trillion tokens to evaluate 12 different rephrasing prompts, 6 model families (135M to 27B parameters), and multiple source/mix-in dataset combinations.
• Key finding: Structured pedagogical formats (math, table, FAQ, tutorial) consistently outperform curated web baselines (DCLM) and all prior synthetic methods (Nemotron-HQ-Synth, REWIRE, etc.).
• Key finding: Generator performance saturates at ~1B parameters; scaling to 12B or 27B provides no additional benefit while increasing compute costs 5–10×.
• The authors release FINEPHRASE, a 486-billion-token open synthetic dataset that outperforms all existing synthetic baselines while achieving up to 30× reduction in generation cost.

Introduction and Theoretical Foundation

The pretraining data landscape has evolved through three paradigm shifts:

1. Curated corpora era: Early models (GPT, BERT) relied on BookCorpus and Wikipedia.
2. Large-scale web crawl era: C4, The Pile, FineWeb, and DCLM scaled to trillion-token corpora following scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022).
3. Synthetic data era: As web data plateaus, LLMs are used to rephrase existing text into optimized variants.

The motivation is the exhaustion of usable real-world data and the need to maximize the utility of available text. Prior work explored individual approaches in isolation: WRAP (stylistic rewriting), Nemotron-CC (QA pairs, knowledge lists), BeyondWeb (continuation, summarization), and REWIRE (guided transformation). However, no systematic comparison exists across these methods, and the interactions between design choices remain unexplored.

The theoretical foundation draws on observations that educational clarity drives model performance (Gunasekar et al., 2023) and that structured signals benefit pretraining (Galke et al., 2024; Cheng et al., 2024; Yamaguchi et al., 2026). The paper defines a pedagogical format as "a data structure that organizes information into discrete, logical, and queryable units."

The formal problem setting constructs pretraining configurations from:

• A rephrasing prompt $\mathcal{P}$
• An instruction-tuned generator model $G$
• A mix-in dataset $\mathcal{D}_{\text{mix}}$ and source dataset $\mathcal{D}_{\text{source}}$

Validation trains an LM $M$ from scratch on a mixture of rephrased samples and $\mathcal{D}_{\text{mix}}$ for 21B tokens.

Methodology

Rephrasing Strategy (Prompt Design)

Established prompts (8 total):

• Nemotron-HQ-Synth (5 prompts): Diverse QA Pairs, Extract Knowledge, Distill, Wikipedia, Knowledge List
• REWIRE (1 prompt): Guided Rewrite (expert persona with meta-reasoning)
• BeyondWeb (2 prompts): Continue, Summarize

New pedagogical structured prompts (4 formats):

Generator Model

• Parameter scale: Gemma 3 (270M, 1B, 4B, 12B, 27B) and SmolLM2 (135M, 360M, 1.7B)
• Architectural families (at ~1B scale): Gemma 3, Llama 3.2, Qwen 3, Granite 3.1, Falcon 3, SmolLM2

Mix-in and Source Data

• Source datasets ($\mathcal{D}_{\text{source}}$): DCLM, Cosmopedia, FineWeb-HQ (scores 4–5), FineWeb-LQ (scores 0–1)
• Mix-in datasets ($\mathcal{D}_{\text{mix}}$): DCLM, Cosmopedia, FineWeb, FWHQ, FWLQ

Experimental Setup

• Student model: 1.2B-parameter Qwen 2 architecture (28 layers, hidden dim 2048, GQA with 16 query heads, 8 KV heads)
• Tokenizer: Llama 3.2 tokenizer (vocab 128,256)
• Training: 64 NVIDIA H100 GPUs, batch size 512, sequence length 4096, 10,000 steps (~21B tokens), AdamW ($lr = 5 \times 10^{-4}$, $\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay 0.1)
• Default setup: 50/50 synthetic-to-original mixture, Gemma 3 1B generator, FWHQ for both source and mix-in
• Evaluation: 12 benchmarks across 6 categories (General Knowledge, Reading Comprehension, Reasoning, NLU, Math, Table Understanding) with 3-shot cloze prompting

Empirical Validation / Results

Rephrasing Strategy Results

Existing datasets baseline (Table 2):

Established prompts (Table 3): Only Diverse QA Pairs (14.58), Continue (13.73), and Guided Rewrite (13.72) rival DCLM. Extract Knowledge (-1.96) and Knowledge List (-0.66) show substantial deficits.

Pedagogical prompts (Table 4): All four formats beat DCLM:

• math: 15.31 (+1.54)
• table: 14.83 (+1.06)
• faq: 14.45 (+0.68)
• tutorial: 14.30 (+0.53)

Generator Model Results

Parameter scale: Performance saturates at 1B parameters. Gemma 3 1B (15.31) outperforms 27B (14.76). SmolLM2 scaling: 135M → 12.69, 360M → 14.45, 1.7B → 15.88.

Multi-scale student validation (Table 5): The saturation trend holds across student sizes (500M to 6.2B):

Exception: The complex Guided Rewrite prompt benefits from 4B models (14.58 vs. 13.29 for 270M), suggesting multi-step prompts raise the minimum capability threshold.

Architectural families (Table 6): SmolLM2 1.7B (16.55) outperforms all others by 1.01–2.06 points, primarily due to superior reading comprehension (SQuAD v2, +5.12 over average), likely from its Smol-Rewrite instruction tuning data.

Mix-in and Source Data Results

Necessity of mixing (Table 7): Synthetic-only training underperforms mixed training across all prompts. Pure synthetic data fails to provide NLU capabilities (WinoGrande, PIQA, HellaSwag) and risks model collapse.

Mix-in selection (Table 8): DCLM (14.32) and FWHQ (14.30) are optimal mix-in choices. With LQ source, FWHQ mix-in (12.99) slightly beats DCLM (12.62).

Source quality and up-cycling (Table 9): When $\mathcal{D}_{\text{mix}}$ is fixed at high quality, source quality matters less (max delta 1.78 points). Key result: rephrased FWLQ + FWHQ mix-in (12.99) outperforms 100% FWHQ alone (11.82) by +1.17 points, demonstrating successful up-cycling of noisy web text.

Qualitative Analysis

• Template collapse: Qwen 3 achieves 100% instruction adherence but suffers from severe template collapse (7,619/10k outputs share identical prefix "Problem:"). SmolLM2 maintains 1,897 distinct openings despite only 68% complete solutions.
• Diversity over consistency: Semantic Vendi scores confirm a negative correlation ($\rho = -0.29$, $p < 0.01$) between rigid formatting consistency and downstream performance.

Cost-Benefit Analysis

• Small models (1B class) with structured prompts dominate the Pareto frontier
• Scaling to 12B/27B increases GPU time 5–10× while often decreasing performance
• Most efficient: table + SmolLM2 1.7B (8 GPU-days); most expensive: Guided Rewrite + Gemma 3 27B (>15 GPU-months)

FINEPHRASE Results

FINEPHRASE-Table achieves a peak macro-average of 17.18, improving over DCLM by +3.41 and Nemotron-HQ-Synth by +3.63, while consuming 24× less compute than REWIRE.

Theoretical and Practical Implications

1. Prompt design dominates: The choice of rephrasing strategy is the single most important factor for downstream performance, outweighing generator scale and source data quality.

2. Small generators are sufficient: The finding that ~1B generators saturate performance challenges the common assumption that larger models are necessary for high-quality synthetic data, contradicting REWIRE's claims. This has significant cost implications for the community.

3. Structured pedagogy works: Converting flat web text into mathematical problems, tables, FAQs, and tutorials creates higher-density learning signals by "explicitly surfacing reasoning chains and relational data that are otherwise implicit or noisy in source web text."

4. Diversity > consistency: The template collapse phenomenon reveals that rigid formatting consistency can saturate learning signals. Training data diversity matters more than surface-level adherence to instructions.

5. Synergy between synthetic and web data: Synthetic data provides logical depth and specialized knowledge, while original web text preserves commonsense reasoning and linguistic diversity. The optimal configuration is a mixture, not pure synthetic data.

6. Up-cycling low-quality data: Rephrasing noisy web text can expand the usable data reservoir, with rephrased low-quality data + high-quality mix-in outperforming high-quality data alone.

Conclusion

The paper's main takeaways:

1. Rephrasing prompt design is the dominant factor for synthetic pretraining data quality
2. Structured pedagogical formats (math, table, FAQ, tutorial) exceed simple paraphrasing
3. Generator capacity beyond 1B parameters yields negligible gains for most prompts
4. Source quality has minimal impact when paired with robust mix-in data
5. Generative diversity outweighs formatting consistency
6. Original web data remains essential for commonsense reasoning and linguistic variety

The authors release FINEPHRASE (486B tokens), all prompts, and the generation framework (https://github.com/huggingface/finephrase, https://hf.co/datasets/HuggingFaceFW/finephrase) to enable reproducible research.

Limitations: Proxy 1.2B student model (though validated up to 6.2B), English-only benchmarks, computational constraints on full multi-seed confidence intervals, and hardware-dependent cost comparisons. Future work could explore larger student scales, multilingual evaluation, and longer training horizons.