# How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

> Structured pedagogical formats (math, table, FAQ, tutorial) outperform all prior synthetic and curated data, with generator performance saturating at 1B parameters for 30x cost reduction.

- **Source:** [arXiv](https://arxiv.org/abs/2604.13977)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/o8rY3b
- **Whiteboard:** https://picx.dev/p/o8rY3b/image

## Summary

## Summary (Overview)

- This paper presents the first systematic, controlled study of synthetic pretraining data generation across three key design axes: **rephrasing strategy (prompt design)**, **generator model** (scale and architecture), and **source/mix-in data composition**.
- The authors generated **over one trillion tokens** to evaluate 12 different rephrasing prompts, 6 model families (135M to 27B parameters), and multiple source/mix-in dataset combinations.
- **Key finding**: Structured pedagogical formats (math, table, FAQ, tutorial) consistently outperform curated web baselines (DCLM) and all prior synthetic methods (Nemotron-HQ-Synth, REWIRE, etc.).
- **Key finding**: Generator performance saturates at ~1B parameters; scaling to 12B or 27B provides no additional benefit while increasing compute costs 5–10×.
- The authors release **FINEPHRASE**, a 486-billion-token open synthetic dataset that outperforms all existing synthetic baselines while achieving up to **30× reduction in generation cost**.

## Introduction and Theoretical Foundation

The pretraining data landscape has evolved through three paradigm shifts:

1. **Curated corpora era**: Early models (GPT, BERT) relied on BookCorpus and Wikipedia.
2. **Large-scale web crawl era**: C4, The Pile, FineWeb, and DCLM scaled to trillion-token corpora following scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022).
3. **Synthetic data era**: As web data plateaus, LLMs are used to rephrase existing text into optimized variants.

The motivation is the **exhaustion of usable real-world data** and the need to maximize the utility of available text. Prior work explored individual approaches in isolation: WRAP (stylistic rewriting), Nemotron-CC (QA pairs, knowledge lists), BeyondWeb (continuation, summarization), and REWIRE (guided transformation). However, **no systematic comparison exists across these methods**, and the interactions between design choices remain unexplored.

The theoretical foundation draws on observations that **educational clarity drives model performance** (Gunasekar et al., 2023) and that **structured signals benefit pretraining** (Galke et al., 2024; Cheng et al., 2024; Yamaguchi et al., 2026). The paper defines a *pedagogical format* as "a data structure that organizes information into discrete, logical, and queryable units."

The formal problem setting constructs pretraining configurations from:
- A rephrasing prompt $\mathcal{P}$
- An instruction-tuned generator model $G$
- A mix-in dataset $\mathcal{D}_{\text{mix}}$ and source dataset $\mathcal{D}_{\text{source}}$

Validation trains an LM $M$ from scratch on a mixture of rephrased samples and $\mathcal{D}_{\text{mix}}$ for 21B tokens.

## Methodology

### Rephrasing Strategy (Prompt Design)

**Established prompts** (8 total):
- **Nemotron-HQ-Synth** (5 prompts): Diverse QA Pairs, Extract Knowledge, Distill, Wikipedia, Knowledge List
- **REWIRE** (1 prompt): Guided Rewrite (expert persona with meta-reasoning)
- **BeyondWeb** (2 prompts): Continue, Summarize

**New pedagogical structured prompts** (4 formats):

| Format | Prompt Description |
|--------|-------------------|
| **math** | "Rewrite the document to create a mathematical word problem... Provide a step-by-step solution that shows the calculation process clearly." |
| **faq** | "Rewrite the document as a comprehensive FAQ... Order questions logically from foundational to advanced..." |
| **table** | "Rewrite the document as a structured table that organizes the key information, then generate one question-answer pair based on the table." |
| **tutorial** | "Rewrite the document as a clear, step-by-step tutorial... Use numbered steps or bullet points where appropriate." |

### Generator Model

- **Parameter scale**: Gemma 3 (270M, 1B, 4B, 12B, 27B) and SmolLM2 (135M, 360M, 1.7B)
- **Architectural families** (at ~1B scale): Gemma 3, Llama 3.2, Qwen 3, Granite 3.1, Falcon 3, SmolLM2

### Mix-in and Source Data

- **Source datasets** ($\mathcal{D}_{\text{source}}$): DCLM, Cosmopedia, FineWeb-HQ (scores 4–5), FineWeb-LQ (scores 0–1)
- **Mix-in datasets** ($\mathcal{D}_{\text{mix}}$): DCLM, Cosmopedia, FineWeb, FWHQ, FWLQ

### Experimental Setup

- **Student model**: 1.2B-parameter Qwen 2 architecture (28 layers, hidden dim 2048, GQA with 16 query heads, 8 KV heads)
- **Tokenizer**: Llama 3.2 tokenizer (vocab 128,256)
- **Training**: 64 NVIDIA H100 GPUs, batch size 512, sequence length 4096, 10,000 steps (~21B tokens), AdamW ($lr = 5 \times 10^{-4}$, $\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay 0.1)
- **Default setup**: 50/50 synthetic-to-original mixture, Gemma 3 1B generator, FWHQ for both source and mix-in
- **Evaluation**: 12 benchmarks across 6 categories (General Knowledge, Reading Comprehension, Reasoning, NLU, Math, Table Understanding) with 3-shot cloze prompting

## Empirical Validation / Results

### Rephrasing Strategy Results

**Existing datasets baseline** (Table 2):

| Dataset | Macro Avg |
|---------|-----------|
| DCLM | **13.77** |
| Ultra-FineWeb | 13.00 |
| FineWeb-HQ | 11.82 |
| FineWeb-LQ | 8.83 |
| Nemotron-HQ-Synth★ | 13.54 |
| REWIRE★ | 13.49 |
| Cosmopedia★ | 10.33 |
| SYNTH★ | 10.03 |

**Established prompts** (Table 3): Only Diverse QA Pairs (14.58), Continue (13.73), and Guided Rewrite (13.72) rival DCLM. Extract Knowledge (-1.96) and Knowledge List (-0.66) show substantial deficits.

**Pedagogical prompts** (Table 4): All four formats beat DCLM:
- **math**: 15.31 (+1.54)
- **table**: 14.83 (+1.06)
- **faq**: 14.45 (+0.68)
- **tutorial**: 14.30 (+0.53)

### Generator Model Results

**Parameter scale**: Performance saturates at 1B parameters. Gemma 3 1B (15.31) outperforms 27B (14.76). SmolLM2 scaling: 135M → 12.69, 360M → 14.45, 1.7B → 15.88.

**Multi-scale student validation** (Table 5): The saturation trend holds across student sizes (500M to 6.2B):

| Student Scale | 270M Gen | 1B Gen | 4B Gen | 12B Gen | 27B Gen |
|---------------|----------|--------|--------|---------|---------|
| 500M | 10.96 | 12.14 | 11.50 | 11.49 | 11.50 |
| 1.2B | 13.80 | **15.31** | 15.06 | 14.68 | 14.76 |
| 2.9B | 13.80 | 15.97 | 14.75 | **16.40** | 15.72 |
| 6.2B | 14.39 | 16.85 | 16.69 | **17.06** | 16.69 |

**Exception**: The complex Guided Rewrite prompt benefits from 4B models (14.58 vs. 13.29 for 270M), suggesting multi-step prompts raise the minimum capability threshold.

**Architectural families** (Table 6): SmolLM2 1.7B (16.55) outperforms all others by 1.01–2.06 points, primarily due to superior reading comprehension (SQuAD v2, +5.12 over average), likely from its Smol-Rewrite instruction tuning data.

### Mix-in and Source Data Results

**Necessity of mixing** (Table 7): Synthetic-only training underperforms mixed training across all prompts. Pure synthetic data fails to provide NLU capabilities (WinoGrande, PIQA, HellaSwag) and risks model collapse.

**Mix-in selection** (Table 8): DCLM (14.32) and FWHQ (14.30) are optimal mix-in choices. With LQ source, FWHQ mix-in (12.99) slightly beats DCLM (12.62).

**Source quality and up-cycling** (Table 9): When $\mathcal{D}_{\text{mix}}$ is fixed at high quality, source quality matters less (max delta 1.78 points). Key result: **rephrased FWLQ + FWHQ mix-in (12.99) outperforms 100% FWHQ alone (11.82)** by +1.17 points, demonstrating successful up-cycling of noisy web text.

### Qualitative Analysis

- **Template collapse**: Qwen 3 achieves 100% instruction adherence but suffers from severe template collapse (7,619/10k outputs share identical prefix "Problem:"). SmolLM2 maintains 1,897 distinct openings despite only 68% complete solutions.
- **Diversity over consistency**: Semantic Vendi scores confirm a negative correlation ($\rho = -0.29$, $p < 0.01$) between rigid formatting consistency and downstream performance.

### Cost-Benefit Analysis

- Small models (1B class) with structured prompts dominate the Pareto frontier
- Scaling to 12B/27B increases GPU time 5–10× while often decreasing performance
- Most efficient: table + SmolLM2 1.7B (8 GPU-days); most expensive: Guided Rewrite + Gemma 3 27B (>15 GPU-months)

### FINEPHRASE Results

| Dataset | Generator | Tokens | GPU-hrs | Tok/GPU-hr |
|---------|-----------|--------|---------|------------|
| Cosmopedia | Mixtral 8x7B | 25B | >10K | <2.5M |
| SYNTH | fine-tuned | 80B | 4K | 20M |
| REWIRE | Llama 3.3 70B | 400B | ~352K | ~1.1M |
| **FINEPHRASE** | **SmolLM2 1.7B** | **486B** | **~14.7K** | **~33.1M** |

FINEPHRASE-Table achieves a peak macro-average of **17.18**, improving over DCLM by +3.41 and Nemotron-HQ-Synth by +3.63, while consuming 24× less compute than REWIRE.

## Theoretical and Practical Implications

1. **Prompt design dominates**: The choice of rephrasing strategy is the single most important factor for downstream performance, outweighing generator scale and source data quality.

2. **Small generators are sufficient**: The finding that ~1B generators saturate performance challenges the common assumption that larger models are necessary for high-quality synthetic data, contradicting REWIRE's claims. This has significant cost implications for the community.

3. **Structured pedagogy works**: Converting flat web text into mathematical problems, tables, FAQs, and tutorials creates higher-density learning signals by "explicitly surfacing reasoning chains and relational data that are otherwise implicit or noisy in source web text."

4. **Diversity > consistency**: The template collapse phenomenon reveals that rigid formatting consistency can saturate learning signals. Training data diversity matters more than surface-level adherence to instructions.

5. **Synergy between synthetic and web data**: Synthetic data provides logical depth and specialized knowledge, while original web text preserves commonsense reasoning and linguistic diversity. The optimal configuration is a mixture, not pure synthetic data.

6. **Up-cycling low-quality data**: Rephrasing noisy web text can expand the usable data reservoir, with rephrased low-quality data + high-quality mix-in outperforming high-quality data alone.

## Conclusion

The paper's main takeaways:

1. **Rephrasing prompt design** is the dominant factor for synthetic pretraining data quality
2. **Structured pedagogical formats** (math, table, FAQ, tutorial) exceed simple paraphrasing
3. **Generator capacity beyond 1B parameters** yields negligible gains for most prompts
4. **Source quality has minimal impact** when paired with robust mix-in data
5. **Generative diversity outweighs formatting consistency**
6. **Original web data remains essential** for commonsense reasoning and linguistic variety

The authors release FINEPHRASE (486B tokens), all prompts, and the generation framework (https://github.com/huggingface/finephrase, https://hf.co/datasets/HuggingFaceFW/finephrase) to enable reproducible research.

**Limitations**: Proxy 1.2B student model (though validated up to 6.2B), English-only benchmarks, computational constraints on full multi-seed confidence intervals, and hardware-dependent cost comparisons. Future work could explore larger student scales, multilingual evaluation, and longer training horizons.

---

_Markdown view of https://picx.dev/p/o8rY3b, served by PicX — AI-generated visual whiteboard summaries of research papers._