Summary (Overview)

• SYNPRO is a synthetic data generation framework designed to help LLMs more thoroughly learn from limited organic (human-written) text in the data-bound scaling regime, where available data falls short of compute-optimal requirements.
• The framework applies two operations—rephrasing (lexical/syntactic diversification) and reformat (task-oriented transformation)—both optimized via reinforcement learning with quality, faithfulness, and data influence rewards.
• SYNPRO achieves 3.7–5.2× the effective tokens of standard repetition at 400M and 1.1B scales, and even surpasses the non-data-bound oracle (training on equivalent unique organic data) at the 1.1B scale.
• The generator is continuously updated as pretraining plateaus, targeting content the current model has yet to absorb, making the synthetic data model-aware.
• Analyses confirm that faithful, grounded synthesis preserves both pointwise and distributional properties of the organic corpus, avoiding distribution collapse and distillation effects.

---

Introduction and Theoretical Foundation

The Data-Bound Paradigm Shift

Frontier model pretraining is undergoing a fundamental transition from compute-bound to data-bound scaling. While compute-optimal data requirements (Hoffmann et al., 2022) grow steadily with increasing compute, the supply of high-quality human-written text cannot keep pace (Villalobos & Ho, 2022; Maini et al., 2025). Beyond this transition point, "the next frontier of scaling demands an order of magnitude more data than is currently available" (Villalobos et al., 2024).

The available data ratio is formally defined as:

$$\alpha = \frac{|\mathcal{D}_{\mathrm{org}}|}{D^{*}(C)} \tag{1}$$

where $\mathcal{D}_{\mathrm{org}}$ is the organic corpus and $D^{*}(C)$ is the compute-optimal data requirement for compute budget $C$. In the data-bound regime, $\alpha$ is around 10%, meaning available organic data covers only a fraction of compute-optimal requirements.

Key Theoretical Insights

1. Data wall phenomenon: Repeated passes over the same corpus yield diminishing returns after only ~4 epochs (Muennighoff et al., 2023).
2. Underutilization hypothesis: Prior work (Frank, 2023; Warstadt et al., 2023) suggests effective learning may require much less data than current practice, implying available data may still be underutilized.
3. Synthetic data risks: Unconstrained generation can lead to distribution collapse (Shumailov et al., 2024; Dohmatob et al., 2025) or distill the generator's parametric knowledge in ways that hurt generalization (Chen et al., 2024).

This motivates the central research question: Can we generate synthetic data grounded in organic data to help LLMs learn more?

---

Methodology

1. Framework Overview

SYNPRO operates in three repeating stages:

• Stage 1 (LM pretraining): Continue pretraining on the accumulated training set until reference loss saturates.
• Stage 2 (Policy update): Update the generation policy via RL to produce more informative synthetic data for the current model.
• Stage 3 (Generation): Generate new synthetic data from the organic corpus and append to the training set.

The initial synthetic corpus and training set are constructed as:

$$\mathcal{D}_{\mathrm{syn}}^{0} = \{\pi_{0}(p, x) \mid x \in \mathcal{D}_{\mathrm{org}}\} \tag{2}$$ $$\mathcal{D}_{\mathrm{train}} = \mathcal{D}_{\mathrm{org}} \cup \mathcal{D}_{\mathrm{syn}}^{0} \tag{3}$$

2. Saturation Detection

Pretraining at each iteration continues until the reference loss fails to improve over the best of the previous two epochs:

$$\mathcal{M}_{i} = \mathcal{M}_{i-1}^{*}, \quad \mathcal{L}_{0} = \mathcal{L}(\mathcal{D}_{\mathrm{ref}} \mid \mathcal{M}_{i-1}^{*}) \tag{4}$$ $$\text{For } t = 1, 2, \ldots: \tag{5}$$ $$\mathcal{M}_{i}^{\prime} \leftarrow \mathcal{A}(\mathcal{M}_{i}, \mathcal{D}_{\mathrm{train}}), \quad \mathcal{L}_{t} = \mathcal{L}(\mathcal{D}_{\mathrm{ref}} \mid \mathcal{M}_{i}^{\prime}) \tag{6}$$ $$\text{If } t > 1 \text{ and } \mathcal{L}_{t} \geq \min(\mathcal{L}_{t-1}, \mathcal{L}_{t-2}): \text{ break} \tag{7}$$

3. Composite Reward Design

The generation policy is optimized under a composite reward:

$$r_{i}(x, \tilde{x}) = \lambda_{\text{quality}} r_{\text{quality}}(\tilde{x}) + \lambda_{\text{faithful}} r_{\text{faithful}}(x, \tilde{x}) + \lambda_{\text{influence}} r_{\text{influence}}(\tilde{x} \mid \mathcal{M}_{i}^{*}) \tag{10}$$

Quality reward: Uses DataMan (Peng et al., 2025), a tuned small LM evaluating text across 13 quality criteria:

$$r_{\text{quality}}(\tilde{x}) = \text{DataMan}(\tilde{x}) \tag{13}$$

Data influence reward: Computed as the loss reduction on a synthetic sample after the model is updated on the reference set:

$$r_{\text{influence}}(\tilde{x} \mid \mathcal{M}_{i}^{*}) = \mathcal{L}(\tilde{x} \mid \mathcal{M}_{i}^{*}) - \mathcal{L}(\tilde{x} \mid \mathcal{A}(\mathcal{M}_{i}^{*}, \mathcal{D}_{\text{ref}})) \tag{14}$$

Faithfulness reward (for rephrasing):

$$r_{\text{faithful}}(x, \tilde{x}) = \mathbf{1}[\text{BERTScore}(x, \tilde{x}) \geq \tau_{\text{sem}}] \cdot \mathbf{1}[\text{Structure}(x, \tilde{x})] \cdot \mathbf{1}\left[\frac{\text{Len}(\tilde{x})}{\text{Len}(x)} \leq \tau_{\text{len}}\right] \tag{15}$$

For reformat, a small reward model is trained (distilled from an LLM) to classify faithfulness as a binary reward.

4. Policy Optimization

The policy is updated via standard reinforcement learning:

$$\pi_{i} = \arg \max_{\pi} \mathbb{E}_{x \sim \mathcal{D}_{\mathrm{org}}, \tilde{x} \sim \pi(\cdot | p, x)}[r_{i}(x, \tilde{x})] \tag{11}$$

5. Experimental Setup

• Models: 400M and 1.1B decoder-only Transformers (OLMo2 architecture), plus a MoE-7B-A1B setting (OLMoE architecture).
• Data: DCLM-Baseline (Li et al., 2024) as organic corpus; $\alpha \in \{5\%, 10\%, 15\%\}$ for 400M, $\alpha = 10\%$ for 1.1B and MoE.
• Baselines: Repeat, QBSR (quality-based selective repetition), MATES (influence-based adaptive curriculum), RePro (rephrasing baseline).
• Evaluation: Zero-shot accuracy on 9 downstream tasks (ARC-Easy, ARC-Challenge, SciQ, Open-BookQA, HellaSwag, PIQA, WinoGrande, CommonsenseQA, SIQA).
• Hyperparameters: $\lambda_{\text{quality}} = 1$, $\lambda_{\text{faithful}} = 1$, $\lambda_{\text{influence}} = 3$; $\tau_{\text{sem}} = 0.65$, $\tau_{\text{len}} = 1.25$.

---

Empirical Validation / Results

Main Results

SYNPRO consistently outperforms all baselines across scales and data regimes:

Key findings:

• SYNPRO surpasses the non-data-bound oracle (Unique Data) at both 400M/α=15% and 1.1B/α=10%.
• The effective token count at α=15% (67.6B) exceeds the actual training tokens (61.2B).
• SYNPRO achieves ~3× the effective tokens of RePro across all settings.

Ablation Studies

Pointwise Faithfulness

• SYNPRO achieves BERTScore 0.75 (vs. 0.68 for base model; RL without faithfulness drops to 0.52).
• 96.1% faithfulness rate vs. 34.8% for base model and 55.0% for RL without faithfulness reward.
• 76% of SYNPRO samples exceed 80% named entity recall vs. 62% for base model.

Distribution Preservation

• SYNPRO retains 99.2% of cluster assignments vs. 42.2% for the base model (t-SNE/Voronoi analysis).
• Perplexity distribution of SYNPRO closely matches organic data, while the base model produces a tighter, lower-perplexity shape that misses the long tail.

Model-Awareness

• The influence reward steers the generator toward content the model has yet to absorb.
• Positive influence ratio for SYNPRO's dynamically updated policies remains consistently higher than static approaches.
• Factual reformat proportion increases from 45.3% to 54.1% across iterations, indicating the generator adapts toward the model's factual gaps.

---

Theoretical and Practical Implications

Theoretical Significance

1. Organic data is underutilized, not exhausted: SYNPRO demonstrates that the data wall is not a fundamental limit but a consequence of naive repetition. Faithful synthesis can unlock significantly more value from limited data.

2. Faithfulness prevents collapse: Grounding synthetic data in organic sources enriches rather than distorts the training distribution, avoiding the distribution collapse and distillation effects that plague unconstrained generation.

3. Model-awareness sustains gains: Static approaches decay as the model memorizes repeated content; dynamically updating the generator to target unabsorbed content maintains sustained improvement.

Practical Implications

1. Data-bound scaling is viable: SYNPRO provides a practical path for frontier pretraining beyond the organic data limit, achieving performance comparable to or exceeding non-data-bound regimes.

2. Effective token economics: At α=15%, SYNPRO achieves 8.6× the effective tokens of repetition, meaning practitioners can achieve the same performance with far less unique data.

3. General applicability: The framework works across dense (400M, 1.1B) and sparse (MoE-7B-A1B) architectures, suggesting broad applicability for future scaling efforts.

---

Conclusion

SYNPRO introduces a model-aware synthetic data generation framework that helps pretraining models more thoroughly utilize limited organic corpora. Two key insights emerge:

1. Organic data is underutilized, not exhausted — faithful synthesis can match or exceed performance achieved with much more unique data.
2. Faithfulness is essential — grounding synthetic outputs in organic sources enriches the training distribution, while unconstrained generation risks collapse and distillation.

The authors hope SYNPRO motivates future work to break the data wall and sustain LLM scaling in the data-bound regime. The code is open-sourced at https://github.com/cxcscmu/SynPro.