Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

Summary (Overview)

• Core contribution: This paper proposes a novel rewriting scaling law that predicts held-out loss as a function of model size ($N$), training tokens ($D$), rewrite fraction ($R$), and seed set size ($T$), while accounting for the compute cost of generating synthetic rewrites.
• Two settings studied: (1) Compute-limited — abundant natural data with an external, more-powerful rewriter; (2) Data-limited — fixed seed set with a bootstrapped rewriter of equal size to the target model.
• Key finding (RQ1): The scaling law accurately predicts the optimal rewrite fraction and achieved loss at held-out budgets, extrapolating 33× in FLOPs (from $3 \times 10^{18}$ to $10^{22}$ FLOPs) with final loss within 1.1% of prediction.
• Key finding (RQ2): Models trained on rewrites can surpass their rewriter's capabilities — a $10^{22}$-FLOP run with 32% rewrites outperforms SmolLM2-1.7B-Instruct on 10/16 Paloma web domains at one-tenth the rewriter's training compute.
• Optimal rewrite fraction increases with compute: from under 10% at $3 \times 10^{18}$ FLOPs to 32% at $10^{22}$ FLOPs in the compute-limited setting, and increases with epoch budget in the data-limited setting.

Introduction and Theoretical Foundation

Background and Motivation

Large language models improve predictably with scale via scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022). However, compute availability has grown faster than high-quality natural pretraining data (Villalobos et al., 2022; Kim et al., 2025). This has motivated synthetic rewrites — documents generated by prompting a language model to rewrite a natural document in a target style (e.g., Wikipedia-style, math textbook-style, FAQ, table, tutorial).

Key Challenges

Synthetic rewrites have two fundamental drawbacks:

1. Compute cost: Rewriting a document costs a significant fraction of the compute of training on it.
2. Capability bound: Rewrites are generated by a model with finite capability — it is unclear whether this bounds the capabilities of models trained on these rewrites.

Theoretical Foundation

The paper extends the Chinchilla scaling law (Hoffmann et al., 2022) by adding two terms: a rewrite-fraction-dependent irreducible loss $E(R)$ and a repetition penalty $F(R, D, T)$.

Methodology

The Rewriting Scaling Law

The proposed scaling law predicts held-out loss $L$ for an $N$-parameter model trained on $D$ tokens with seed set size $T$ and rewrite fraction $R$:

$$L(N, D, R, T) = \underbrace{\frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}}_{\text{Chinchilla terms}} + \underbrace{E(R)}_{\text{rewrite-fraction-dependent irreducible loss}} + \underbrace{F(R, D, T)}_{\text{repetition penalty}}.\tag{1}$$

Components:

• Chinchilla terms ($A/N^{\alpha} + B/D^{\beta}$): Standard model size and dataset size contributions.
• Rewrite-fraction-dependent irreducible loss: $$E(R) = E_0 - \lambda_n \log(1 - (1 - \rho_n)R) - \lambda_s \log(\rho_s + (1 - \rho_s)\bar{R})$$ where $\lambda_n, \lambda_s, \rho_n, \rho_s, E_0$ are fitted parameters.
• Repetition penalty (from Lovelace et al., 2026): $$F(R, D, T) = P(1-R)^s \left((1-R)D/T - 1\right)_+$$ where $P$ and $s$ are fitted parameters. This term is nonzero only when natural tokens are epoched (data-limited setting).

Compute Accounting

Total FLOPs account for both target-model training and rewrite generation:

$$C_{\text{total}} := \underbrace{6ND}_{\text{target-model training}} + \underbrace{2N_r DR}_{\text{source prefill}} + \underbrace{2N_r DR}_{\text{rewrite decoding}}.\tag{2}$$

The optimal rewrite fraction is estimated as:

$$\widehat{R}^{*} = \arg\min_{R \in [0,1]} \min_{N,D} L(N, D, R, T) \quad \text{s.t.} \quad 6ND + 4N_r RD = C_{\text{total}}.\tag{3}$$

Experimental Setup

• Architecture: Qwen3 dense models with Llama 3 tokenizer, trained with Adam and Hyperball.
• Compute-limited data: FineWeb-Edu with FinePhrase rewrites from SmolLM2-1.7B (trained at ~$10^{23}$ FLOPs).
• Data-limited data: 5B-token seed set from FineWeb-Edu; bootstrapped rewriters pretrained on the seed set and SFT on 7M tokens of GLM-5.2 rewrites.
• Rewrite styles: math exposition, FAQ, table, tutorial (FinePhrase protocol).
• Evaluation: Paloma BPB (macro-averaged) and Olmo 3 base easy suite (QA, Math, Code) — all reported as bits-per-byte (BPB).
• Scaling ladders: IsoFLOP ladders at 11 rewrite fractions $R \in \{0, 0.1, \ldots, 1.0\}$.

Empirical Validation / Results

Compute-Limited Setting (RQ1)

The estimated optimal rewrite fraction $\widehat{R}^{*}$ monotonically increases with FLOP budget across all evaluations (Figure 2). Key observations:

• Code and Math: $\widehat{R}^*$ rises to 1.0 within the scaling ladder — the rewriter's training data (FineMath, Stack-Edu, StarCoderData) contains higher quality math/code data than FineWeb-Edu.
• QA and Paloma: Extrapolated asymptotes of 0.77 and 0.32 respectively.
• Substantial loss differences: At $10^{22}$ FLOPs, $R=1$ vs $R=0$ reduces Math BPB from 1.33 to 0.80 and Code BPB from 1.04 to 0.53.

Validation at $10^{22}$ FLOPs: The scaling law predicted $R = 0.32$ as optimal; among four trained runs ($R \in \{0, 0.32, 0.5, 1.0\}$), $R = 0.32$ achieved the lowest Paloma BPB, with forecast error of only -1.1%.

Compute-Limited Setting (RQ2)

The $10^{22}$-FLOP run at $R = 0.32$ outperforms its rewriter (SmolLM2-1.7B-Instruct) on 10 of 16 Paloma sources:

Table 1: Per-source Paloma BPB comparison. The $R=0.32$ run at $10^{22}$ FLOPs achieves lower BPB than the rewriter on 10/16 sources; bold marks lowest BPB per row. The rewriter's performance does not represent a hard limit on gains from its rewrites.

Data-Limited Setting (RQ1)

• The best observed rewrite fraction increases with epoch budget $W$ at every target size (e.g., for 3B: from $R=0.13$ at $W=3$ to $R=0.71$ at $W=24$).
• Scaling law fit to budgets through $W=18$ predicts held-out $W=21$ and $W=24$ runs:
  • Measured loss at $\widehat{R}^*$ within 0.74% of best observed at all target sizes for both held-out budgets.
  • Worst case: 3B at $W=24$, where $\widehat{R}^*=0.47$ achieves loss 1.14% above best observed $R=0.71$.

Data-Limited Setting (RQ2)

• Rewrites beat natural-only epoching at every model size (550M to 3B).
• Natural-only runs overfit for epoch budgets $W > 3$.
• Task dependence: On Math and Code, rewrites beat natural-only at every budget; on QA, natural-only is best through $W=6$ (by 0.014 BPB), after which rewrites win.

Theoretical and Practical Implications

Theoretical Implications

1. Synthetic rewriting is predictable via scaling laws: The optimal methodological choice (rewrite fraction) can be forecast from small-scale experiments, extending the Chinchilla framework to account for synthetic data generation costs.

2. Scaling does not break down at the rewriter's capability: Models trained on rewrites can exceed the rewriter's performance, suggesting that rewrites act as a form of data distillation that can transfer knowledge beyond the rewriter's own evaluation performance.

3. Compute accounting matters: Prior scaling law studies of synthetic data (Kang et al., 2025; Qin et al., 2026) did not charge for generation compute, answering a different question ("how much to rewrite when rewrites are free"). This paper's cost-adjusted framework is more practically relevant.

Practical Implications

1. Optimal rewrite fraction scales with budget: Practitioners should increase the fraction of synthetic rewrites as compute budgets grow, since rewrite generation costs shrink relative to training costs.

2. Bootstrapped rewriting works: Even when the rewriter is the same size as the target and trained on the same seed set, rewrites provide gains — critical for frontier-scale workflows where no stronger external rewriter exists.

3. Task-specific tuning: The optimal rewrite fraction varies dramatically by domain (math/code benefit more than general web text), suggesting that task-aware data mixing is valuable.

Conclusion

This paper demonstrates that synthetic rewriting can be studied through the scaling law lens. The proposed rewriting scaling law (Eq. 1) accurately predicts:

• The optimal rewrite fraction at held-out compute budgets (within 1.1–1.14% of best observed loss).
• That positive rewrite fractions are optimal even when targeting capabilities beyond the rewriter.

Key limitations:

• Results are specific to FineWeb-Edu (DCLM shows smaller gains, suggesting strong natural data benefits less from rewriting).
• Only four FinePhrase rewrite styles and one external rewriter tested per setting.
• Compute accounting is approximate ($4N_r$ FLOPs per rewritten token, assuming rewrite length equals source length, no rewriter pretraining cost).

Future directions: Studying how optimal rewrite fractions vary across rewrite prompt distributions and rewriter families, and refining compute accounting for decoding costs.