# Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

> A rewriting scaling law predicts optimal synthetic data fractions within 1.1% of held-out loss, and models trained on rewrites can surpass their rewriter's capabilities.

- **Source:** [arXiv](https://arxiv.org/abs/2610.31867)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/Qp1A73
- **Whiteboard:** https://picx.dev/p/Qp1A73/image

## Summary

# Rewriting Scaling Laws for Synthetic Data with Compute and Data Constraints

## Summary (Overview)

- **Core contribution**: This paper proposes a novel **rewriting scaling law** that predicts held-out loss as a function of model size ($N$), training tokens ($D$), rewrite fraction ($R$), and seed set size ($T$), while accounting for the compute cost of generating synthetic rewrites.
- **Two settings studied**: (1) **Compute-limited** — abundant natural data with an external, more-powerful rewriter; (2) **Data-limited** — fixed seed set with a bootstrapped rewriter of equal size to the target model.
- **Key finding (RQ1)**: The scaling law accurately predicts the optimal rewrite fraction and achieved loss at held-out budgets, extrapolating 33× in FLOPs (from $3 \times 10^{18}$ to $10^{22}$ FLOPs) with final loss within 1.1% of prediction.
- **Key finding (RQ2)**: Models trained on rewrites can **surpass their rewriter's capabilities** — a $10^{22}$-FLOP run with 32% rewrites outperforms SmolLM2-1.7B-Instruct on 10/16 Paloma web domains at one-tenth the rewriter's training compute.
- **Optimal rewrite fraction increases with compute**: from under 10% at $3 \times 10^{18}$ FLOPs to 32% at $10^{22}$ FLOPs in the compute-limited setting, and increases with epoch budget in the data-limited setting.

## Introduction and Theoretical Foundation

### Background and Motivation

Large language models improve predictably with scale via scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022). However, compute availability has grown faster than high-quality natural pretraining data (Villalobos et al., 2022; Kim et al., 2025). This has motivated **synthetic rewrites** — documents generated by prompting a language model to rewrite a natural document in a target style (e.g., Wikipedia-style, math textbook-style, FAQ, table, tutorial).

### Key Challenges

Synthetic rewrites have two fundamental drawbacks:
1. **Compute cost**: Rewriting a document costs a significant fraction of the compute of training on it.
2. **Capability bound**: Rewrites are generated by a model with finite capability — it is unclear whether this bounds the capabilities of models trained on these rewrites.

### Theoretical Foundation

The paper extends the Chinchilla scaling law (Hoffmann et al., 2022) by adding two terms: a rewrite-fraction-dependent irreducible loss $E(R)$ and a repetition penalty $F(R, D, T)$.

## Methodology

### The Rewriting Scaling Law

The proposed scaling law predicts held-out loss $L$ for an $N$-parameter model trained on $D$ tokens with seed set size $T$ and rewrite fraction $R$:

$$
L(N, D, R, T) = \underbrace{\frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}}_{\text{Chinchilla terms}} + \underbrace{E(R)}_{\text{rewrite-fraction-dependent irreducible loss}} + \underbrace{F(R, D, T)}_{\text{repetition penalty}}.\tag{1}
$$

**Components:**
- **Chinchilla terms** ($A/N^{\alpha} + B/D^{\beta}$): Standard model size and dataset size contributions.
- **Rewrite-fraction-dependent irreducible loss**:
  $$E(R) = E_0 - \lambda_n \log(1 - (1 - \rho_n)R) - \lambda_s \log(\rho_s + (1 - \rho_s)\bar{R})$$
  where $\lambda_n, \lambda_s, \rho_n, \rho_s, E_0$ are fitted parameters.
- **Repetition penalty** (from Lovelace et al., 2026):
  $$F(R, D, T) = P(1-R)^s \left((1-R)D/T - 1\right)_+$$
  where $P$ and $s$ are fitted parameters. This term is nonzero only when natural tokens are epoched (data-limited setting).

### Compute Accounting

Total FLOPs account for both target-model training and rewrite generation:

$$
C_{\text{total}} := \underbrace{6ND}_{\text{target-model training}} + \underbrace{2N_r DR}_{\text{source prefill}} + \underbrace{2N_r DR}_{\text{rewrite decoding}}.\tag{2}
$$

The optimal rewrite fraction is estimated as:

$$
\widehat{R}^{*} = \arg\min_{R \in [0,1]} \min_{N,D} L(N, D, R, T) \quad \text{s.t.} \quad 6ND + 4N_r RD = C_{\text{total}}.\tag{3}
$$

### Experimental Setup

- **Architecture**: Qwen3 dense models with Llama 3 tokenizer, trained with Adam and Hyperball.
- **Compute-limited data**: FineWeb-Edu with FinePhrase rewrites from SmolLM2-1.7B (trained at ~$10^{23}$ FLOPs).
- **Data-limited data**: 5B-token seed set from FineWeb-Edu; bootstrapped rewriters pretrained on the seed set and SFT on 7M tokens of GLM-5.2 rewrites.
- **Rewrite styles**: math exposition, FAQ, table, tutorial (FinePhrase protocol).
- **Evaluation**: Paloma BPB (macro-averaged) and Olmo 3 base easy suite (QA, Math, Code) — all reported as bits-per-byte (BPB).
- **Scaling ladders**: IsoFLOP ladders at 11 rewrite fractions $R \in \{0, 0.1, \ldots, 1.0\}$.

## Empirical Validation / Results

### Compute-Limited Setting (RQ1)

The estimated optimal rewrite fraction $\widehat{R}^{*}$ **monotonically increases** with FLOP budget across all evaluations (Figure 2). Key observations:

- **Code and Math**: $\widehat{R}^*$ rises to 1.0 within the scaling ladder — the rewriter's training data (FineMath, Stack-Edu, StarCoderData) contains higher quality math/code data than FineWeb-Edu.
- **QA and Paloma**: Extrapolated asymptotes of 0.77 and 0.32 respectively.
- **Substantial loss differences**: At $10^{22}$ FLOPs, $R=1$ vs $R=0$ reduces Math BPB from 1.33 to 0.80 and Code BPB from 1.04 to 0.53.

**Validation at $10^{22}$ FLOPs**: The scaling law predicted $R = 0.32$ as optimal; among four trained runs ($R \in \{0, 0.32, 0.5, 1.0\}$), $R = 0.32$ achieved the lowest Paloma BPB, with forecast error of only **-1.1%**.

### Compute-Limited Setting (RQ2)

The $10^{22}$-FLOP run at $R = 0.32$ outperforms its rewriter (SmolLM2-1.7B-Instruct) on **10 of 16 Paloma sources**:

| Paloma source | R = 0 | R = 0.32 | Rewriter |
|---|---|---|---|
| M2D2 Wikipedia | 0.726 | **0.716** | 0.758 |
| Penn Treebank | 0.851 | **0.843** | 0.882 |
| WikiText-103 | 0.682 | **0.673** | 0.711 |
| C4 English | 0.827 | **0.828** | 0.844 |
| Falcon RefinedWeb | 0.894 | **0.887** | 0.897 |
| Gab | 1.719 | **1.579** | 1.587 |
| C4 100 Domains | 0.790 | **0.793** | 0.797 |
| M2D2 S2ORC | 0.766 | **0.760** | 0.764 |
| mC4 | 0.841 | **0.825** | 0.828 |
| TwitterAAE | 2.591 | **2.453** | 2.455 |
| Dolma v1.5 | 0.876 | 0.804 | **0.791** |
| Dolma 100 Subreddits | 1.089 | 1.022 | **1.001** |
| Manosphere | 1.112 | 1.057 | **1.030** |
| RedPajama | 0.946 | 0.775 | **0.737** |
| 4chan | 1.162 | 1.023 | **0.952** |
| Dolma 100 PLs | 1.106 | 0.694 | **0.511** |
| **Macro average** | **1.061** | **0.983** | **0.972** |

**Table 1**: Per-source Paloma BPB comparison. The $R=0.32$ run at $10^{22}$ FLOPs achieves lower BPB than the rewriter on 10/16 sources; bold marks lowest BPB per row. The rewriter's performance does **not** represent a hard limit on gains from its rewrites.

### Data-Limited Setting (RQ1)

- The best observed rewrite fraction **increases with epoch budget** $W$ at every target size (e.g., for 3B: from $R=0.13$ at $W=3$ to $R=0.71$ at $W=24$).
- Scaling law fit to budgets through $W=18$ predicts held-out $W=21$ and $W=24$ runs:
  - Measured loss at $\widehat{R}^*$ within **0.74%** of best observed at all target sizes for both held-out budgets.
  - Worst case: 3B at $W=24$, where $\widehat{R}^*=0.47$ achieves loss **1.14%** above best observed $R=0.71$.

### Data-Limited Setting (RQ2)

- **Rewrites beat natural-only epoching at every model size** (550M to 3B).
- Natural-only runs **overfit** for epoch budgets $W > 3$.
- Task dependence: On Math and Code, rewrites beat natural-only at every budget; on QA, natural-only is best through $W=6$ (by 0.014 BPB), after which rewrites win.

## Theoretical and Practical Implications

### Theoretical Implications

1. **Synthetic rewriting is predictable via scaling laws**: The optimal methodological choice (rewrite fraction) can be forecast from small-scale experiments, extending the Chinchilla framework to account for synthetic data generation costs.

2. **Scaling does not break down at the rewriter's capability**: Models trained on rewrites can **exceed the rewriter's performance**, suggesting that rewrites act as a form of *data distillation* that can transfer knowledge beyond the rewriter's own evaluation performance.

3. **Compute accounting matters**: Prior scaling law studies of synthetic data (Kang et al., 2025; Qin et al., 2026) did not charge for generation compute, answering a different question ("how much to rewrite when rewrites are free"). This paper's cost-adjusted framework is more practically relevant.

### Practical Implications

1. **Optimal rewrite fraction scales with budget**: Practitioners should increase the fraction of synthetic rewrites as compute budgets grow, since rewrite generation costs shrink relative to training costs.

2. **Bootstrapped rewriting works**: Even when the rewriter is the same size as the target and trained on the same seed set, rewrites provide gains — critical for frontier-scale workflows where no stronger external rewriter exists.

3. **Task-specific tuning**: The optimal rewrite fraction varies dramatically by domain (math/code benefit more than general web text), suggesting that task-aware data mixing is valuable.

## Conclusion

This paper demonstrates that **synthetic rewriting can be studied through the scaling law lens**. The proposed rewriting scaling law (Eq. 1) accurately predicts:
- The **optimal rewrite fraction** at held-out compute budgets (within 1.1–1.14% of best observed loss).
- That **positive rewrite fractions are optimal** even when targeting capabilities beyond the rewriter.

**Key limitations**:
- Results are specific to FineWeb-Edu (DCLM shows smaller gains, suggesting strong natural data benefits less from rewriting).
- Only four FinePhrase rewrite styles and one external rewriter tested per setting.
- Compute accounting is approximate ($4N_r$ FLOPs per rewritten token, assuming rewrite length equals source length, no rewriter pretraining cost).

**Future directions**: Studying how optimal rewrite fractions vary across rewrite prompt distributions and rewriter families, and refining compute accounting for decoding costs.

---

_Markdown view of https://picx.dev/p/Qp1A73, served by PicX — AI-generated visual whiteboard summaries of research papers._