Bridging Compute- and Data-Optimal Pretraining

Summary (Overview)

• Unified scaling law framework: The paper introduces Compute-Data (CD) scaling laws that bridge two limiting regimes: compute-optimal scaling (where data is unbounded) and data-optimal scaling (where the corpus is fixed and compute grows unbounded).

• Token effectiveness function: Central to the framework is an effectiveness function $\eta^{strat} \in [0,1]$ that quantifies how much a derived token (from multi-epoch repetition or paraphrasing) is worth relative to a fresh token.

• Empirical findings: Fitting $\eta$ across model sizes from 14M to 600M parameters on the Dolma-3 corpus reveals that token effectiveness decays with both model size and tokens-per-parameter ratio, and saturates as the corpus is expanded.

• Three training regimes: The CD scaling law partitions training into compute-bound, data-bound, and model-bound regimes, showing that classic compute-optimal allocation is suboptimal across most practically relevant settings.

• Practical guidance: The framework prescribes optimal compute-data allocation, identifies when to switch between repetition and paraphrasing, and shows that the "4-epoch rule" holds only for medium-scale models near 1× Chinchilla data budgets.

Introduction and Theoretical Foundation

Background and Motivation

Classical scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020) assume an unbounded supply of fresh pretraining data, prescribing compute-optimal allocation between model size $N$ and dataset size $D$. However, this assumption is becoming inaccurate:

• Compute continues to grow at exponential rates (Sevilla et al., 2022)
• High-quality pretraining data remains finite and costly to curate (Villalobos et al., 2024)

The central question becomes: For a model of size $N$ trained on a fixed corpus $D$, how much can additional compute reduce loss?

Two Limiting Regimes

The paper frames the problem through two extremes:

1. Compute-optimal scaling $L^{Chin}$: Data is unbounded, compute is the bottleneck (classical scaling laws)
2. Data-optimal scaling $L^{D}$: $D$ is fixed, compute is unbounded (the asymptote of additional training on a given corpus)

Modern pretraining sits between these two limits.

Key Theoretical Contribution

The CD scaling law extends the classical Chinchilla law:

$$\mathcal{L}^{\mathrm{Chin}}(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

to incorporate derived tokens $D'$:

$$\mathcal{L}^{\mathrm{CD}}(N, D, D') = E + \frac{A}{N^{\alpha}} + \frac{B}{(D + \eta^{\mathrm{strat}}(N, \mathrm{TPP}, D'/D) \cdot D')^{\beta}}$$

where:

• $\eta = 1$ means derived tokens are as informative as fresh data
• $\eta = 0$ means derived tokens provide no benefit
• TPP = $D/N$ is the tokens-per-parameter ratio

Methodology

Experimental Setup

Training configuration:

• Models: $N$ = 14M to 600M parameters
• Fresh data: $D$ = 30M to 30B tokens from Dolma-3 150B corpus
• Derived tokens: $D'$ = 30M to 120B tokens
• Sequence length: 4096, batch size: 512
• Optimizer: AdamW with cosine learning-rate decay
• Hyperparameter sweep: learning rate $\in \{1e-4, 3e-4, 1e-3, 3e-3\}$, weight decay $\in \{0.1, 0.2, 0.4, 0.8, 1.6\}$

Data Expansion Strategies

1. Multi-epoch repetition: Repeat the original corpus for additional epochs
2. Paraphrasing: Rephrase documents using SmolLM2-1.7B-Instruct, sampling up to 16 paraphrases per document with styles drawn uniformly from {question, math/wiki, FAQ, table}

Empirical Characterization of $\eta$

The empirical coefficient $\eta_{emp}$ is defined as the unique value that reproduces observed loss when substituted into Eq. (2). Three key observations emerge:

1. $\eta$ decays monotonically with TPP (fresh data availability)
2. $\eta$ decays with expansion ratio $r = D'/D$
3. The quantity $\eta \cdot r$ saturates to a finite limit as $r \to \infty$

Functional Form

The exponential form for $\eta$:

$$\eta = \frac{R^{*}}{r}\left(1 - e^{-r/R^{*}}\right)$$

where $R^{*}$ is the saturation ceiling (maximum fresh-equivalent tokens a corpus can yield through expansion), modeled as:

$$R^{*}(D, N) = K \cdot (D/N)^{\rho} \cdot N^{\sigma}$$

Empirical Validation / Results

Fitted Parameters

Key Results

Cross-scale validation: Fitting on $N \in \{14, 30\}$M only and predicting held-out losses at $N \in \{60, 100, 190, 370, 600\}$M achieves RMSE 0.079 on held-out points (vs. 0.048 in-sample), demonstrating clean transfer across an order of magnitude in model size.

Strategy comparison: At Chinchilla-optimal budget (TPP = 20):

• Paraphrasing significantly wins for small models ($N \lesssim 50$M)
• Repetition wins for large models ($N \gtrsim 190$M)
• Crossover at $N \approx 94$M

Loss-to-downstream transfer: Runs from all strategies and scales collapse onto a single curve when plotted against validation BPB, confirming validation loss is a sufficient statistic for downstream capability.

Ablations: The constant-$R^{*}$ baseline (Muennighoff et al., 2025) is rejected by ~40% in relative leave-one-out RMSE. Both exponents in Eq. (5) are individually necessary.

Data-Optimal Scaling Limit

As $D' \to \infty$, the CD law approaches:

$$\mathcal{L}^{D}(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{[D(1 + R^{*}(D, N))]^{\beta}}$$

Theoretical and Practical Implications

Three Training Regimes

The CD scaling law identifies three distinct regions:

1. Compute-bound (between $L^{Chin}$ and $L^D$): Additional derived tokens $D'$ still reduce loss; compute is the binding resource
2. Data-bound (between $L^D$ and $L^N$): Derived data has saturated to $R^{*}D$; only fresh data can further reduce loss
3. Model-bound (at $L^N$): Both compute and data have saturated; model size $N$ is the binding resource

Compute-Data Pareto Frontier

The framework generalizes Chinchilla's compute-only optimization into a joint compute-data Pareto frontier. The empirical optimal path:

• Prefers scaling compute in the data-scarce regime
• Then scales both data and compute in equal proportion
• Enters a basin where fresh data dominates and further compute yields diminishing returns

Strategy Selection Guidance

• Recommended epochs decrease as model size and data budget grow; 4-epoch training is recommended only for medium-scale models (~3B) near 1× Chinchilla
• Paraphrasing is preferred for models $\lesssim 600$M and small data budgets ($< 1\times$ Chinchilla)
• Paraphrasing becomes ineffective for $N \geq 7$B or large data budgets ($\geq 4\times$ Chinchilla)

Conclusion

Main Takeaways

1. The CD scaling law provides a unified framework bridging compute-optimal and data-optimal training through the effectiveness function $\eta^{strat}$
2. The effectiveness of derived tokens diminishes with both model size and data availability, with the saturation ceiling $R^{*}$ following a power law in $(D/N)$ and $N$
3. The framework identifies which resource (compute, data, or model) is the binding constraint for any training configuration

Future Directions

• Predicting $\eta$ from corpus statistics: Using summary statistics like token diversity, n-gram overlap, or perplexity distributions to assess expansion strategies before committing compute
• Additional strategies: Self-distillation, synthetic structured data, and combinations of strategies that may yield higher effective $\eta$ than any single approach

Limitations

• The functional form absorbs residual misspecification of the 1-epoch fit (can push $\eta_{emp}$ above 1)
• Only learning rate and weight decay were swept; other hyperparameters could interact with $\eta$
• Over 250k H100-hours of compute were consumed; only two expansion strategies were fit at small-to-medium scale