# Bridging Compute- and Data-Optimal Pretraining

> The CD scaling law unifies compute- and data-optimal pretraining, showing derived-token effectiveness decays with model size and data availability, making classic Chinchilla allocation suboptimal in practice.

- **Source:** [arXiv](https://arxiv.org/abs/2607.25271)
- **Published:** 2026-08-18
- **Permalink:** https://picx.dev/p/XxxrQK
- **Whiteboard:** https://picx.dev/p/XxxrQK/image

## Summary

# Bridging Compute- and Data-Optimal Pretraining

## Summary (Overview)

- **Unified scaling law framework**: The paper introduces Compute-Data (CD) scaling laws that bridge two limiting regimes: compute-optimal scaling (where data is unbounded) and data-optimal scaling (where the corpus is fixed and compute grows unbounded).

- **Token effectiveness function**: Central to the framework is an effectiveness function $\eta^{strat} \in [0,1]$ that quantifies how much a derived token (from multi-epoch repetition or paraphrasing) is worth relative to a fresh token.

- **Empirical findings**: Fitting $\eta$ across model sizes from 14M to 600M parameters on the Dolma-3 corpus reveals that token effectiveness decays with both model size and tokens-per-parameter ratio, and saturates as the corpus is expanded.

- **Three training regimes**: The CD scaling law partitions training into compute-bound, data-bound, and model-bound regimes, showing that classic compute-optimal allocation is suboptimal across most practically relevant settings.

- **Practical guidance**: The framework prescribes optimal compute-data allocation, identifies when to switch between repetition and paraphrasing, and shows that the "4-epoch rule" holds only for medium-scale models near 1× Chinchilla data budgets.

## Introduction and Theoretical Foundation

### Background and Motivation

Classical scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020) assume an unbounded supply of fresh pretraining data, prescribing compute-optimal allocation between model size $N$ and dataset size $D$. However, this assumption is becoming inaccurate:

- Compute continues to grow at exponential rates (Sevilla et al., 2022)
- High-quality pretraining data remains finite and costly to curate (Villalobos et al., 2024)

The central question becomes: **For a model of size $N$ trained on a fixed corpus $D$, how much can additional compute reduce loss?**

### Two Limiting Regimes

The paper frames the problem through two extremes:

1. **Compute-optimal scaling** $L^{Chin}$: Data is unbounded, compute is the bottleneck (classical scaling laws)
2. **Data-optimal scaling** $L^{D}$: $D$ is fixed, compute is unbounded (the asymptote of additional training on a given corpus)

Modern pretraining sits between these two limits.

### Key Theoretical Contribution

The CD scaling law extends the classical Chinchilla law:

$$\mathcal{L}^{\mathrm{Chin}}(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

to incorporate derived tokens $D'$:

$$\mathcal{L}^{\mathrm{CD}}(N, D, D') = E + \frac{A}{N^{\alpha}} + \frac{B}{(D + \eta^{\mathrm{strat}}(N, \mathrm{TPP}, D'/D) \cdot D')^{\beta}}$$

where:
- $\eta = 1$ means derived tokens are as informative as fresh data
- $\eta = 0$ means derived tokens provide no benefit
- TPP = $D/N$ is the tokens-per-parameter ratio

## Methodology

### Experimental Setup

**Training configuration:**
- Models: $N$ = 14M to 600M parameters
- Fresh data: $D$ = 30M to 30B tokens from Dolma-3 150B corpus
- Derived tokens: $D'$ = 30M to 120B tokens
- Sequence length: 4096, batch size: 512
- Optimizer: AdamW with cosine learning-rate decay
- Hyperparameter sweep: learning rate $\in \{1e-4, 3e-4, 1e-3, 3e-3\}$, weight decay $\in \{0.1, 0.2, 0.4, 0.8, 1.6\}$

### Data Expansion Strategies

1. **Multi-epoch repetition**: Repeat the original corpus for additional epochs
2. **Paraphrasing**: Rephrase documents using SmolLM2-1.7B-Instruct, sampling up to 16 paraphrases per document with styles drawn uniformly from {question, math/wiki, FAQ, table}

### Empirical Characterization of $\eta$

The empirical coefficient $\eta_{emp}$ is defined as the unique value that reproduces observed loss when substituted into Eq. (2). Three key observations emerge:

1. $\eta$ decays monotonically with TPP (fresh data availability)
2. $\eta$ decays with expansion ratio $r = D'/D$
3. The quantity $\eta \cdot r$ saturates to a finite limit as $r \to \infty$

### Functional Form

The exponential form for $\eta$:

$$\eta = \frac{R^{*}}{r}\left(1 - e^{-r/R^{*}}\right)$$

where $R^{*}$ is the saturation ceiling (maximum fresh-equivalent tokens a corpus can yield through expansion), modeled as:

$$R^{*}(D, N) = K \cdot (D/N)^{\rho} \cdot N^{\sigma}$$

## Empirical Validation / Results

### Fitted Parameters

| Parameter | Value [95% CI] |
|-----------|----------------|
| $E$ | 1.35 [1.0, 1.6] |
| $A$ | 205 [90, 676] |
| $B$ | 16,597 [7,677, 34,475] |
| $\alpha$ | 0.283 [0.22, 0.37] |
| $\beta$ | 0.435 [0.40, 0.47] |

| Strategy | log K | $\rho$ | $\sigma$ | RMSE |
|----------|-------|--------|----------|------|
| Repetition | 10.93 [9.7, 13.5] | -0.42 [-0.71, -0.15] | -0.41 [-0.55, -0.35] | 0.035 |
| Paraphrase | 30.50 [30.4, 30.8] | -1.52 [-1.71, -1.22] | -1.30 [-1.35, -1.26] | 0.024 |

### Key Results

**Cross-scale validation**: Fitting on $N \in \{14, 30\}$M only and predicting held-out losses at $N \in \{60, 100, 190, 370, 600\}$M achieves RMSE 0.079 on held-out points (vs. 0.048 in-sample), demonstrating clean transfer across an order of magnitude in model size.

**Strategy comparison**: At Chinchilla-optimal budget (TPP = 20):
- Paraphrasing significantly wins for small models ($N \lesssim 50$M)
- Repetition wins for large models ($N \gtrsim 190$M)
- Crossover at $N \approx 94$M

**Loss-to-downstream transfer**: Runs from all strategies and scales collapse onto a single curve when plotted against validation BPB, confirming validation loss is a sufficient statistic for downstream capability.

**Ablations**: The constant-$R^{*}$ baseline (Muennighoff et al., 2025) is rejected by ~40% in relative leave-one-out RMSE. Both exponents in Eq. (5) are individually necessary.

### Data-Optimal Scaling Limit

As $D' \to \infty$, the CD law approaches:

$$\mathcal{L}^{D}(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{[D(1 + R^{*}(D, N))]^{\beta}}$$

## Theoretical and Practical Implications

### Three Training Regimes

The CD scaling law identifies three distinct regions:

1. **Compute-bound** (between $L^{Chin}$ and $L^D$): Additional derived tokens $D'$ still reduce loss; compute is the binding resource
2. **Data-bound** (between $L^D$ and $L^N$): Derived data has saturated to $R^{*}D$; only fresh data can further reduce loss
3. **Model-bound** (at $L^N$): Both compute and data have saturated; model size $N$ is the binding resource

### Compute-Data Pareto Frontier

The framework generalizes Chinchilla's compute-only optimization into a joint compute-data Pareto frontier. The empirical optimal path:
- Prefers scaling compute in the data-scarce regime
- Then scales both data and compute in equal proportion
- Enters a basin where fresh data dominates and further compute yields diminishing returns

### Strategy Selection Guidance

- **Recommended epochs decrease** as model size and data budget grow; 4-epoch training is recommended only for medium-scale models (~3B) near 1× Chinchilla
- **Paraphrasing is preferred** for models $\lesssim 600$M and small data budgets ($< 1\times$ Chinchilla)
- **Paraphrasing becomes ineffective** for $N \geq 7$B or large data budgets ($\geq 4\times$ Chinchilla)

## Conclusion

### Main Takeaways

1. The CD scaling law provides a unified framework bridging compute-optimal and data-optimal training through the effectiveness function $\eta^{strat}$
2. The effectiveness of derived tokens diminishes with both model size and data availability, with the saturation ceiling $R^{*}$ following a power law in $(D/N)$ and $N$
3. The framework identifies which resource (compute, data, or model) is the binding constraint for any training configuration

### Future Directions

- **Predicting $\eta$ from corpus statistics**: Using summary statistics like token diversity, n-gram overlap, or perplexity distributions to assess expansion strategies before committing compute
- **Additional strategies**: Self-distillation, synthetic structured data, and combinations of strategies that may yield higher effective $\eta$ than any single approach

### Limitations

- The functional form absorbs residual misspecification of the 1-epoch fit (can push $\eta_{emp}$ above 1)
- Only learning rate and weight decay were swept; other hyperparameters could interact with $\eta$
- Over 250k H100-hours of compute were consumed; only two expansion strategies were fit at small-to-medium scale

---

_Markdown view of https://picx.dev/p/XxxrQK, served by PicX — AI-generated visual whiteboard summaries of research papers._