# Prescriptive Scaling Laws for Data Constrained Training

> A new scaling law with an additive overfitting penalty reveals that beyond a compute threshold, repeating data becomes counterproductive, favoring larger models over more epochs.

- **Source:** [arXiv](https://arxiv.org/abs/2605.01640)
- **Published:** 2026-08-15
- **Permalink:** https://picx.dev/p/fZmmSS
- **Whiteboard:** https://picx.dev/p/fZmmSS/image

## Summary

## Summary (Overview)

- **New scaling law for data-constrained training**: The authors propose a simple additive overfitting penalty that augments the Chinchilla scaling law to model the cost of data repetition, addressing a critical gap in existing formulations.
- **Superlinear repetition damage**: Empirical analysis reveals that the excess loss from repeating data grows superlinearly with repetition count ($\delta > 1.0$), with steeper penalties for larger models and smaller unique data budgets.
- **Qualitatively new compute-optimal advice**: Beyond a data-dependent compute threshold, further repetition becomes counterproductive; compute is better spent on model capacity. This contradicts prior laws that never recommend stopping repetition.
- **Prescriptive validation**: The proposed law's recommended configurations achieve the best perplexity and downstream performance across all tested (token budget, compute budget) pairs, outperforming both Chinchilla and Muennighoff et al. (2023) formulations.
- **Weight decay case study**: Strong weight decay ($\lambda = 1.0$) reduces the overfitting coefficient $P$ by approximately 70%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.

## Introduction and Theoretical Foundation

### Background

Training compute is scaling faster than the supply of high-quality data. While raw text is abundant, aggressive quality filtering, upsampling of curated subsets, and mid-training on domain-specific corpora reflect a new reality: **data, not compute, is the bottleneck**. In specialized domains (mathematics, code, low-resource languages), domain-specific datasets are often orders of magnitude smaller than the compute budget can absorb.

### The Chinchilla Scaling Law

The widely adopted Chinchilla scaling law (Hoffmann et al., 2022) expresses the final loss of a trained model as:

$$
L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}, \tag{1}
$$

where:
- $N$ = number of model parameters
- $D$ = number of training tokens
- $E$ = irreducible loss floor (natural language unpredictability)
- $A/N^{\alpha}$ = cost of a model too small to represent patterns
- $B/D^{\beta}$ = cost of training on too little data

**Critical limitation**: The Chinchilla law assumes every training token is unique, which is violated in practice when models train for multiple epochs over the same data.

### Prior Work: Effective-Data Approach

Muennighoff et al. (2023) extended Chinchilla by replacing raw token count with an effective data quantity:

$$
\widehat{D}(U_D, R_D) = U_D \cdot \left(1 + R_D^* \cdot \left(1 - e^{-R_D / R_D^*}\right)\right), \tag{2}
$$

where $U_D$ = number of unique tokens and $R_D$ = additional epochs beyond the first. They also apply a similar saturating form to model parameters:

$$
\widehat{N}(U_N, R_N) = U_N + U_N \cdot R_N^* \cdot \left(1 - e^{-R_N / R_N^*}\right), \tag{4}
$$

**Key limitations of effective-data approach:**
1. Cannot represent the regime where loss *increases* from overfitting
2. Does not capture the interaction between overfitting and model size
3. Mechanism is indirect—models overparameterization as diminishing returns rather than explicit overfitting cost

## Methodology

### Experimental Setup

- **Architecture**: Decoder-only language models using Llama 2 architecture and tokenizer
- **Data**: FineWeb dataset (Penedo et al., 2024)
- **Grid**: Over 300 models spanning:
  - Model sizes: 15M–1B parameters
  - Unique data budgets: 50M–6B tokens
  - Repetition counts: $R_D \in \{0, 1, 3, 7, 11, 15\}$ (up to 16 epochs)
  - Two weight decay strengths: $\lambda = 0.1$ (standard) and $\lambda = 1.0$ (strong)
- **Evaluation**: Validation perplexity and OLMES (Open Language Model Evaluation System) average bits-per-byte (BPB) across 19 downstream tasks

### Key Insight: Additive Overfitting Penalty

The authors fit the Chinchilla law to single-epoch runs, then use this fit to predict multi-epoch loss by treating repeated tokens as fresh data ($D = U_D \cdot (1 + R_D)$). The residual between observed and predicted loss reveals a **power-law relationship** with repetition count.

### Complexity Ladder of Penalty Forms

**One-parameter form** (linear in $R_D$, dimensionless ratio $N/U_D$):

$$
L(N, U_D, R_D) = E + \frac{A}{N^{\alpha}} + \frac{B}{(U_D \cdot (1 + R_D))^{\beta}} + P \cdot R_D \cdot \frac{N}{U_D}. \tag{6}
$$

**Two-parameter form** (adds exponent $\kappa$ on capacity ratio):

$$
L(N, U_D, R_D) = E + \frac{A}{N^{\alpha}} + \frac{B}{(U_D \cdot (1 + R_D))^{\beta}} + P \cdot R_D \cdot \left(\frac{N}{U_D}\right)^{\kappa}. \tag{7}
$$

**Four-parameter form** (adds superlinear exponent $\delta$ on repetition, decouples data-budget exponent $\gamma$ from model-size exponent $\kappa$):

$$
L(N, U_D, R_D) = E + \frac{A}{N^{\alpha}} + \frac{B}{(U_D \cdot (1 + R_D))^{\beta}} + P \cdot R_D^{\delta} \cdot \left(\frac{N}{U_D^{\gamma}}\right)^{\kappa}. \tag{8}
$$

At $R_D = 0$ (single epoch), all forms reduce exactly to the Chinchilla law. The key conceptual difference: **repeated tokens play a dual role**—they continue to reduce the data-sufficiency term while simultaneously incurring a growing overfitting cost.

## Empirical Validation / Results

### Fit Quality

The proposed additive penalty laws were validated against two independent scaling sweeps:
1. The authors' own CLM sweep
2. The public Muennighoff et al. (2023) scaling sweep (up to 64 epochs)

**Key findings:**
- Even the one-parameter form (Equation 6) substantially outperforms both the $\widehat{D}(U_D, R_D)$ and $\widehat{D}(U_D, R_D), \widehat{N}(U_N, R_N)$ formulations
- The four-parameter form (Equation 8) achieves near-perfect fit on the authors' data
- Improvement generalizes to the held-out Muennighoff et al. (2023) data, confirming robustness across architectures (GPT-2 vs. Llama-2) and tokenization

### Prescriptive Validation

For each (token budget, compute budget) pair, the authors solve each law for the optimal model size and epoch count, train the recommended configuration, and evaluate:

**Table 1: Prescriptive validation results**

| $U_D$ | C | Scaling law | Params | Epochs | Perplexity ↓ | OLMES BPB ↓ |
|-------|---|-------------|--------|--------|--------------|-------------|
| 250M | $5 \times 10^{18}$ | Chinchilla | 280M | 12 | 25.31 | 1.52 |
| | | Eff. Param. | 500M | 7 | 23.91 | 1.50 |
| | | **Ours** | **700M** | **5** | **22.90** | **1.45** |
| 500M | $1 \times 10^{19}$ | Chinchilla | 390M | 8 | 18.95 | 1.35 |
| | | Eff. Param. | 550M | 6 | 18.65 | 1.35 |
| | | **Ours** | **700M** | **5** | **18.48** | **1.30** |
| 500M | $2 \times 10^{19}$ | Chinchilla | 670M | 10 | 18.90 | 1.37 |
| | | Eff. Param. | 950M | 7 | 19.34 | 1.40 |
| | | **Ours** | **2.2B** | **3** | **17.73** | **1.34** |

**Our law consistently recommends larger models with fewer epochs and achieves the best perplexity and downstream performance across all settings.**

### Compute-Optimal Allocation

The superlinear repetition penalty yields qualitatively different compute-optimal allocation advice:
- **Chinchilla law**: Always recommends more repetition (optimal total token count grows linearly with compute)
- **Muennighoff et al. (2023)**: Moderates this, prescribing diminishing returns but never recommending repetition stop
- **Our law**: Predicts a compute budget beyond which additional repetition is counterproductive—the allocation frontier turns back, recommending scaling model size while reducing epochs

### Weight Decay Case Study

**Table 2: Weight decay prescriptive validation**

| $U_D$ | C | Weight Decay | Params | Epochs | Perplexity ↓ | BPB ↓ |
|-------|---|-------------|--------|--------|--------------|-------|
| 250M ($C^\times \approx 3\times 10^{18}$) | $3\times 10^{18}$ | λ=0.1 | 350M | 6 | 23.38 | 1.46 |
| | | λ=1.0 | 350M | 6 | 22.93 | 1.47 |
| | $5\times 10^{18}$ | λ=0.1 | 700M | 5 | 22.91 | 1.47 |
| | | λ=1.0 | 550M | 6 | 21.66 | 1.41 |
| | $1\times 10^{19}$ | λ=0.1 | 3B | 2 | 23.13 | 1.52 |
| | | λ=1.0 | 1B | 6 | 20.34 | 1.36 |
| 500M ($C^\times \approx 1\times 10^{19}$) | $1\times 10^{19}$ | λ=0.1 | 700M | 5 | 18.52 | 1.36 |
| | | λ=1.0 | 830M | 4 | 18.75 | 1.32 |
| | $3\times 10^{19}$ | λ=0.1 | 5B | 2 | 18.16 | 1.35 |
| | | λ=1.0 | 2.5B | 4 | 16.65 | 1.30 |

**Key findings:**
- Strong weight decay ($\lambda = 1.0$) reduces the overfitting coefficient $P$ by **approximately 70%**
- Despite a single-epoch loss premium, strong weight decay crosses over at $C \approx 3.2 \times 10^{18}$ FLOPs (for $U_D = 250$M) and achieves lower absolute loss at higher compute budgets
- At $3\times$ past the crossover, strong weight decay reduces perplexity by 2.8 points

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Isolated overfitting coefficient**: The one-parameter form isolates overfitting in a single coefficient $P$, enabling direct comparison across training configurations (architectures, regularization strengths, data pipelines).

2. **Explicit overfitting mechanism**: Unlike the indirect effective-data approach, the additive penalty explicitly models overfitting as a separate cost that grows superlinearly with both repetition count and the model-to-data ratio.

3. **Pareto frontier of complexity**: The 1-, 2-, and 4-parameter forms trace a complexity-quality frontier, allowing practitioners to choose the appropriate trade-off between fit quality and simplicity.

### Practical Implications

1. **Compute allocation guidance**: Given a fixed data budget, there is a compute threshold beyond which repeating data further is counterproductive. Past this threshold, compute is better allocated to model capacity.

2. **Regularization as a lever**: Strong weight decay reduces the overfitting penalty by ~70%, providing two complementary levers for data-constrained practitioners: increasing regularization strength and choosing the right model size–epoch tradeoff.

3. **Replacing expensive hyperparameter sweeps**: The scaling law provides prescriptive guidance that can replace the expensive per-configuration hyperparameter sweeps required by prior empirical work (Kim et al., 2026).

### Limitations

- Study spans models up to 1B parameters and repetition up to 16 epochs; fitted exponents may not hold at frontier scales
- Law may not extrapolate to extreme repetition counts beyond the study
- Does not encapsulate phenomena such as double descent (Nakkiran et al., 2020)
- Only two weight decay values tested ($\lambda \in \{0.1, 1.0\}$); extending the law to incorporate regularization strength directly is future work

## Conclusion

The authors present a simple data-constrained scaling law that models the cost of data repetition with an additive overfitting penalty. Key takeaways:

1. **Even the simplest one-parameter form** substantially outperforms prior effective-data formulations, and the four-parameter form achieves near-perfect fit.

2. **The overfitting penalty provides a new axis for evaluating training configurations**, directly quantifying robustness to data repetition—increasingly important as compute grows faster than data.

3. **Strong weight decay ($\lambda = 1.0$) reduces the overfitting penalty by ~70%**, providing prescriptive guidance that explains, and can replace, expensive per-configuration hyperparameter sweeps.

4. **Qualitatively new compute-optimal allocation advice**: Beyond a threshold compute budget, further data repetition is counterproductive; practitioners should scale model capacity instead.

5. **Future directions**: Extending the law to incorporate regularization strength directly as a parameter, and validating at frontier scales beyond 1B parameters.

The findings provide concrete guidance for the increasingly common setting where high-quality data, not compute, is the binding constraint.

---

_Markdown view of https://picx.dev/p/fZmmSS, served by PicX — AI-generated visual whiteboard summaries of research papers._