Summary (Overview)

• New scaling law for data-constrained training: The authors propose a simple additive overfitting penalty that augments the Chinchilla scaling law to model the cost of data repetition, addressing a critical gap in existing formulations.
• Superlinear repetition damage: Empirical analysis reveals that the excess loss from repeating data grows superlinearly with repetition count ($\delta > 1.0$), with steeper penalties for larger models and smaller unique data budgets.
• Qualitatively new compute-optimal advice: Beyond a data-dependent compute threshold, further repetition becomes counterproductive; compute is better spent on model capacity. This contradicts prior laws that never recommend stopping repetition.
• Prescriptive validation: The proposed law's recommended configurations achieve the best perplexity and downstream performance across all tested (token budget, compute budget) pairs, outperforming both Chinchilla and Muennighoff et al. (2023) formulations.
• Weight decay case study: Strong weight decay ($\lambda = 1.0$) reduces the overfitting coefficient $P$ by approximately 70%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.

Introduction and Theoretical Foundation

Background

Training compute is scaling faster than the supply of high-quality data. While raw text is abundant, aggressive quality filtering, upsampling of curated subsets, and mid-training on domain-specific corpora reflect a new reality: data, not compute, is the bottleneck. In specialized domains (mathematics, code, low-resource languages), domain-specific datasets are often orders of magnitude smaller than the compute budget can absorb.

The Chinchilla Scaling Law

The widely adopted Chinchilla scaling law (Hoffmann et al., 2022) expresses the final loss of a trained model as:

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}, \tag{1}$$

where:

• $N$ = number of model parameters
• $D$ = number of training tokens
• $E$ = irreducible loss floor (natural language unpredictability)
• $A/N^{\alpha}$ = cost of a model too small to represent patterns
• $B/D^{\beta}$ = cost of training on too little data

Critical limitation: The Chinchilla law assumes every training token is unique, which is violated in practice when models train for multiple epochs over the same data.

Prior Work: Effective-Data Approach

Muennighoff et al. (2023) extended Chinchilla by replacing raw token count with an effective data quantity:

$$\widehat{D}(U_D, R_D) = U_D \cdot \left(1 + R_D^* \cdot \left(1 - e^{-R_D / R_D^*}\right)\right), \tag{2}$$

where $U_D$ = number of unique tokens and $R_D$ = additional epochs beyond the first. They also apply a similar saturating form to model parameters:

$$\widehat{N}(U_N, R_N) = U_N + U_N \cdot R_N^* \cdot \left(1 - e^{-R_N / R_N^*}\right), \tag{4}$$

Key limitations of effective-data approach:

1. Cannot represent the regime where loss increases from overfitting
2. Does not capture the interaction between overfitting and model size
3. Mechanism is indirect—models overparameterization as diminishing returns rather than explicit overfitting cost

Methodology

Experimental Setup

• Architecture: Decoder-only language models using Llama 2 architecture and tokenizer
• Data: FineWeb dataset (Penedo et al., 2024)
• Grid: Over 300 models spanning:
  • Model sizes: 15M–1B parameters
  • Unique data budgets: 50M–6B tokens
  • Repetition counts: $R_D \in \{0, 1, 3, 7, 11, 15\}$ (up to 16 epochs)
  • Two weight decay strengths: $\lambda = 0.1$ (standard) and $\lambda = 1.0$ (strong)
• Evaluation: Validation perplexity and OLMES (Open Language Model Evaluation System) average bits-per-byte (BPB) across 19 downstream tasks

Key Insight: Additive Overfitting Penalty

The authors fit the Chinchilla law to single-epoch runs, then use this fit to predict multi-epoch loss by treating repeated tokens as fresh data ($D = U_D \cdot (1 + R_D)$). The residual between observed and predicted loss reveals a power-law relationship with repetition count.

Complexity Ladder of Penalty Forms

One-parameter form (linear in $R_D$, dimensionless ratio $N/U_D$):

$$L(N, U_D, R_D) = E + \frac{A}{N^{\alpha}} + \frac{B}{(U_D \cdot (1 + R_D))^{\beta}} + P \cdot R_D \cdot \frac{N}{U_D}. \tag{6}$$

Two-parameter form (adds exponent $\kappa$ on capacity ratio):

$$L(N, U_D, R_D) = E + \frac{A}{N^{\alpha}} + \frac{B}{(U_D \cdot (1 + R_D))^{\beta}} + P \cdot R_D \cdot \left(\frac{N}{U_D}\right)^{\kappa}. \tag{7}$$

Four-parameter form (adds superlinear exponent $\delta$ on repetition, decouples data-budget exponent $\gamma$ from model-size exponent $\kappa$):

$$L(N, U_D, R_D) = E + \frac{A}{N^{\alpha}} + \frac{B}{(U_D \cdot (1 + R_D))^{\beta}} + P \cdot R_D^{\delta} \cdot \left(\frac{N}{U_D^{\gamma}}\right)^{\kappa}. \tag{8}$$

At $R_D = 0$ (single epoch), all forms reduce exactly to the Chinchilla law. The key conceptual difference: repeated tokens play a dual role—they continue to reduce the data-sufficiency term while simultaneously incurring a growing overfitting cost.

Empirical Validation / Results

Fit Quality

The proposed additive penalty laws were validated against two independent scaling sweeps:

1. The authors' own CLM sweep
2. The public Muennighoff et al. (2023) scaling sweep (up to 64 epochs)

Key findings:

• Even the one-parameter form (Equation 6) substantially outperforms both the $\widehat{D}(U_D, R_D)$ and $\widehat{D}(U_D, R_D), \widehat{N}(U_N, R_N)$ formulations
• The four-parameter form (Equation 8) achieves near-perfect fit on the authors' data
• Improvement generalizes to the held-out Muennighoff et al. (2023) data, confirming robustness across architectures (GPT-2 vs. Llama-2) and tokenization

Prescriptive Validation

For each (token budget, compute budget) pair, the authors solve each law for the optimal model size and epoch count, train the recommended configuration, and evaluate:

Table 1: Prescriptive validation results

Our law consistently recommends larger models with fewer epochs and achieves the best perplexity and downstream performance across all settings.

Compute-Optimal Allocation

The superlinear repetition penalty yields qualitatively different compute-optimal allocation advice:

• Chinchilla law: Always recommends more repetition (optimal total token count grows linearly with compute)
• Muennighoff et al. (2023): Moderates this, prescribing diminishing returns but never recommending repetition stop
• Our law: Predicts a compute budget beyond which additional repetition is counterproductive—the allocation frontier turns back, recommending scaling model size while reducing epochs

Weight Decay Case Study

Table 2: Weight decay prescriptive validation

Key findings:

• Strong weight decay ($\lambda = 1.0$) reduces the overfitting coefficient $P$ by approximately 70%
• Despite a single-epoch loss premium, strong weight decay crosses over at $C \approx 3.2 \times 10^{18}$ FLOPs (for $U_D = 250$M) and achieves lower absolute loss at higher compute budgets
• At $3\times$ past the crossover, strong weight decay reduces perplexity by 2.8 points

Theoretical and Practical Implications

Theoretical Contributions

1. Isolated overfitting coefficient: The one-parameter form isolates overfitting in a single coefficient $P$, enabling direct comparison across training configurations (architectures, regularization strengths, data pipelines).

2. Explicit overfitting mechanism: Unlike the indirect effective-data approach, the additive penalty explicitly models overfitting as a separate cost that grows superlinearly with both repetition count and the model-to-data ratio.

3. Pareto frontier of complexity: The 1-, 2-, and 4-parameter forms trace a complexity-quality frontier, allowing practitioners to choose the appropriate trade-off between fit quality and simplicity.

Practical Implications

1. Compute allocation guidance: Given a fixed data budget, there is a compute threshold beyond which repeating data further is counterproductive. Past this threshold, compute is better allocated to model capacity.

2. Regularization as a lever: Strong weight decay reduces the overfitting penalty by ~70%, providing two complementary levers for data-constrained practitioners: increasing regularization strength and choosing the right model size–epoch tradeoff.

3. Replacing expensive hyperparameter sweeps: The scaling law provides prescriptive guidance that can replace the expensive per-configuration hyperparameter sweeps required by prior empirical work (Kim et al., 2026).

Limitations

• Study spans models up to 1B parameters and repetition up to 16 epochs; fitted exponents may not hold at frontier scales
• Law may not extrapolate to extreme repetition counts beyond the study
• Does not encapsulate phenomena such as double descent (Nakkiran et al., 2020)
• Only two weight decay values tested ($\lambda \in \{0.1, 1.0\}$); extending the law to incorporate regularization strength directly is future work

Conclusion

The authors present a simple data-constrained scaling law that models the cost of data repetition with an additive overfitting penalty. Key takeaways:

1. Even the simplest one-parameter form substantially outperforms prior effective-data formulations, and the four-parameter form achieves near-perfect fit.

2. The overfitting penalty provides a new axis for evaluating training configurations, directly quantifying robustness to data repetition—increasingly important as compute grows faster than data.

3. Strong weight decay ($\lambda = 1.0$) reduces the overfitting penalty by ~70%, providing prescriptive guidance that explains, and can replace, expensive per-configuration hyperparameter sweeps.

4. Qualitatively new compute-optimal allocation advice: Beyond a threshold compute budget, further data repetition is counterproductive; practitioners should scale model capacity instead.

5. Future directions: Extending the law to incorporate regularization strength directly as a parameter, and validating at frontier scales beyond 1B parameters.

The findings provide concrete guidance for the increasingly common setting where high-quality data, not compute, is the binding constraint.