A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling

Summary (Overview)

• Core Problem: The paper addresses the emerging regime where language model pretraining is limited by total available data rather than compute, requiring new scaling laws that account for data reuse (multiple training epochs) and ensembling.

• Key Empirical Finding: On FineWeb language-model experiments, logit averaging across independently trained models reduces late-training deterioration, delays optimal stopping time, and outperforms wider or deeper single models at matched parameter-token budgets.

• Theoretical Contribution: The authors derive a dynamical mean-field theory (DMFT) for multi-epoch stochastic gradient descent in power-law random-feature ensembles, tracking error correlations across epochs and between ensemble members.

• Scaling Laws: They derive compute-allocation laws at fixed data and model sizes, showing ensemble size grows with compute at least as fast as training time per member, with excess test loss decreasing as a power of compute.

• Optimal Stopping: The theory predicts that ensembling reduces prediction variance responsible for late-time growth in expected test error, fundamentally changing the optimal stopping time.

Introduction and Theoretical Foundation

Background and Motivation

Traditional scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020) assume one-pass training where model size and data grow together with compute. However, the supply of high-quality unique text may not grow at the same rate as compute (Kim et al., 2025; Muennighoff et al., 2023; Villalobos et al., 2024). When a corpus is reused, additional optimization has two opposing effects:

1. It continues to fit the available data
2. It amplifies noise tied to the particular dataset, model, or training order

This creates a non-monotone validation curve even as training loss keeps falling.

Theoretical Framework

The paper builds on the DMFT framework of Bordelon et al. (2024), extending it to fixed-order, multi-epoch SGD. The model considers an ensemble of $E$ random-feature models, each with $N$ trainable parameters and independently sampled feature matrices, trained on the same $P$ examples using fixed-order SGD.

The random-feature predictor is defined as:

$$f(x) = \frac{1}{N} w^\top A\psi(x), \quad y(x) = w_\star \cdot \psi(x) + \sigma\epsilon$$

where the training feature vectors are independent draws from $\mathcal{N}(0, \Lambda)$ with $\Lambda = \text{diag}(\lambda_1, \lambda_2, \ldots)$, $A \in \mathbb{R}^{N \times M}$ has independent standard Gaussian entries, and $\epsilon \sim \mathcal{N}(0,1)$.

Key Notation

Methodology

Multi-Epoch Dynamics

The discrepancy between target and learned feature coefficients is:

$$v^0_n(t) := w_\star - \frac{1}{N} A^\top w_n(t)$$

The exact residual recursion for fixed-order SGD is:

$$v^0_n(t+1) = v^0_n(t) - \eta \left(\frac{1}{N}A^\top A\right) \frac{1}{B} \Psi(t)^\top \left(\Psi(t) v^0_n(t) + \sigma\epsilon(t)\right)$$ $$v^0_{n+1}(0) = v^0_n(T)$$

DMFT Equations

The response equations characterize the system through:

$$H_k = [I + \eta\lambda_k \Theta R_3 R_1]^{-1}, \quad \bar{H} = \sum_k \lambda_k H_k$$ $$R_1 = \left(I + \frac{\eta}{B} D(\bar{H}\Theta R_3)\right)^{-1}, \quad R_3 = \left(I + \frac{\eta}{N} R_1 \bar{H}\Theta\right)^{-1}$$

The correlations and losses are given by:

$$[C_0]_{\alpha,\beta} = \sum_k \lambda_k \langle v^0_{k,\alpha} v^0_{k,\beta}\rangle, \quad [C_1]_{\alpha,\beta} = \langle v^1_\alpha v^1_\beta\rangle$$ $$L_{\text{test}}(\alpha) = [C_0]_{\alpha\alpha} + \sigma^2, \quad \hat{L}_{\text{train}}(\alpha) = [C_1]_{\alpha\alpha}$$

Power-Law Source/Capacity Conditions

The analysis assumes power-law spectra:

$$\lambda_k \sim k^{-b}, \quad \lambda_k(w^\star_k)^2 \sim k^{-a}, \quad a > 1, \; b > 1$$

where $b$ describes feature-spectrum decay and $a$ describes target power.

Empirical Validation / Results

Late-Time Growth Near Interpolation

At $N = P$ (the interpolation threshold), the growing contribution to a single model's test loss is:

$$L_{\text{grow}}(\tau) \asymp P^{-(a-1+b/2)} \tau^{1/2}, \quad \tau \gtrsim \tau_c \asymp P^b$$

This shows the expected test error grows as $\sqrt{\tau}$ at late times—the root cause of late-training overfitting.

Ensembling and Optimal Stopping

For an ensemble of $E$ models at $N = P$:

$$L(\tau, P, P, E) = C_{\text{cross}}(\tau) + \frac{C_{\text{self}}(\tau) - C_{\text{cross}}(\tau)}{E}$$

where $C_{\text{self}}$ is one member's expected squared error and $C_{\text{cross}}$ is the expected product of errors from two distinct members. The asymptotic expansions are:

$$C_{\text{cross}}(\tau) = C_\infty + J\tau^{-1/2} + D\tau^{-1} + O(\tau^{-3/2})$$ $$C_{\text{self}}(\tau) = H\sqrt{\tau} + O(1), \quad D, H > 0$$

Compute Allocation Results

Table 2: Optimal stopping and compute allocation at fixed N = P

Language-Model Experiments (FineWeb)

Key empirical results:

1. Validation loss follows a U-shaped trajectory while training loss continues to fall, with minima after roughly 9–14 epochs across tested corpus sizes.

2. Ensembling reduces late-time deterioration: Logit averaging lowers validation loss and reduces its late-time rise. An ensemble of four base models attains validation loss 0.16 nats/token lower than a single model with twice the width.

3. Larger models reach minima earlier: Optimal epoch count falls from 21 to 6 over the tested model-size range (11M–679M parameters).

4. Ensembles outperform larger single models: Four-member ensembles achieve validation losses 0.12–0.20 nats lower than corresponding wider or deeper single models at matched parameter-token budgets.

5. Ensembling delays optimal stopping: The optimal stopping epoch scales as $\tau^\star \propto E^{0.27}$ across model configurations.

Theoretical and Practical Implications

Theoretical Implications

1. Breakdown of Chinchilla-style scaling: The theory predicts compute-optimal scaling breaks down around compute scale $C \sim P^{\chi}$ where $\chi = b + 1$, with an "effectively online" regime at low compute and a "possible overfit" regime at higher compute.

2. Mechanistic understanding of ensembling: Ensembling reduces prediction variance across random projections by a factor of $1/E$, while the shared-data error (bias) remains. This changes the bias-variance tradeoff that determines optimal stopping.

3. Universal scaling laws: The excess loss above the limiting loss decreases as a power of compute: $(C/P^{b+1})^{-\gamma}$ where $\gamma \in \{1/4, 2/5, 1\}$ depending on the source-capacity relationship.

Practical Implications

1. Compute allocation strategy: When data is fixed, additional compute should be allocated to increasing ensemble size at least as fast as training time per member, rather than solely increasing model size.

2. Optimal stopping rules: Training should be stopped earlier for larger models and later for ensembles, with the optimal epoch count depending on both model size and ensemble size.

3. Architecture considerations: At approximately 170M parameters, a deeper, narrower model attains loss 0.06 nats/token below a shallower, wider model, suggesting architecture matters beyond parameter count.

Conclusion

This paper provides a comprehensive theoretical and empirical framework for understanding multi-epoch pretraining in the data-constrained regime. The key contributions are:

1. A multi-epoch DMFT that tracks error correlations within epochs, across epochs, and between ensemble members, with a finite-horizon loss-approximation bound for Gaussian features.

2. Ensemble-dependent stopping laws for noiseless Gaussian gradient flow at $N = P$, showing how ensembling changes optimal training time.

3. Compute-allocation laws under power-law source/capacity conditions, showing ensemble size should grow with compute at least as fast as training time per member.

The authors note that a quantitative connection to Transformer language models requires accounting for feature learning and optimizer dynamics, and that their label-noise analysis provides a starting point for extending these laws to noisy targets. Future work includes analyzing reshuffling effects and how changing data order affects cross-epoch correlations.

Main takeaway: When training on a fixed dataset, the number of training epochs and ensemble members should be chosen jointly, as ensembling suppresses the late-time error growth that makes longer training harmful, effectively converting a data-limited problem into one where additional compute can still yield substantial gains.