Summary (Overview)

• Proposes a new closed-form scaling law $L(N, D, T) = E + (L_0 - E) \cdot h/(1+h)$ that extends Chinchilla's law to the data-constrained, multi-epoch regime by decomposing loss into undercapacity, undertraining, and overfitting terms.
• Addresses three structural failures of Chinchilla: lack of baseline saturation (loss diverges as data → 0 instead of saturating at the uninformed baseline $L_0$), unreachable overfitting regime (loss strictly decreases in $N$ at fixed $D$), and conflation of total examples seen ($T$) with unique examples ($D$).
• Validated on 9 calibration grids spanning 4 architecture families (MLPs, ResNets, Fourier neural operators, transformers) across vision, scientific ML, and language domains, achieving state-of-the-art held-out RMSE on all 10 published LLM grid columns (10 of 10) and 6 of 8 constructed-experiment cells.
• Introduces a cost-aware allocation framework that converts a data-to-compute price ratio $\eta = \rho_D/\rho_C$ into an optimal allocation $(N^*, D^*, T^*)$, recovering Chinchilla's optimum when data is free and shifting toward smaller corpora, more epochs, and smaller models as data grows expensive.
• Reduces to Chinchilla's form in the data-rich, single-epoch limit, making it a strict refinement rather than a replacement.

Introduction and Theoretical Foundation

Background and Motivation

The dominant scaling law in modern ML, Chinchilla's form, was calibrated in a regime where data is abundant and training is single-epoch:

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \tag{1}$$

However, many real-world settings face a fundamentally different tradeoff: data is expensive (physics surrogates requiring PDE solvers, expert-labeled medical imaging, RL transitions, LLM finetuning), and practitioners must decide between acquiring more unique data or spending compute to revisit existing data.

Three Structural Failures of Chinchilla

1. No baseline saturation: Loss should be bounded above by $L_0$ (the loss of an uninformed model). Chinchilla has $L \to \infty$ as $D \to 0$ instead of saturating at $L_0$.

2. Overfitting unreachable: At fixed $D$, the form is strictly decreasing in $N$, but neural networks with $N \gg D$ overfit—validation loss can rise with capacity past a point.

3. No compute axis independent of $(N, D)$: The form cannot distinguish between total examples seen $T$ and unique examples $D$, precluding analysis of multi-epoch training.

Target Limiting Behaviors

The paper enumerates six limiting behaviors that any form in $(N, D, T)$ should exhibit (Table 1):

Methodology

The Proposed Form

$$L(N, D, T) = E + (L_0 - E) \cdot \frac{h(N, D, T)}{1 + h(N, D, T)}, \qquad h = \frac{a}{N^{\alpha}} + \frac{b}{T^{\beta}} + c\frac{N^{\gamma}}{D^{\delta}} \tag{2}$$

Notation: $N$ = parameters, $D$ = unique training examples, $T$ = total training examples seen (counted with repetition).

Three Decomposed Terms

The difficulty $h$ is a sum of power laws representing three distinct underperformance phenomena:

$$h(N, D, T) = \underbrace{\frac{a}{N^{\alpha}}}_{\text{undercapacity}} + \underbrace{\frac{b}{T^{\beta}}}_{\text{undertraining}} + \underbrace{c\frac{N^{\gamma}}{D^{\delta}}}_{\text{overfitting}} \tag{3}$$

• Undercapacity ($a/N^{\alpha}$): The only term that doesn't vanish as $D$ and $T$ grow. Captures the residual cost of finite model capacity.
• Undertraining ($b/T^{\beta}$): Decays with total examples seen. At $T = D$, collapses to Chinchilla's data term $b/D^{\beta}$.
• Overfitting ($cN^{\gamma}/D^{\delta}$): Depends only on capacity $N$ and unique-data count $D$. The two-exponent shape allows independent calibration of $D$-scaling and $N$-scaling.

The Saturating Wrapper

The wrapper $h/(1+h)$ is a monotone bijection from $[0, \infty)$ to $[0, 1)$ with two key properties:

1. Small-h linearity: For $h \ll 1$, $h/(1+h) = h - h^2 + h^3 - \cdots$, so Eq. (2) reduces at leading order to $L \approx E + (L_0 - E)h$—recovering Chinchilla-like behavior.
2. Large-h saturation: For $h \to \infty$, $h/(1+h) \to 1$ and $L \to L_0$.

Baseline Loss $L_0$

$L_0$ is determined by the loss type, not fitted:

• Cross-entropy over $K$ outcomes: $L_0 = \ln K$ (e.g., $K = V$ for next-token prediction)
• Relative-$L_2$ regression on z-normalized targets: $L_0 = 1$ exactly

Experimental Setup

Datasets: MNIST (MLPs), CIFAR-100 (PreActResNets), PDEBench Darcyflow (Fourier Neural Operators), TinyStories (decoder-only transformers), plus 5 published LLM grids (Chinchilla, Muennighoff, Gadre, Porian, Farseer).

Baselines compared: Chinchilla, Muennighoff, M4, BNSL, Farseer, plus 4 ablations of the proposed form.

Fitting: Huber loss on log-residuals with BFGS, multistart random initialization, bootstrap resampling for confidence intervals.

Extrapolation protocols: high-C holdout (predicting expensive runs from cheap ones) and high-D holdout (predicting large-D performance from small-D data), each targeting ~10% of rows held out groupwise.

Empirical Validation / Results

Key Results from Table 2 (held-out RMSE in log space)

high-C holdout (selected columns):

high-D holdout (selected columns):

Headline results:

• State-of-the-art on all 10 external LLM grid columns (both holdout protocols across 5 published grids), averaging 49% lower RMSE than the second-best approach.
• Wins 6 of 8 cells in constructed experiments; underperforms only on high-C TinyStories and high-D MNIST.
• Key insight: Wins extend beyond the multi-epoch, data-constrained regime—4 of 5 published LLM grids are single-epoch and data-unconstrained, yet the form wins every column.

Compute Footprint

• ~206 GPU-hours total on AWS Batch (NVIDIA A10G): MNIST ~46, CIFAR-100 ~14, Darcy ~48, TinyStories ~98.
• Aggregate training compute: ~$6.0 \times 10^{18}$ FLOPs.

Cost-Aware Allocation Results

The Lagrange first-order condition for budget-constrained optimization:

$$\frac{\partial L/\partial N}{\rho_C k T} = \frac{\partial L/\partial D}{\rho_D} = \frac{\partial L/\partial T}{\rho_C k N} \tag{4}$$

Illustrative example (Table 3) at fixed budget with Chinchilla exponents $(\alpha, \beta) = (0.34, 0.28)$:

Increasing $\eta$ by 1000× shrinks $D^*$ by two orders of magnitude, shrinks $N^*$ by ~20×, and walks $T^*/D^*$ from 1 to ~1,250 epochs.

Theoretical and Practical Implications

Theoretical Contributions

1. First explicit closed-form overfitting term in parametric scaling laws: the $cN^{\gamma}/D^{\delta}$ term wrapped in $h/(1+h)$ has no analog in prior work.

2. Structural refinement of Chinchilla: The form reduces to Chinchilla in the small-$h$ single-epoch limit, making it a strict generalization rather than an alternative formulation.

3. U-shape behavior: At fixed $D$, the interplay between overfitting ($cN^{\gamma}/D^{\delta}$) and undercapacity ($a/N^{\alpha}$) produces an interior minimum in $L(N)$ as $T \to \infty$, implying that for finite data, there's a model size beyond which best achievable performance degrades—a structural feature absent from other parametric forms.

Practical Implications

1. Cost-aware allocation: A calibrated $L(N, D, T)$ converts a single price ratio $\eta = \rho_D/\rho_C$ into a concrete cost-optimal allocation $(N^*, D^*, T^*)$—the data-free limit recovers Chinchilla compute-optimal, and as data grows expensive, the allocation shifts to smaller corpora, more epochs, and smaller models.

2. Broad applicability: The form works across vision, scientific ML, and language domains, spanning four architecture families, demonstrating it's not architecture-specific.

3. Extrapolation reliability: State-of-the-art extrapolation to higher compute and larger unique data than seen at fit time—the primary practical use of a scaling law.

Conclusion

Main Takeaways

The paper introduces a closed-form scaling law $L(N, D, T) = E + (L_0 - E)h/(1+h)$ that:

• Satisfies six target limiting behaviors, five of which Chinchilla violates
• Reduces to Chinchilla in the data-rich single-epoch limit
• Includes an explicit closed-form overfitting term with no analog in prior parametric scaling laws
• Wins 16 of 18 high-C and high-D extrapolation cells across nine calibration grids

Future Directions

1. One-sided log-space prior on $E$: Remediates optimistic irreducible-loss estimates in far-compute extrapolation (recovers SOTA on Farseer and TinyStories high-C); broader applicability across forms and datasets is future work.

2. Structural fix for low-D regime: The cross term $cN^{\gamma}/D^{\delta}$ over-extrapolates from small-D extreme overfitting regimes; a structural fix keeping the term well-behaved across the full $D$ range is future work.

3. Beyond in-distribution validation loss: Downstream task performance, task-level metrics, emergent capabilities, and out-of-distribution behavior are separate layers in the scaling-law stack and out of scope.

4. Double descent: The form is asymptotic and smooth, so it doesn't predict the peak at $N \approx D$; capturing this structure is left to future work.

Key Limitations

• In-distribution validation loss only: Doesn't address downstream task scaling or OOD behavior.
• Universality of form, not of fits: Fitted constants are recipe-dependent and not portable across settings.
• Optimistic $E$ in far compute extrapolation: The saturating wrapper makes $E$ identified only when observations approach the asymptote.