# Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World

> This paper introduces a closed-form scaling law that extends Chinchilla to data-constrained, multi-epoch regimes, achieving state-of-the-art extrapolation across vision, scientific, and language domains.

- **Source:** [arXiv](https://arxiv.org/abs/2605.09189)
- **Published:** 2026-08-22
- **Permalink:** https://picx.dev/p/IsG1aY
- **Whiteboard:** https://picx.dev/p/IsG1aY/image

## Summary

## Summary (Overview)

- **Proposes a new closed-form scaling law** $L(N, D, T) = E + (L_0 - E) \cdot h/(1+h)$ that extends Chinchilla's law to the data-constrained, multi-epoch regime by decomposing loss into undercapacity, undertraining, and overfitting terms.
- **Addresses three structural failures of Chinchilla**: lack of baseline saturation (loss diverges as data → 0 instead of saturating at the uninformed baseline $L_0$), unreachable overfitting regime (loss strictly decreases in $N$ at fixed $D$), and conflation of total examples seen ($T$) with unique examples ($D$).
- **Validated on 9 calibration grids** spanning 4 architecture families (MLPs, ResNets, Fourier neural operators, transformers) across vision, scientific ML, and language domains, achieving state-of-the-art held-out RMSE on all 10 published LLM grid columns (10 of 10) and 6 of 8 constructed-experiment cells.
- **Introduces a cost-aware allocation framework** that converts a data-to-compute price ratio $\eta = \rho_D/\rho_C$ into an optimal allocation $(N^*, D^*, T^*)$, recovering Chinchilla's optimum when data is free and shifting toward smaller corpora, more epochs, and smaller models as data grows expensive.
- **Reduces to Chinchilla's form in the data-rich, single-epoch limit**, making it a strict refinement rather than a replacement.

## Introduction and Theoretical Foundation

### Background and Motivation

The dominant scaling law in modern ML, Chinchilla's form, was calibrated in a regime where data is abundant and training is single-epoch:

$$
L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \tag{1}
$$

However, many real-world settings face a fundamentally different tradeoff: data is expensive (physics surrogates requiring PDE solvers, expert-labeled medical imaging, RL transitions, LLM finetuning), and practitioners must decide between acquiring more unique data or spending compute to revisit existing data.

### Three Structural Failures of Chinchilla

1. **No baseline saturation**: Loss should be bounded above by $L_0$ (the loss of an uninformed model). Chinchilla has $L \to \infty$ as $D \to 0$ instead of saturating at $L_0$.

2. **Overfitting unreachable**: At fixed $D$, the form is strictly decreasing in $N$, but neural networks with $N \gg D$ overfit—validation loss can rise with capacity past a point.

3. **No compute axis independent of $(N, D)$**: The form cannot distinguish between total examples seen $T$ and unique examples $D$, precluding analysis of multi-epoch training.

### Target Limiting Behaviors

The paper enumerates six limiting behaviors that any form in $(N, D, T)$ should exhibit (Table 1):

| # | Limit | Expected L | Chinchilla |
|---|-------|-----------|------------|
| 1 | $N \to 0$; $D$, $T$ finite | $L_0$ | fails ($L \to \infty$) |
| 2 | $D \to 0$; $N$, $T$ finite | $L_0$ | fails ($L \to \infty$) |
| 3 | $T \to 0$; $N$, $D$ finite | $L_0$ | fails |
| 4 | $N \to \infty$; $T$, $D$ finite | $L_0$ | fails |
| 5 | $N, T \to \infty$; $D$ finite | $L_0$ | fails (unreachable) |
| 6 | $N, D, T \to \infty$ jointly | $E$ | pass |

## Methodology

### The Proposed Form

$$
L(N, D, T) = E + (L_0 - E) \cdot \frac{h(N, D, T)}{1 + h(N, D, T)}, \qquad h = \frac{a}{N^{\alpha}} + \frac{b}{T^{\beta}} + c\frac{N^{\gamma}}{D^{\delta}} \tag{2}
$$

**Notation**: $N$ = parameters, $D$ = unique training examples, $T$ = total training examples seen (counted with repetition).

### Three Decomposed Terms

The difficulty $h$ is a sum of power laws representing three distinct underperformance phenomena:

$$
h(N, D, T) = \underbrace{\frac{a}{N^{\alpha}}}_{\text{undercapacity}} + \underbrace{\frac{b}{T^{\beta}}}_{\text{undertraining}} + \underbrace{c\frac{N^{\gamma}}{D^{\delta}}}_{\text{overfitting}} \tag{3}
$$

- **Undercapacity** ($a/N^{\alpha}$): The only term that doesn't vanish as $D$ and $T$ grow. Captures the residual cost of finite model capacity.
- **Undertraining** ($b/T^{\beta}$): Decays with total examples seen. At $T = D$, collapses to Chinchilla's data term $b/D^{\beta}$.
- **Overfitting** ($cN^{\gamma}/D^{\delta}$): Depends only on capacity $N$ and unique-data count $D$. The two-exponent shape allows independent calibration of $D$-scaling and $N$-scaling.

### The Saturating Wrapper

The wrapper $h/(1+h)$ is a monotone bijection from $[0, \infty)$ to $[0, 1)$ with two key properties:

1. **Small-h linearity**: For $h \ll 1$, $h/(1+h) = h - h^2 + h^3 - \cdots$, so Eq. (2) reduces at leading order to $L \approx E + (L_0 - E)h$—recovering Chinchilla-like behavior.
2. **Large-h saturation**: For $h \to \infty$, $h/(1+h) \to 1$ and $L \to L_0$.

### Baseline Loss $L_0$

$L_0$ is determined by the loss type, not fitted:
- Cross-entropy over $K$ outcomes: $L_0 = \ln K$ (e.g., $K = V$ for next-token prediction)
- Relative-$L_2$ regression on z-normalized targets: $L_0 = 1$ exactly

### Experimental Setup

**Datasets**: MNIST (MLPs), CIFAR-100 (PreActResNets), PDEBench Darcyflow (Fourier Neural Operators), TinyStories (decoder-only transformers), plus 5 published LLM grids (Chinchilla, Muennighoff, Gadre, Porian, Farseer).

**Baselines compared**: Chinchilla, Muennighoff, M4, BNSL, Farseer, plus 4 ablations of the proposed form.

**Fitting**: Huber loss on log-residuals with BFGS, multistart random initialization, bootstrap resampling for confidence intervals.

**Extrapolation protocols**: high-C holdout (predicting expensive runs from cheap ones) and high-D holdout (predicting large-D performance from small-D data), each targeting ~10% of rows held out groupwise.

## Empirical Validation / Results

### Key Results from Table 2 (held-out RMSE in log space)

**high-C holdout** (selected columns):

| Form | MNIST | CIFAR-100 | TinyStories | Chinchilla | Farseer |
|------|-------|-----------|-------------|------------|---------|
| Chinchilla | 0.332 | 0.156 | 0.173 | 0.024 | 0.028 |
| Muennighoff | 0.333 | 0.151 | **0.095** | 0.024 | 0.028 |
| Farseer | 1.389 | 0.607 | 0.119 | 0.030 | 0.020 |
| **Ours (Eq. 2)** | **0.127** | **0.081** | 0.184 | **0.007** | **0.008** |

**high-D holdout** (selected columns):

| Form | MNIST | CIFAR-100 | TinyStories | Chinchilla | Farseer |
|------|-------|-----------|-------------|------------|---------|
| Chinchilla | 0.123 | 0.182 | 0.057 | 0.028 | 0.017 |
| Muennighoff | **0.122** | 0.171 | 0.095 | 0.028 | 0.017 |
| Farseer | 1.490 | 0.899 | 0.060 | 0.012 | 0.041 |
| **Ours (Eq. 2)** | 0.137 | **0.069** | **0.053** | **0.010** | **0.005** |

**Headline results**:
- **State-of-the-art on all 10 external LLM grid columns** (both holdout protocols across 5 published grids), averaging **49% lower RMSE** than the second-best approach.
- **Wins 6 of 8 cells** in constructed experiments; underperforms only on high-C TinyStories and high-D MNIST.
- **Key insight**: Wins extend beyond the multi-epoch, data-constrained regime—4 of 5 published LLM grids are single-epoch and data-unconstrained, yet the form wins every column.

### Compute Footprint

- ~206 GPU-hours total on AWS Batch (NVIDIA A10G): MNIST ~46, CIFAR-100 ~14, Darcy ~48, TinyStories ~98.
- Aggregate training compute: ~$6.0 \times 10^{18}$ FLOPs.

### Cost-Aware Allocation Results

The Lagrange first-order condition for budget-constrained optimization:

$$
\frac{\partial L/\partial N}{\rho_C k T} = \frac{\partial L/\partial D}{\rho_D} = \frac{\partial L/\partial T}{\rho_C k N} \tag{4}
$$

**Illustrative example** (Table 3) at fixed budget with Chinchilla exponents $(\alpha, \beta) = (0.34, 0.28)$:

| $\eta$ | $N^*$ | $D^*$ | $T^*/D^*$ | $L^*$ | data share |
|--------|-------|-------|-----------|-------|------------|
| 0 (Chinchilla CO) | $5.2 \times 10^9$ | $3.2 \times 10^{11}$ | 1.0 | 2.14 | 0% |
| $10^{10}$ (web tokens) | $4.4 \times 10^9$ | $1.2 \times 10^{11}$ | 2.7 | 2.14 | 12% |
| $10^{12}$ (licensed corpora) | $1.2 \times 10^9$ | $6.3 \times 10^9$ | 80 | 2.30 | 63% |
| $10^{13}$ (expert/sim) | $2.1 \times 10^8$ | $8.6 \times 10^8$ | 1,250 | 2.66 | 86% |

Increasing $\eta$ by 1000× shrinks $D^*$ by two orders of magnitude, shrinks $N^*$ by ~20×, and walks $T^*/D^*$ from 1 to ~1,250 epochs.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **First explicit closed-form overfitting term** in parametric scaling laws: the $cN^{\gamma}/D^{\delta}$ term wrapped in $h/(1+h)$ has no analog in prior work.

2. **Structural refinement of Chinchilla**: The form reduces to Chinchilla in the small-$h$ single-epoch limit, making it a strict generalization rather than an alternative formulation.

3. **U-shape behavior**: At fixed $D$, the interplay between overfitting ($cN^{\gamma}/D^{\delta}$) and undercapacity ($a/N^{\alpha}$) produces an interior minimum in $L(N)$ as $T \to \infty$, implying that for finite data, there's a model size beyond which best achievable performance degrades—a structural feature absent from other parametric forms.

### Practical Implications

1. **Cost-aware allocation**: A calibrated $L(N, D, T)$ converts a single price ratio $\eta = \rho_D/\rho_C$ into a concrete cost-optimal allocation $(N^*, D^*, T^*)$—the data-free limit recovers Chinchilla compute-optimal, and as data grows expensive, the allocation shifts to smaller corpora, more epochs, and smaller models.

2. **Broad applicability**: The form works across vision, scientific ML, and language domains, spanning four architecture families, demonstrating it's not architecture-specific.

3. **Extrapolation reliability**: State-of-the-art extrapolation to higher compute and larger unique data than seen at fit time—the primary practical use of a scaling law.

## Conclusion

### Main Takeaways

The paper introduces a closed-form scaling law $L(N, D, T) = E + (L_0 - E)h/(1+h)$ that:
- Satisfies six target limiting behaviors, five of which Chinchilla violates
- Reduces to Chinchilla in the data-rich single-epoch limit
- Includes an explicit closed-form overfitting term with no analog in prior parametric scaling laws
- Wins 16 of 18 high-C and high-D extrapolation cells across nine calibration grids

### Future Directions

1. **One-sided log-space prior on $E$**: Remediates optimistic irreducible-loss estimates in far-compute extrapolation (recovers SOTA on Farseer and TinyStories high-C); broader applicability across forms and datasets is future work.

2. **Structural fix for low-D regime**: The cross term $cN^{\gamma}/D^{\delta}$ over-extrapolates from small-D extreme overfitting regimes; a structural fix keeping the term well-behaved across the full $D$ range is future work.

3. **Beyond in-distribution validation loss**: Downstream task performance, task-level metrics, emergent capabilities, and out-of-distribution behavior are separate layers in the scaling-law stack and out of scope.

4. **Double descent**: The form is asymptotic and smooth, so it doesn't predict the peak at $N \approx D$; capturing this structure is left to future work.

### Key Limitations

- **In-distribution validation loss only**: Doesn't address downstream task scaling or OOD behavior.
- **Universality of form, not of fits**: Fitted constants are recipe-dependent and not portable across settings.
- **Optimistic $E$ in far compute extrapolation**: The saturating wrapper makes $E$ identified only when observations approach the asymptote.

---

_Markdown view of https://picx.dev/p/IsG1aY, served by PicX — AI-generated visual whiteboard summaries of research papers._