Summary of "Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling"

Summary (Overview)

• Core problem: Existing data-constrained scaling prescriptions suffer from two critical limitations: (1) they hold hyperparameters (like weight decay) fixed across scales, despite optimal values varying wildly with model size and epochs; (2) they neglect ensembling as a scaling axis, leading to prescriptions where loss can increase with compute budget.
• Key contributions: The authors (1) fit empirical power laws for hyperparameter transfer across scale, and (2) propose a new scaling law that models ensembling as a third axis (alongside model size and epochs) for compute-optimal allocation under data constraints.
• Main result: Training on 100M tokens from FineWeb, their recipe reduces validation loss by over 0.15 nats compared to prior recipes (a 14% decrease in perplexity), achieving monotonic improvements with compute through ensembling.
• Validation: Gains are achieved using prescriptions fit to smaller runs and validated at up to 12× larger compute budgets, demonstrating robust extrapolation.
• Key insight: The compute-optimal allocation shifts from growing model size and epochs to growing ensemble size once model size and epochs reach overfitting-prone levels.

Introduction and Theoretical Foundation

Background

The paper situates itself across three training regimes:

1. Traditional regime: Small datasets, where regularization and ensembling were essential (Lakshminarayanan et al., 2017; Loshchilov & Hutter, 2019).
2. Current regime: Internet-scale data, where scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) prescribe compute-optimal configurations assuming single-epoch training and effectively infinite data.
3. Emerging data-constrained regime: Compute budgets grow faster than high-quality data supply (Villalobos et al., 2024 predict exhaustion of public text by 2026), necessitating multi-epoch training and renewed attention to regularization and ensembling.

Theoretical Foundation: Scaling Laws

The Chinchilla scaling law (Hoffmann et al., 2022) forms the baseline:

$$L(U, N) = E + \frac{A}{N^{\alpha}} + \frac{B}{U^{\beta}} \tag{1}$$

where $L$ is validation loss, $N$ is model size (non-embedding parameters), $U$ is dataset size in tokens, and $\{E, A, \alpha, B, \beta\}$ are free parameters. Compute is approximated as $C \approx 6UN$.

Data-constrained extension (Lovelace et al., 2026) explicitly models overfitting from multi-epoch training:

$$L(U, N, R) = E + \frac{A}{N^{\alpha}} + \frac{B}{(UR)^{\beta}} + P(R-1)^{\delta}\left(\frac{N}{U^{\gamma}}\right)^{\kappa} \tag{2}$$

with four new parameters $\{P, \delta, \kappa, \gamma\}$. When $R=1$, the last term vanishes, recovering the Chinchilla law.

The Pathology

The authors identify a critical failure: existing recipes (e.g., Lovelace et al., 2026) predict that compute-optimal loss increases past mild compute budgets (~$10^{18}$–$10^{19}$ FLOPs), rather than decreasing monotonically. This pathology arises from:

• Fixed hyperparameters across scales (suboptimal at larger scales)
• No consideration of ensembling as an alternative compute allocation

Methodology

Experimental Setup

• Models: Decoder-only transformers on FineWeb (Penedo et al., 2024) with GPT-2 tokenization.
• Grid: Unique tokens $U \in \{25M, 50M, 100M, 200M, 400M\}$, model sizes $N = 30M$–$398M$ parameters, epochs $R = 2$–$64$ (some runs at 128 and 256 epochs).
• Optimizer: Muon (instead of AdamW), reflecting a more competitive data-constrained setup inspired by the Slowrun competition (Vegesna et al., 2026).

Hyperparameter Scaling Laws (Section 4)

Sweep strategy: Two-dimensional logarithmic grid over weight decay $\lambda$ and effective learning rate $\text{eLR} = \eta \lambda$ (coupled weight decay), with $\sqrt{2}$-spaced lattice across 110 $(U, N, R)$ rungs spanning ~3 orders of magnitude of compute.

Localization: For each rung, fit a least-squares quadratic $L(\ln \lambda, \ln \text{eLR})$ to estimate optimal hyperparameters. Key observations:

• $\eta$ is most sensitive at low $R$; $\lambda$ is most sensitive at high $R$.
• Optimal hyperparameters become more sensitive at lower $U$ or higher $N$.

Fitted power laws (weighted least-squares, weighting by $1/\max\{0.1, \text{SE}\}^2$):

$$\text{eLR}^* = 3.95 \times 10^4 \cdot U^{-1.014} N^{0.273} R^{0.120}$$ $$\lambda^* = 1.08 \times 10^2 \cdot U^{-0.870} N^{0.398} R^{0.660}$$

These match prior observations that optimal weight decay grows with $N$ and $R$ (Kim et al., 2026; Qin et al., 2026).

Ensembling Scaling Law (Section 5)

The proposed scaling law extends Equation (2) with an ensemble term:

$$L(U, N, R, M) = E + \frac{A}{N^{\alpha}} + \frac{B}{(UR)^{\beta}} + P(R-1)^{\delta}\left(\frac{N}{U^{\gamma}}\right)^{\kappa} - \frac{F}{N^{\nu}U^{\epsilon}R^{\rho}}\left(1 - \frac{1}{M}\right)^{\mu} \tag{3}$$

Motivation for the ensemble term: For $M$ i.i.d. random variables with variance $\sigma^2$, the mean's squared standard error is $\sigma^2/M$, so the reduction vs. $M=1$ is $\sigma^2(1 - 1/M)$. This motivates $\mu \approx 1$, but allowing $\mu \neq 1$ captures whether gains concentrate at small $M$ ($\mu < 1$) or large $M$ ($\mu > 1$).

Fitting procedure:

• For $M=1$: use minima of fit quadratics from hyperparameter sweeps.
• For $M>1$: adjust loss by the gap between observed and predicted optimal loss at $M=1$.
• Fit with Huber loss in log-loss space ($\delta = 0.001$), weighting rungs by compute $C = 6UNRM$.

Fit parameters:

Empirical Validation / Results

Hyperparameter Transfer Validation

• Extrapolation to 2× compute: Prescriptions consistently predict near-optimal hyperparameters, outperforming all heuristics (including $\mu$P-based learning rate transfer and various weight decay scaling rules).
• Extrapolation to 16× compute: When refitting on rungs within a 16× lower budget and predicting for a target rung ($U=100M$, $N=398M$, $R=32$), the power law prescriptions significantly outperform heuristics (Figure 2, right). Critically, they dominate the "fixed hyperparameters" recipe used in prior work.

Ensembling Scaling Law Validation

• Extrapolation to unseen runs: The scaling law accurately tracks observed loss vs. $M$ for rungs where no $M>1$ data was seen (Figure 3, left).
• Fit quality: Measured via (a) leave-one-out cross-validation Huber loss withholding one $(U, N, R)$ rung, and (b) Huber loss fitting on $C \leq 4 \times 10^{18}$ or $C \leq 4 \times 10^{20}$ FLOPs and evaluating on held-out rungs.
• Baselines compared: Equation (2) (ignores ensembling) and a naive variant swapping $N \to NM$ (treating ensembling like overparameterization). The proposed form achieves much better fits on both the authors' runs and runs from Kim et al. (2026).

Large-Scale Prescription Testing (Section 6.1)

Four production budgets up to $10^{20}$ FLOPs (12× sweep budget) at $U = 100M$:

• Baselines: (1) Prior recipe (fixed hyperparameters, no ensembling), (2) hyperparameter transfer only, (3) ensembling only.
• Key result (Figure 4): The prior recipe's observed loss increases with compute. Including either hyperparameter transfer or ensembling improves performance, but combining both yields the largest gains—over 0.15 nats improvement.
• Specific numbers: Prior recipe reaches lowest loss of 3.35 nats at $10^{19}$ FLOPs; the proposed recipe achieves 3.27 nats at $10^{19}$ FLOPs (8% perplexity decrease) and under 3.20 nats at $10^{20}$ FLOPs (14% decrease).

Caveat: The hyperparameter-transfer-only recipe's observed loss drifts from predicted loss at scale (increasing loss at higher compute), likely because it extrapolates to $N$ or $R$ much larger than seen in sweeps.

Compute-Optimal Allocation Analysis (Section 6.2)

• For recipes supporting ensembling, prescribed $N^*$ and $R^*$ grow initially but then plateau, with additional compute allocated to increasing ensemble size $M$ (Figure 5, left). This avoids overfitting.
• Rule-of-thumb for ensemble size: Ensembling ($M^* \geq 2$) becomes compute-optimal for budgets exceeding approximately: $$C_2 \approx 1.4 \times 10^4 \times U^{1.9}$$ and $M^* = 3$ becomes optimal at approximately: $$C_3 \approx 4.8 \times 10^4 \times U^{1.9}$$ Beyond this, $M^*$ grows roughly linearly: $M^* \approx 2 + C/C_3$.

Theoretical and Practical Implications

Theoretical Implications

1. Scaling laws must account for the full resource allocation: The paper demonstrates that omitting ensembling as a scaling axis leads to qualitatively wrong prescriptions (non-monotonic loss with compute). This challenges the assumption that model size alone captures the benefits of increased parameters—ensembling behaves fundamentally differently from overparameterization.

2. Hyperparameters are scale-dependent: The finding that optimal weight decay scales as $\lambda^* \propto N^{0.398} R^{0.660} U^{-0.870}$ contradicts common heuristics (e.g., $\lambda \sim 1/U$ or $\lambda \sim 1/\sqrt{UN}$) and underscores that fixed hyperparameters across scale are a major source of suboptimality.

3. Variance-reduction motivates ensemble terms: The $(1 - 1/M)^\mu$ form is theoretically grounded in variance reduction of ensemble averaging, and the fitted $\mu = 1.12$ is close to the theoretical $\mu = 1$, lending credibility to the functional form.

Practical Implications

1. Immediate recipe improvements: Practitioners training under data constraints can adopt the fitted power laws for hyperparameter transfer and the scaling law for ensemble allocation, achieving significant loss reductions without additional tuning.

2. Ensembling is compute-optimal: Contrary to some prior doubts (Abe et al., 2022), the paper shows ensembling is compute-optimal for data-constrained training, especially at higher compute budgets relative to dataset size.

3. A template for future extensions: The methodology provides a template for incorporating new primitives (e.g., synthetic data pre-training, model growth during training) into data-constrained scaling laws.

Conclusion

The paper addresses two critical limitations of existing data-constrained scaling recipes—fixed hyperparameters and neglect of ensembling—and demonstrates that both are essential for monotonic improvements with compute. Key takeaways:

1. Hyperparameter transfer laws (power laws in $U$, $N$, $R$) significantly outperform fixed hyperparameters and existing heuristics, especially at larger scales.
2. Ensembling as a third scaling axis enables compute-optimal allocation that avoids overfitting, with gains growing with compute.
3. Combined contributions yield over 0.15 nats improvement (14% perplexity decrease) over prior recipes at $10^{20}$ FLOPs, validated at 12× the sweep budget.

Future directions: The authors emphasize that the data-constrained regime will continue to evolve, and new axes (synthetic data pre-training, dynamic model growth) may further improve the frontier. They advocate for systematic adaptation of scaling prescriptions to changing training conditions rather than assuming current regimes persist.