# Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling

> Data-constrained scaling laws must treat ensembling as a third compute-allocation axis and scale hyperparameters via power laws, yielding 14% lower perplexity than prior recipes.

- **Source:** [arXiv](https://arxiv.org/abs/2610.37842)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/VhjN3A
- **Whiteboard:** https://picx.dev/p/VhjN3A/image

## Summary

# Summary of "Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling"

## Summary (Overview)

- **Core problem**: Existing data-constrained scaling prescriptions suffer from two critical limitations: (1) they hold hyperparameters (like weight decay) fixed across scales, despite optimal values varying wildly with model size and epochs; (2) they neglect ensembling as a scaling axis, leading to prescriptions where loss can *increase* with compute budget.
- **Key contributions**: The authors (1) fit empirical power laws for hyperparameter transfer across scale, and (2) propose a new scaling law that models ensembling as a third axis (alongside model size and epochs) for compute-optimal allocation under data constraints.
- **Main result**: Training on 100M tokens from FineWeb, their recipe reduces validation loss by over 0.15 nats compared to prior recipes (a 14% decrease in perplexity), achieving monotonic improvements with compute through ensembling.
- **Validation**: Gains are achieved using prescriptions fit to smaller runs and validated at up to 12× larger compute budgets, demonstrating robust extrapolation.
- **Key insight**: The compute-optimal allocation shifts from growing model size and epochs to growing ensemble size once model size and epochs reach overfitting-prone levels.

## Introduction and Theoretical Foundation

### Background

The paper situates itself across three training regimes:

1. **Traditional regime**: Small datasets, where regularization and ensembling were essential (Lakshminarayanan et al., 2017; Loshchilov & Hutter, 2019).
2. **Current regime**: Internet-scale data, where scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) prescribe compute-optimal configurations assuming single-epoch training and effectively infinite data.
3. **Emerging data-constrained regime**: Compute budgets grow faster than high-quality data supply (Villalobos et al., 2024 predict exhaustion of public text by 2026), necessitating multi-epoch training and renewed attention to regularization and ensembling.

### Theoretical Foundation: Scaling Laws

The Chinchilla scaling law (Hoffmann et al., 2022) forms the baseline:

$$
L(U, N) = E + \frac{A}{N^{\alpha}} + \frac{B}{U^{\beta}} \tag{1}
$$

where $L$ is validation loss, $N$ is model size (non-embedding parameters), $U$ is dataset size in tokens, and $\{E, A, \alpha, B, \beta\}$ are free parameters. Compute is approximated as $C \approx 6UN$.

**Data-constrained extension** (Lovelace et al., 2026) explicitly models overfitting from multi-epoch training:

$$
L(U, N, R) = E + \frac{A}{N^{\alpha}} + \frac{B}{(UR)^{\beta}} + P(R-1)^{\delta}\left(\frac{N}{U^{\gamma}}\right)^{\kappa} \tag{2}
$$

with four new parameters $\{P, \delta, \kappa, \gamma\}$. When $R=1$, the last term vanishes, recovering the Chinchilla law.

### The Pathology

The authors identify a critical failure: existing recipes (e.g., Lovelace et al., 2026) predict that compute-optimal loss *increases* past mild compute budgets (~$10^{18}$–$10^{19}$ FLOPs), rather than decreasing monotonically. This pathology arises from:
- Fixed hyperparameters across scales (suboptimal at larger scales)
- No consideration of ensembling as an alternative compute allocation

## Methodology

### Experimental Setup

- **Models**: Decoder-only transformers on FineWeb (Penedo et al., 2024) with GPT-2 tokenization.
- **Grid**: Unique tokens $U \in \{25M, 50M, 100M, 200M, 400M\}$, model sizes $N = 30M$–$398M$ parameters, epochs $R = 2$–$64$ (some runs at 128 and 256 epochs).
- **Optimizer**: Muon (instead of AdamW), reflecting a more competitive data-constrained setup inspired by the Slowrun competition (Vegesna et al., 2026).

### Hyperparameter Scaling Laws (Section 4)

**Sweep strategy**: Two-dimensional logarithmic grid over weight decay $\lambda$ and effective learning rate $\text{eLR} = \eta \lambda$ (coupled weight decay), with $\sqrt{2}$-spaced lattice across 110 $(U, N, R)$ rungs spanning ~3 orders of magnitude of compute.

**Localization**: For each rung, fit a least-squares quadratic $L(\ln \lambda, \ln \text{eLR})$ to estimate optimal hyperparameters. Key observations:
- $\eta$ is most sensitive at low $R$; $\lambda$ is most sensitive at high $R$.
- Optimal hyperparameters become more sensitive at lower $U$ or higher $N$.

**Fitted power laws** (weighted least-squares, weighting by $1/\max\{0.1, \text{SE}\}^2$):

$$
\text{eLR}^* = 3.95 \times 10^4 \cdot U^{-1.014} N^{0.273} R^{0.120}
$$

$$
\lambda^* = 1.08 \times 10^2 \cdot U^{-0.870} N^{0.398} R^{0.660}
$$

These match prior observations that optimal weight decay grows with $N$ and $R$ (Kim et al., 2026; Qin et al., 2026).

### Ensembling Scaling Law (Section 5)

The proposed scaling law extends Equation (2) with an ensemble term:

$$
L(U, N, R, M) = E + \frac{A}{N^{\alpha}} + \frac{B}{(UR)^{\beta}} + P(R-1)^{\delta}\left(\frac{N}{U^{\gamma}}\right)^{\kappa} - \frac{F}{N^{\nu}U^{\epsilon}R^{\rho}}\left(1 - \frac{1}{M}\right)^{\mu} \tag{3}
$$

**Motivation for the ensemble term**: For $M$ i.i.d. random variables with variance $\sigma^2$, the mean's squared standard error is $\sigma^2/M$, so the reduction vs. $M=1$ is $\sigma^2(1 - 1/M)$. This motivates $\mu \approx 1$, but allowing $\mu \neq 1$ captures whether gains concentrate at small $M$ ($\mu < 1$) or large $M$ ($\mu > 1$).

**Fitting procedure**: 
- For $M=1$: use minima of fit quadratics from hyperparameter sweeps.
- For $M>1$: adjust loss by the gap between observed and predicted optimal loss at $M=1$.
- Fit with Huber loss in log-loss space ($\delta = 0.001$), weighting rungs by compute $C = 6UNRM$.

**Fit parameters**:

| $E$ | $A$ | $\alpha$ | $B$ | $\beta$ | $P$ | $\delta$ | $\kappa$ | $\gamma$ | $F$ | $\mu$ | $\nu$ | $\epsilon$ | $\rho$ |
|-----|-----|----------|-----|---------|-----|----------|----------|----------|-----|-------|-------|------------|-------|
| 2.18 | 165 | 0.298 | 1759 | 0.408 | 117 | 0.0791 | 0.162 | 3.04 | 192 | 1.12 | 0.224 | 0.137 | 0.263 |

## Empirical Validation / Results

### Hyperparameter Transfer Validation

- **Extrapolation to 2× compute**: Prescriptions consistently predict near-optimal hyperparameters, outperforming all heuristics (including $\mu$P-based learning rate transfer and various weight decay scaling rules).
- **Extrapolation to 16× compute**: When refitting on rungs within a 16× lower budget and predicting for a target rung ($U=100M$, $N=398M$, $R=32$), the power law prescriptions significantly outperform heuristics (Figure 2, right). Critically, they dominate the "fixed hyperparameters" recipe used in prior work.

### Ensembling Scaling Law Validation

- **Extrapolation to unseen runs**: The scaling law accurately tracks observed loss vs. $M$ for rungs where no $M>1$ data was seen (Figure 3, left).
- **Fit quality**: Measured via (a) leave-one-out cross-validation Huber loss withholding one $(U, N, R)$ rung, and (b) Huber loss fitting on $C \leq 4 \times 10^{18}$ or $C \leq 4 \times 10^{20}$ FLOPs and evaluating on held-out rungs.
- **Baselines compared**: Equation (2) (ignores ensembling) and a naive variant swapping $N \to NM$ (treating ensembling like overparameterization). The proposed form achieves much better fits on both the authors' runs and runs from Kim et al. (2026).

### Large-Scale Prescription Testing (Section 6.1)

Four production budgets up to $10^{20}$ FLOPs (12× sweep budget) at $U = 100M$:

- **Baselines**: (1) Prior recipe (fixed hyperparameters, no ensembling), (2) hyperparameter transfer only, (3) ensembling only.
- **Key result** (Figure 4): The prior recipe's observed loss *increases* with compute. Including either hyperparameter transfer or ensembling improves performance, but combining both yields the largest gains—over 0.15 nats improvement.
- **Specific numbers**: Prior recipe reaches lowest loss of 3.35 nats at $10^{19}$ FLOPs; the proposed recipe achieves 3.27 nats at $10^{19}$ FLOPs (8% perplexity decrease) and under 3.20 nats at $10^{20}$ FLOPs (14% decrease).

**Caveat**: The hyperparameter-transfer-only recipe's observed loss drifts from predicted loss at scale (increasing loss at higher compute), likely because it extrapolates to $N$ or $R$ much larger than seen in sweeps.

### Compute-Optimal Allocation Analysis (Section 6.2)

- For recipes supporting ensembling, prescribed $N^*$ and $R^*$ grow initially but then plateau, with additional compute allocated to increasing ensemble size $M$ (Figure 5, left). This avoids overfitting.
- **Rule-of-thumb for ensemble size**: Ensembling ($M^* \geq 2$) becomes compute-optimal for budgets exceeding approximately:
  $$C_2 \approx 1.4 \times 10^4 \times U^{1.9}$$
  and $M^* = 3$ becomes optimal at approximately:
  $$C_3 \approx 4.8 \times 10^4 \times U^{1.9}$$
  Beyond this, $M^*$ grows roughly linearly: $M^* \approx 2 + C/C_3$.

## Theoretical and Practical Implications

### Theoretical Implications

1. **Scaling laws must account for the full resource allocation**: The paper demonstrates that omitting ensembling as a scaling axis leads to qualitatively wrong prescriptions (non-monotonic loss with compute). This challenges the assumption that model size alone captures the benefits of increased parameters—ensembling behaves fundamentally differently from overparameterization.

2. **Hyperparameters are scale-dependent**: The finding that optimal weight decay scales as $\lambda^* \propto N^{0.398} R^{0.660} U^{-0.870}$ contradicts common heuristics (e.g., $\lambda \sim 1/U$ or $\lambda \sim 1/\sqrt{UN}$) and underscores that fixed hyperparameters across scale are a major source of suboptimality.

3. **Variance-reduction motivates ensemble terms**: The $(1 - 1/M)^\mu$ form is theoretically grounded in variance reduction of ensemble averaging, and the fitted $\mu = 1.12$ is close to the theoretical $\mu = 1$, lending credibility to the functional form.

### Practical Implications

1. **Immediate recipe improvements**: Practitioners training under data constraints can adopt the fitted power laws for hyperparameter transfer and the scaling law for ensemble allocation, achieving significant loss reductions without additional tuning.

2. **Ensembling is compute-optimal**: Contrary to some prior doubts (Abe et al., 2022), the paper shows ensembling is compute-optimal for data-constrained training, especially at higher compute budgets relative to dataset size.

3. **A template for future extensions**: The methodology provides a template for incorporating new primitives (e.g., synthetic data pre-training, model growth during training) into data-constrained scaling laws.

## Conclusion

The paper addresses two critical limitations of existing data-constrained scaling recipes—fixed hyperparameters and neglect of ensembling—and demonstrates that both are essential for monotonic improvements with compute. Key takeaways:

1. **Hyperparameter transfer laws** (power laws in $U$, $N$, $R$) significantly outperform fixed hyperparameters and existing heuristics, especially at larger scales.
2. **Ensembling as a third scaling axis** enables compute-optimal allocation that avoids overfitting, with gains growing with compute.
3. **Combined contributions** yield over 0.15 nats improvement (14% perplexity decrease) over prior recipes at $10^{20}$ FLOPs, validated at 12× the sweep budget.

**Future directions**: The authors emphasize that the data-constrained regime will continue to evolve, and new axes (synthetic data pre-training, dynamic model growth) may further improve the frontier. They advocate for systematic adaptation of scaling prescriptions to changing training conditions rather than assuming current regimes persist.

---

_Markdown view of https://picx.dev/p/VhjN3A, served by PicX — AI-generated visual whiteboard summaries of research papers._