Small-Scale Experiments: Are We There Yet?

Authors: Nicholas Lourie (NYU), Kyunghyun Cho (NYU), Karen Ullrich (Meta FAIR), Sanae Lotfi (Meta FAIR)

Summary (Overview)

  • Key finding: Scaling laws do exist at very small scales (as few as 4M parameters), but they only emerge on the fully tuned hyperparameter frontier. The confounding factor has been hyperparameter sensitivity, not scale itself.
  • Hyperparameter tuning is the single most important ingredient in estimating a scaling law — more important than parameter definitions, learning rate decay, or tying scaling exponents. Poor tuning (e.g., only 4–16 configurations) makes scaling laws entirely invisible.
  • As models scale up, the hyperparameter loss surface becomes lower-dimensional: the effective number of hyperparameters (γ\gamma) drops to 1, explaining why larger models are easier to tune.
  • The authors propose a new methodology for model-centric research combining three tools: the noisy quadratic limit (to verify tuning completeness), scaling laws (to describe loss evolution), and perplexity-capability correspondence (to link loss to downstream capability).
  • Case study demonstration: Using small-scale experiments, the authors recover the known large-scale result that pre-normalization outperforms post-normalization in transformers, a question that previously took the field years and expensive experiments to settle.

Introduction and Theoretical Foundation

Background and Motivation

Foundation models have transformed computing, but their scale makes experimentation prohibitively expensive. The natural solution — running small-scale experiments and transferring insights to the large scale — was promised by scaling laws (Kaplan et al., 2020), yet has not materialized after half a decade. Instead, pretraining research costs have grown exponentially, with experiments dominating the expense of model development.

The paper identifies a major roadblock: scaling laws are notoriously elusive at small scales. The two most influential works reached contradictory conclusions:

  • Kaplan et al. (2020) argued parameters should scale much faster than data
  • Hoffmann et al. (2022) found they should scale at the same rate

Porian et al. (2024) traced this discrepancy to methodological choices (e.g., whether to count embeddings, warmup length, scale-dependent hyperparameter tuning) that disproportionately affect small models. Li et al. (2025c) found scaling laws based on small models (≤100M parameters) unreliable, coiling this the "scaling gap."

Theoretical Foundations

The paper weaves together three threads from the literature:

1. The Noisy Quadratic Limit (Lourie et al., 2025a)

Near optima, the loss surface is (1) quadratic (approximated by second-order Taylor expansion), (2) low-rank (governed by few directions), and (3) normal (additive normal noise). Formally, given hyperparameters X\mathbf{X} and validation score YY:

L(X)≈y∗+(X−x∗)THx∗(X−x∗)+E,E∼N(0,σ)(1)\mathcal {L} (\pmb {X}) \approx y _ {*} + (\pmb {X} - \pmb {x} _ {*}) ^ {T} H _ {\pmb {x} _ {*}} (\pmb {X} - \pmb {x} _ {*}) + E, \qquad E \sim \mathcal {N} (0, \sigma)\tag{1}

The marginal distribution over random search scores converges to a noisy quadratic distribution Qmin⁡(α,β,γ,σ)\mathcal{Q}_{\min}(\alpha,\beta,\gamma,\sigma):

\mathbb {P} (Y \leq y) \approx F (y; \alpha , \beta , \gamma , \sigma) \quad \text { as } \quad y \to y _ {*} \tag{2}$$ where $\alpha$ = best attainable loss, $\beta$ = concentration of scores, $\gamma$ = **effective number of hyperparameters** (intrinsic dimension), $\sigma$ = noise from random seeds. #### 2. Scaling Laws The canonical joint scaling law (Hoffmann et al., 2022) relates parameters $p$ and data $d$ to pretraining loss $\mathcal{L}$:

\mathcal {L} (p, d) = \epsilon + \frac {\zeta}{p ^ {\iota}} + \frac {\eta}{d ^ {\kappa}}\tag{3}

withthecomputeconstraint: with the compute constraint:

c \propto p d\tag{4}

Theoptimalityconditionforbudgetallocationis: The optimality condition for budget allocation is:

\iota \frac {\zeta}{p ^ {\iota}} = \kappa \frac {\eta}{d ^ {\kappa}}\tag{5}

Leadingtotheoptimalparameteranddataallocationsaspowerlawsofcompute: Leading to the optimal parameter and data allocations as power laws of compute:

p \propto c ^ {\frac {\kappa}{\iota + \kappa}} \quad d \propto c ^ {\frac {\iota}{\iota + \kappa}} \tag{6}

Theexcesslossfollowsapowerlaw:The excess loss follows a power law:

\mathcal {L} (p, d) - \epsilon \propto c ^ {- \frac {\iota \kappa}{\iota + \kappa}}\tag{7}

#### 3. Perplexity-Capability Correspondence (Mayilvahanan et al., 2025) Downstream capabilities are highly unpredictable (emergent, inverse, or U-shaped scaling behavior). However, **as long as pretraining data is held fixed, models achieving the same pretraining loss obtain the same capabilities** — regardless of scale, hyperparameters, or even architecture. This lets researchers compare models by pretraining loss without needing to predict downstream tasks. --- ## Methodology ### Model Scaling and Experimental Design - **Effective parameter count**: Parameters adjusted for FLOPs (excluding embedding, including unembedding and attention cost), keeping $\mathbf{c} = 6pd$ exact. - **Scales**: $2^{22} \approx 4M$ to $2^{28} \approx 268M$ parameters, with two architectural designs (hand-crafted model ladder, randomly sampled architectures). ### Training, and Evaluation - **Schedule**: Warmup-stable-decay (WSD) (Hu et al., 2024), following Hägele et al. (2024)'s technique of branching decay phases from checkpoints at $1/8, 2/8, \ldots, 8/8$ through training — yielding evaluations at eight token budgets fraction of the cost. - **Evaluation**: Both stable and decayed checkpoints are evaluated. ### Analysis - Scales are partitioned into **training** (4M–34M), **validation** (67M–134M), and **test** (268M) to measure extrapolation. - The joint scaling law (Equation 3) is fit to best loss per parameter–data budget pair. The two smallest token budgets are discarded for poor fit (consistent with prior work). --- ## Empirical Validation / Results ### 3.1 The Impact of Methodology - **Parameter definitions** (how to count embeddings, unembeddings, attention) make **little difference** to estimated scaling laws (Figure 3). The paper uses "effective parameters" for consistency with $c = 6pd$. - **Refinements** (per-budget hyperparameter tuning, learning rate decay, tying exponents $\iota = \kappa$) improve precision but do **not** enable the law — it is already recognizable without them. - Tuning per parameter-data budget: 50% reduction in test MSE - Learning rate decay: 98% reduction in test MSE - Tying exponents: inconsistent (slightly better test, worse validation) ### 3.2 The Importance of Hyperparameter Tuning | Configurations per scale | Outcome | |---|---| | 4 | Scaling law completely absent | | 16 | Still not reliable — no idea is too noisy from suboptimal configurations | | 64 | Law clearly visible but extrapolation weak | | 256 | Accurate scaling law obtained | > "The one ingredient above all is required to estimate a scaling law: **rigorous hyperparameter tuning**." This explains why small-scale laws are so easy to miss: reaching the tuned frontier requires searches far more extensive than standard practice (which typically tunes a small grid). ### 4.1 Hyperparameter Tuning Gets Easier with Scale - **Near-optimal configurations fill a larger fraction of space** as models scale (Figure 6): the score distribution concentrates on the best achievable loss. - Both **parameters and data** contribute to decreased sensitivity — neither alone collapses the gap between the 25th/10th percentiles and the minimum (Figure 7). ### 4.2 The Hyperparameter Loss Surface Becomes Lower Dimensional - The **effective number of hyperparameters ($\gamma$)** drops from a more complex value to **1** with increasing parameters (Figure 8). - **Parameters** (not data) are the primary driver of the lower-dimensional surface (Figure 9): $\gamma$ drops with parameter count regardless of data budget. ### 5.3 Case Study: Pre-Norm vs. Post-Norm Transformers The authors test their methodology on the historical question of where normalization layers should go (pre-norm vs. post-norm) — a question that took the field years to settle. **Experimental setup**: Three scales (4M, 34M, 134M parameters); several hundred random-search runs per architecture. Cost: equivalent to "a handful of billion-parameter runs" because compute grows quadratically with model size when parameter:data ratio is fixed. **Diagnostic results**: | Diagnostic | Pre-Norm | Post-Norm | |---|---|---| | 1: Noisy quadratic convergence | Emerges readily | Emerges, but only deeper into the tail (smaller asymptotic regime = more sensitive) | | 2: Sensitivity scaling | Distinct peak at optimum, pronounced shift with scale | Persistent second peak at suboptimal value; slight improvement only | | 3: Perplexity-capability correspondence | All models fall on a common trend | Same trend — equal loss implies equal capability | | 4: Scaling law extrapolation | Excellent; stable with joint/free irreducible error | Reduced; would benefit from further tuning | | 5: Scaling comparison | Better near the data, where laws are reliable | Appears to only catch up beyond tested scales, and only with free irreducible error | **The verdict**: Pre-norm is the better choice at scale, recovering the known large-scale result from small-scale experiments. This is supported holistically: post-norm was harder to tune, scales worse near-data, and if both architectures saturate at the same loss (shared irreducible error), pre-norm's advantage grows with compute. --- ## Theoretical and Practical Implications ### Theoretical Implications 1. **Scaling laws are real at tiny scales** if the fully tuned frontier is reached — the previous belief that small-scale laws are fundamentally unreliable was an artifact of undertuning. 2. **The hyperparameter loss surface is a scale-dependent object**: its intrinsic dimension shrinks with model size (parameters more than data). This introduces a new "blessing of dimensionality" — the parameter's geometry cures the curse of dimensionality in the hyperparameter space. 3. **Quantitative extrapolation has statistical limits**: while scaling laws extend an order of magnitude reliably, further extrapolation increasingly reflects the irreducible loss term ($\epsilon$), whose estimates vary wildly across samples. A **holistic, qualitative approach** is required rather than pure extrapolation. ### Practical Implications 1. **Methodology for model-centered research**: Tune thoroughly at small scale (hundreds of runs), use the noisy quadratic limit to verify tuning, establish the scaling law, check perplexity-capability correspondence, then carry hyperparameters up using simple rules (since large models are easier to tune). 2. **Cost-effectiveness**: A 1B-parameter model run costs as much as 64 runs at 134M, 1,024 at 34M, or 65,536 at 4M parameters. Small-scale experiments are feasible for many academic and industrial labs. 3. **Key constraint**: This methodology is limited to **model-centric vs. data-centric research —** changing the pretraining data breaks the perplexity–capability correspondence, limiting small-scale predictability for data-centric work. --- ## Conclusion Scaling laws' promise — cheap experiments that transfer to large models — is realizable, but requires the right tools and understanding: 1. **Small models are hyperparameter-sensitive, but this sensitivity fades with scale.** This is the key confounder that made scaling laws appear absent at tiny scales. 2. **The discovery that the hyperparameter loss surface drops in dimensional with scale** provides a principled explanation and guides experimental design: extensive search at small scales, standard techniques for transfer at large scales. 3. **Quantitative extrapolation is limited** by statistical considerations; a holistic, diagnostic-driven approach outperforms pure extrapolation. 4. The methodology successfully recovers the pre-norm vs. post-norm large-scale conclusion from small-scale experiments requiring ~1,000 runs at up to 134M parameters — demonstrating cost savings and rigor simultaneously. **Future directions** identified: scaling the approach to data-centric research (where perplexity-capability correspondence breaks down), understanding the boundaries and robustness of $\gamma$ across architectures and parametrizations (e.g., $\mu$P), and exploring whether a unified quantitative model of the scaling loss surface is possible.

Related papers