# Small-Scale Experiments: Are We There Yet?

> Scaling laws do exist at 4M parameters, but only emerge with rigorous hyperparameter tuning, which shrinks the loss surface's effective dimensionality as models scale.

- **Source:** [arXiv](https://arxiv.org/abs/2608.11859)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/D7cTAI
- **Whiteboard:** https://picx.dev/p/D7cTAI/image

## Summary

# Small-Scale Experiments: Are We There Yet?

**Authors:** Nicholas Lourie (NYU), Kyunghyun Cho (NYU), Karen Ullrich (Meta FAIR), Sanae Lotfi (Meta FAIR)

## Summary (Overview)

- **Key finding:** Scaling laws *do* exist at very small scales (as few as 4M parameters), but they only emerge on the fully tuned hyperparameter frontier. The confounding factor has been hyperparameter sensitivity, not scale itself.
- **Hyperparameter tuning is the single most important ingredient** in estimating a scaling law — more important than parameter definitions, learning rate decay, or tying scaling exponents. Poor tuning (e.g., only 4–16 configurations) makes scaling laws entirely invisible.
- **As models scale up, the hyperparameter loss surface becomes lower-dimensional**: the effective number of hyperparameters ($\gamma$) drops to 1, explaining why larger models are easier to tune.
- **The authors propose a new methodology** for model-centric research combining three tools: the noisy quadratic limit (to verify tuning completeness), scaling laws (to describe loss evolution), and perplexity-capability correspondence (to link loss to downstream capability).
- **Case study demonstration**: Using small-scale experiments, the authors recover the known large-scale result that **pre-normalization outperforms post-normalization** in transformers, a question that previously took the field years and expensive experiments to settle.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Foundation models have transformed computing, but their scale makes experimentation prohibitively expensive. The natural solution — running small-scale experiments and transferring insights to the large scale — was promised by scaling laws (Kaplan et al., 2020), yet has not materialized after half a decade. Instead, pretraining research costs have grown exponentially, with experiments dominating the expense of model development.

The paper identifies a major roadblock: **scaling laws are notoriously elusive at small scales**. The two most influential works reached contradictory conclusions:
- Kaplan et al. (2020) argued parameters should scale much faster than data
- Hoffmann et al. (2022) found they should scale at the same rate

Porian et al. (2024) traced this discrepancy to methodological choices (e.g., whether to count embeddings, warmup length, scale-dependent hyperparameter tuning) that disproportionately affect small models. Li et al. (2025c) found scaling laws based on small models (≤100M parameters) unreliable, coiling this the "scaling gap."

### Theoretical Foundations

The paper weaves together three threads from the literature:

#### 1. The Noisy Quadratic Limit (Lourie et al., 2025a)

Near optima, the loss surface is (1) **quadratic** (approximated by second-order Taylor expansion), (2) **low-rank** (governed by few directions), and (3) **normal** (additive normal noise). Formally, given hyperparameters $\mathbf{X}$ and validation score $Y$:

$$ \mathcal {L} (\pmb {X}) \approx y _ {*} + (\pmb {X} - \pmb {x} _ {*}) ^ {T} H _ {\pmb {x} _ {*}} (\pmb {X} - \pmb {x} _ {*}) + E, \qquad E \sim \mathcal {N} (0, \sigma)\tag{1} $$

The marginal distribution over random search scores converges to a noisy quadratic distribution $\mathcal{Q}_{\min}(\alpha,\beta,\gamma,\sigma)$:

$$
\mathbb {P} (Y \leq y) \approx F (y; \alpha , \beta , \gamma , \sigma) \quad \text { as } \quad y \to y _ {*}
\tag{2}$$

where $\alpha$ = best attainable loss, $\beta$ = concentration of scores, $\gamma$ = **effective number of hyperparameters** (intrinsic dimension), $\sigma$ = noise from random seeds.

#### 2. Scaling Laws

The canonical joint scaling law (Hoffmann et al., 2022) relates parameters $p$ and data $d$ to pretraining loss $\mathcal{L}$:

$$
\mathcal {L} (p, d) = \epsilon + \frac {\zeta}{p ^ {\iota}} + \frac {\eta}{d ^ {\kappa}}\tag{3}
$$

with the compute constraint:
$$
c \propto p d\tag{4}
$$

The optimality condition for budget allocation is:
$$
\iota \frac {\zeta}{p ^ {\iota}} = \kappa \frac {\eta}{d ^ {\kappa}}\tag{5}
$$

Leading to the optimal parameter and data allocations as power laws of compute:
$$
p \propto c ^ {\frac {\kappa}{\iota + \kappa}} \quad d \propto c ^ {\frac {\iota}{\iota + \kappa}}
\tag{6}
$$

The excess loss follows a power law:
$$
\mathcal {L} (p, d) - \epsilon \propto c ^ {- \frac {\iota \kappa}{\iota + \kappa}}\tag{7}
$$

#### 3. Perplexity-Capability Correspondence (Mayilvahanan et al., 2025)

Downstream capabilities are highly unpredictable (emergent, inverse, or U-shaped scaling behavior). However, **as long as pretraining data is held fixed, models achieving the same pretraining loss obtain the same capabilities** — regardless of scale, hyperparameters, or even architecture. This lets researchers compare models by pretraining loss without needing to predict downstream tasks.

---

## Methodology

### Model Scaling and Experimental Design

- **Effective parameter count**: Parameters adjusted for FLOPs (excluding embedding, including unembedding and attention cost), keeping $\mathbf{c} = 6pd$ exact.
- **Scales**: $2^{22} \approx 4M$ to $2^{28} \approx 268M$ parameters, with two architectural designs (hand-crafted model ladder, randomly sampled architectures).

### Training, and Evaluation

- **Schedule**: Warmup-stable-decay (WSD) (Hu et al., 2024), following Hägele et al. (2024)'s technique of branching decay phases from checkpoints at $1/8, 2/8, \ldots, 8/8$ through training — yielding evaluations at eight token budgets fraction of the cost.
- **Evaluation**: Both stable and decayed checkpoints are evaluated.

### Analysis

- Scales are partitioned into **training** (4M–34M), **validation** (67M–134M), and **test** (268M) to measure extrapolation.
- The joint scaling law (Equation 3) is fit to best loss per parameter–data budget pair. The two smallest token budgets are discarded for poor fit (consistent with prior work).

---

## Empirical Validation / Results

### 3.1 The Impact of Methodology

- **Parameter definitions** (how to count embeddings, unembeddings, attention) make **little difference** to estimated scaling laws (Figure 3). The paper uses "effective parameters" for consistency with $c = 6pd$.
- **Refinements** (per-budget hyperparameter tuning, learning rate decay, tying exponents $\iota = \kappa$) improve precision but do **not** enable the law — it is already recognizable without them.
  - Tuning per parameter-data budget: 50% reduction in test MSE
  - Learning rate decay: 98% reduction in test MSE
  - Tying exponents: inconsistent (slightly better test, worse validation)

### 3.2 The Importance of Hyperparameter Tuning

| Configurations per scale | Outcome |
|---|---|
| 4 | Scaling law completely absent |
| 16 | Still not reliable — no idea is too noisy from suboptimal configurations |
| 64 | Law clearly visible but extrapolation weak |
| 256 | Accurate scaling law obtained |

> "The one ingredient above all is required to estimate a scaling law: **rigorous hyperparameter tuning**."

This explains why small-scale laws are so easy to miss: reaching the tuned frontier requires searches far more extensive than standard practice (which typically tunes a small grid).

### 4.1 Hyperparameter Tuning Gets Easier with Scale

- **Near-optimal configurations fill a larger fraction of space** as models scale (Figure 6): the score distribution concentrates on the best achievable loss.
- Both **parameters and data** contribute to decreased sensitivity — neither alone collapses the gap between the 25th/10th percentiles and the minimum (Figure 7).

### 4.2 The Hyperparameter Loss Surface Becomes Lower Dimensional

- The **effective number of hyperparameters ($\gamma$)** drops from a more complex value to **1** with increasing parameters (Figure 8).
- **Parameters** (not data) are the primary driver of the lower-dimensional surface (Figure 9): $\gamma$ drops with parameter count regardless of data budget.

### 5.3 Case Study: Pre-Norm vs. Post-Norm Transformers

The authors test their methodology on the historical question of where normalization layers should go (pre-norm vs. post-norm) — a question that took the field years to settle.

**Experimental setup**: Three scales (4M, 34M, 134M parameters); several hundred random-search runs per architecture. Cost: equivalent to "a handful of billion-parameter runs" because compute grows quadratically with model size when parameter:data ratio is fixed.

**Diagnostic results**:

| Diagnostic | Pre-Norm | Post-Norm |
|---|---|---|
| 1: Noisy quadratic convergence | Emerges readily | Emerges, but only deeper into the tail (smaller asymptotic regime = more sensitive) |
| 2: Sensitivity scaling | Distinct peak at optimum, pronounced shift with scale | Persistent second peak at suboptimal value; slight improvement only |
| 3: Perplexity-capability correspondence | All models fall on a common trend | Same trend — equal loss implies equal capability |
| 4: Scaling law extrapolation | Excellent; stable with joint/free irreducible error | Reduced; would benefit from further tuning |
| 5: Scaling comparison | Better near the data, where laws are reliable | Appears to only catch up beyond tested scales, and only with free irreducible error |

**The verdict**: Pre-norm is the better choice at scale, recovering the known large-scale result from small-scale experiments. This is supported holistically: post-norm was harder to tune, scales worse near-data, and if both architectures saturate at the same loss (shared irreducible error), pre-norm's advantage grows with compute.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Scaling laws are real at tiny scales** if the fully tuned frontier is reached — the previous belief that small-scale laws are fundamentally unreliable was an artifact of undertuning.
2. **The hyperparameter loss surface is a scale-dependent object**: its intrinsic dimension shrinks with model size (parameters more than data). This introduces a new "blessing of dimensionality" — the parameter's geometry cures the curse of dimensionality in the hyperparameter space.
3. **Quantitative extrapolation has statistical limits**: while scaling laws extend an order of magnitude reliably, further extrapolation increasingly reflects the irreducible loss term ($\epsilon$), whose estimates vary wildly across samples. A **holistic, qualitative approach** is required rather than pure extrapolation.

### Practical Implications

1. **Methodology for model-centered research**: Tune thoroughly at small scale (hundreds of runs), use the noisy quadratic limit to verify tuning, establish the scaling law, check perplexity-capability correspondence, then carry hyperparameters up using simple rules (since large models are easier to tune).
2. **Cost-effectiveness**: A 1B-parameter model run costs as much as 64 runs at 134M, 1,024 at 34M, or 65,536 at 4M parameters. Small-scale experiments are feasible for many academic and industrial labs.
3. **Key constraint**: This methodology is limited to **model-centric vs. data-centric research —** changing the pretraining data breaks the perplexity–capability correspondence, limiting small-scale predictability for data-centric work.

---

## Conclusion

Scaling laws' promise — cheap experiments that transfer to large models — is realizable, but requires the right tools and understanding:

1. **Small models are hyperparameter-sensitive, but this sensitivity fades with scale.** This is the key confounder that made scaling laws appear absent at tiny scales.
2. **The discovery that the hyperparameter loss surface drops in dimensional with scale** provides a principled explanation and guides experimental design: extensive search at small scales, standard techniques for transfer at large scales.
3. **Quantitative extrapolation is limited** by statistical considerations; a holistic, diagnostic-driven approach outperforms pure extrapolation.
4. The methodology successfully recovers the pre-norm vs. post-norm large-scale conclusion from small-scale experiments requiring ~1,000 runs at up to 134M parameters — demonstrating cost savings and rigor simultaneously.

**Future directions** identified: scaling the approach to data-centric research (where perplexity-capability correspondence breaks down), understanding the boundaries and robustness of $\gamma$ across architectures and parametrizations (e.g., $\mu$P), and exploring whether a unified quantitative model of the scaling loss surface is possible.

---

_Markdown view of https://picx.dev/p/D7cTAI, served by PicX — AI-generated visual whiteboard summaries of research papers._