# Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters

> Step Law's power-law form transfers to sub-59M-parameter language models, but its coefficients do not, overestimating optimal learning rate by roughly 4x.

- **Source:** [arXiv](https://arxiv.org/abs/2609.27581)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/zPJYQS
- **Whiteboard:** https://picx.dev/p/zPJYQS/image

## Summary

## Summary (Overview)

- **Core Question**: This paper empirically tests whether Step Law [Li et al., 2025] — a power-law formula for optimal learning rate $\eta^*$ and batch size $B^*$ calibrated on models between 59M and 1B parameters — transfers to small language models with $N < 59M$ parameters.
- **Main Finding (H2 Accepted)**: The power-law functional form holds at small scale, but the coefficients differ significantly from the original Step Law. Direct transfer systematically overestimates the optimal learning rate by a median factor of ~4.0× (range 2.4–6.6×).
- **Recalibrated Laws**: The authors fit new coefficients: $\hat{\eta}^*(N, D) = 0.0985 N^{-0.508} D^{0.238}$ ($R^2 = 0.834$) and $\hat{B}^*(D) = 3.6 \times 10^{-4} D^{0.931}$ ($R^2 = 0.950$).
- **Structural Claim Reproduced**: Step Law's claim that $B^*$ is independent of $N$ holds at small scale ($p(\log N) = 0.87$), but the growth of $B^*$ with data budget $D$ is nearly twice as steep ($\gamma = 0.931$ vs. 0.571).
- **Practical Recommendation**: For small LLMs, divide the Step Law-predicted learning rate by approximately four, or use the recalibrated formula directly.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Pre-training large language models is among the most expensive procedures in modern machine learning. Scaling laws [Kaplan et al., 2020; Hoffmann et al., 2022] characterize how loss depends on parameter count $N$ and data budget $D$, but are silent on optimization hyperparameters. Step Law [Li et al., 2025] fills this gap by providing:

$$\eta^*(N, D) = 1.79 \cdot N^{-0.713} \cdot D^{0.307}, \qquad B^*(D) = 0.58 \cdot D^{0.571} \tag{7}$$

Calibrated on ~3,700 runs at seven values of $N \in [59M, 1B]$, the region $N < 59M$ was never tested empirically by the authors, who explicitly defer generalization beyond the calibrated range to future work.

### Why the Small-Scale Regime Matters

The region $N < 59M$ is practically significant for:
- **Edge inference and mobile devices** with tight memory and energy budgets
- **Interpretability research**, where small models are far easier to analyze mechanistically
- **Single-GPU training**, typical of educational and early-stage projects

There is also a principled reason to distrust naive extrapolation: power laws in machine learning routinely break down at the edges of their calibrated range, and reduced model capacity can change the geometry of the hyperparameter landscape.

### Theoretical Foundation

The paper builds on:
- **Gradient noise scale theory** [McCandlish et al., 2018]: critical batch size $B^*$ is set by the ratio of gradient noise to gradient norm, which changes primarily with training duration $D$ rather than model width $N$.
- **Maximal update parameterization (μP)** [Yang et al., 2022]: an alternative theory-driven approach where optimal learning rate is invariant to model width, but requiring architectural modifications.
- **Small language models** [Eldan and Li, 2023]: models with 1–30M parameters trained on narrow, high-quality corpora produce coherent text and display rudimentary reasoning.

### Research Gap

No prior work covers the region $N < 59M$ on a standard parameterization with a joint dependence of hyperparameters on both $N$ and $D$ — this is the gap addressed by the paper.

**Table 1: Comparison with existing work**

| Work | Range of N | Target | Dependence on D |
|------|------------|--------|-----------------|
| Kaplan et al. [2020] | $10^7 - 10^{10}$ | Loss | Yes |
| Chinchilla [2022] | $10^8 - 10^{11}$ | Loss | Yes |
| McCandlish et al. [2018] | $10^7 - 10^9$ | $B^*$ | Indirect |
| μP [Yang et al., 2022] | $10^7 - 10^{10}$ | $\eta^*$ | No |
| DeepSeek [2024] | $10^9 - 10^{11}$ | $\eta^*, B^*$ | Via C |
| Step Law [Li et al., 2025] | $6 \times 10^7 - 10^9$ | $\eta^*, B^*$ | Explicit |
| **This work** | **$2.5 \times 10^5 - 2 \times 10^6$** | **$\eta^*, B^*$** | **Explicit** |

---

## Methodology

### Problem Setup

For fixed architecture $\mathcal{A}$, data distribution $\mathcal{D}$, parameter count $N$, and data budget $D$, the optimal hyperparameters are:

$$\eta^*, B^* = \arg \min_{\eta, B} \mathcal{L}_{\mathcal{A}, \mathcal{D}, N, D}(\eta, B) \tag{1}$$

Empirically, in log–log coordinates, the optima are described by the power-law family:

$$\eta^*(N, D) = c \cdot N^{\alpha} \cdot D^{\beta}, \qquad B^*(D) = d \cdot D^{\gamma} \tag{2}$$

### Hypotheses

Three pairwise mutually exclusive and exhaustive hypotheses are tested:

- **H1**: Original Step Law coefficients hold directly for small models
- **H2**: Power-law form holds but with different coefficients
- **H3**: No power law describes the optima in this regime

### Experimental Protocol

**Models and data**: nanoGPT-style architecture in standard (non-μP) parameterization, trained on roneneldan/TinyStories with a single in-domain BPE tokenizer (2,048-token vocabulary). Main scale range: $N \in \{255K, 519K, 997K, 2.03M\}$, with two diagnostic anchor cells at $N = 5M$.

**Final grid**: 29 unique $(N, D)$ cells and 935 analysis-ready runs. Main regression uses 25 cells (815 runs) in the working range $4 \leq D/N \leq 600$.

**Table 2: Summary of the final grid**

| Data group | (N,D) cells | Usable runs |
|------------|-------------|-------------|
| All unique cells | 29 | 935 |
| Iteration 1 | 12 | 405 |
| Iteration 2 (excl. N=5M) | 15 | 480 |
| N=5M anchor | 2 | 50 |
| **Main refit** | **25** | **815** |

**Training setup**: Grid search over $(\eta, B)$ per cell; AdamW ($\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay 0.1, gradient clipping 1.0), bf16, warmup–cosine scheduler with warmup = 5% of budget and $\eta_{\min} = 10^{-5}$. Single seed (1337) used throughout.

### Optimum Extraction

The main metric is smoothed training loss:

$$\mathcal{L}_{\text{smooth}}(\eta, B) = \frac{1}{K} \sum_{t = T - K + 1}^{T} \mathcal{L}_t(\eta, B), \quad K = 128 \tag{8}$$

Optima are extracted via a robust 2D-quadratic fit in log–log coordinates: within the window $\mathcal{L} \leq \mathcal{L}_{\min} + 0.15$, fit:

$$\mathcal{L} \approx q(\log \eta, \log B) \tag{9}$$

If the Hessian is positive definite and the vertex lies inside the explored box, the vertex is taken as $(\hat{\eta}, \hat{B})$; otherwise, the observed grid minimum is used.

### Regression Fitting

Power-law coefficients are estimated via OLS in log–log coordinates:

$$\log \hat{\eta} = \log c - \alpha \log N + \beta \log D, \quad \log \hat{B} = \log d + \gamma \log D \tag{10}$$

---

## Empirical Validation / Results

### Direct Transfer of Step Law (H1 Rejected)

Step Law's original formula predicts a peak learning rate that is far too large. Across the 25 main-refit cells, the median ratio $\eta_{SL}/\hat{\eta} \approx 4.0\times$, with a range of 2.4–6.6×.

**Table 3: Overestimation of optimal learning rate by Step Law formula**

| Scale N | Geometric mean η_SL/η̂ | Per-cell range |
|---------|------------------------|----------------|
| 0.25M | 4.97× | 3.9–6.6× |
| 0.52M | 3.83× | 2.8–4.2× |
| 1.0M | 3.69× | 2.4–6.2× |
| 2.0M | 3.28× | 2.8–4.1× |

The overshoot decreases with growing N, consistent with a weaker N-dependence for $\eta^*$ than the original.

### Recalibrated Coefficients (H2 Accepted)

On the 25 main cells, the final refit gives:

$$\hat{\eta}^*(N, D) = 0.0985 N^{-0.508} D^{0.238}, \quad R^2 = 0.834, \text{adj-}R^2 = 0.819 \tag{11}$$

$$\hat{B}^*(D) = 3.6 \times 10^{-4} D^{0.931}, \quad R^2 = 0.950 \tag{12}$$

**Table 4: Final power-law exponents compared with Step Law's coefficients**

| Exponent | Our estimate | 95% bootstrap CI | Step Law | p (difference) |
|----------|--------------|------------------|----------|----------------|
| $\alpha$ in $\eta^* \propto N^{-\alpha}$ | 0.508 | [0.410, 0.606] | 0.713 | $5 \times 10^{-5}$ |
| $\beta$ in $\eta^* \propto D^{\beta}$ | 0.238 | [0.159, 0.300] | 0.307 | 0.051 |
| $\gamma$ in $B^* \propto D^{\gamma}$ | 0.931 | [0.860, 0.994] | 0.571 | $2 \times 10^{-24}$ |

Key observations:
- The N-exponent $\alpha = 0.508$ is significantly shallower than the original 0.713
- The D-exponent for $\eta^*$ ($\beta = 0.238$) is statistically compatible with the original 0.307 at the margin of significance
- The strongest discrepancy is in batch size: $\gamma = 0.931$ vs. 0.571

### Batch Size Independence from N (Structural Claim Reproduced)

Testing the extended model $\log \hat{B} = \log d + a \log N + \gamma \log D$:

$$\hat{B}^* = 4.1 \times 10^{-4} N^{-0.013} D^{0.933}, \qquad R^2 = 0.950 \tag{13}$$

The coefficient on $\log N$ is statistically indistinguishable from zero: $p(\log N) = 0.87$. A hierarchical F-test finds no improvement from adding $\log N$ ($F = 0.03$, $p = 0.87$). The structural part of Step Law therefore reproduces, even though the exponent $\gamma$ on D differs substantially.

### Stability Across Iterations

**Table 5: Stability of coefficients across subsamples**

| Subsample | n | α (on N) | β (on D) | $R^2(\eta)$ | γ | $R^2(B)$ |
|-----------|---|----------|----------|-------------|-----|----------|
| Iteration 1 only | 12 | 0.548 | 0.350 | 0.954 | 1.018 | 0.938 |
| Iteration 2 only (excl. N=5M) | 15 | 0.529 | 0.198 | 0.791 | 0.935 | 0.967 |
| Merged main | 25 | 0.508 | 0.238 | 0.834 | 0.931 | 0.950 |
| All non-N=5M | 27 | 0.550 | 0.274 | 0.871 | - | - |
| Step Law | - | 0.713 | 0.307 | - | 0.571 | - |

The first and second iterations give a consistent picture: $\alpha$ stays around 0.51–0.55, $\gamma$ around 0.93–1.02, and the independence of $B^*$ from N is preserved. The most sensitive quantity is the D-exponent $\beta$ for the learning rate, ranging from 0.20 to 0.35 across subsamples.

### Verdict Summary

**Table 6: Final verdict on the three hypotheses**

| Hyp. | What is tested | Final result | Verdict |
|------|----------------|--------------|---------|
| H1 | Original Step Law coefficients hold at $N < 59M$ | $\eta_{SL}$ overestimates the optimum by a median of ≈4.0×; α and γ differ significantly | **Rejected** |
| H2 | Power-law form holds, coefficients differ | $\eta^*$ well described by power law in N, D ($R^2 = 0.834$); $B^*$ by power law in D ($R^2 = 0.950$); $\log N$ for $B^*$ not significant ($p = 0.87$) | **Accepted** |
| H3 | Power law does not describe small scale | High $R^2$, consistent iterations, reproducible structure of $B^*(D)$ | **Not accepted** |

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Power-law structure is fundamental**: The fact that the power-law form holds at scales 30–250× smaller than the original calibration range suggests that the power-law relationship between optimal hyperparameters and $(N, D)$ is a structural property of language-model optimization, not an artifact of a particular scale regime.

2. **Coefficients are scale-dependent**: The significant differences in exponents (particularly $\alpha$ and $\gamma$) indicate that while the functional form transfers, the specific calibration does not. This has implications for any theory attempting to derive these exponents from first principles.

3. **Gradient noise scale interpretation**: The reproduction of $B^*$'s independence from $N$ is consistent with the gradient noise scale framework [McCandlish et al., 2018], where critical batch size is set by the ratio of gradient noise to gradient norm, which changes primarily with training duration rather than model width.

4. **Comparison with μP**: The shallower N-exponent ($\alpha = 0.508$ vs. 0.713) suggests that at small scale, the optimal learning rate is less sensitive to model width than at large scale, which has implications for hyperparameter transfer approaches.

### Practical Implications

1. **Direct heuristic**: For small LLMs, divide the Step Law-predicted learning rate by approximately four, or use the recalibrated formula $\hat{\eta}^* \approx 0.1 N^{-0.51} D^{0.24}$.

2. **Batch size guidance**: The structural conclusion that $B^*$ is primarily set by the data budget $D$, not model scale $N$, holds — but the slope with respect to $D$ is substantially steeper in the small-scale regime ($\gamma = 0.931$ vs. 0.571), meaning batch size grows nearly linearly with data budget.

3. **Reproducible benchmark**: The paper releases code, configurations, run logs, tables of optima, and refit scripts as an open benchmark at https://github.com/kunikrubika05/step-law-small-scale.

### Hardware Cost Analysis

The paper includes a cost analysis comparing GPUs for the experimental sweep. The cost of processing one million training tokens:

$$c_{1M} = \frac{\pi \cdot t}{3600 \cdot N_{\text{tok}}} \times 10^6 \tag{14}$$

where $\pi$ is the hourly GPU price, $t$ is wall-clock run time, and $N_{\text{tok}}$ is the number of training tokens.

**Table 7: Cost comparison of identical run on A10 vs T4 GPU**

| GPU | π, RUB/h | t, s | $N_{\text{tok}}$ | $c_{1M}$, RUB | Val. loss |
|-----|----------|------|------------------|---------------|-----------|
| A10 | 36.55 | 221.97 | 40,960,000 | 0.055019 | 2.471797 |
| T4 | 21.75 | 1769.49 | 40,960,000 | 0.261002 | 2.470558 |

The A10 is ~7.97× faster than the T4 at a 1.68× difference in hourly price, making it ~4.74× cheaper per token processed, with essentially identical validation loss.

---

## Conclusion

### Main Takeaways

1. **Step Law does not transfer directly to small models**: Direct application of the original coefficients systematically overestimates the optimal learning rate by a median factor of ~4× (range 2.4–6.6×). The N-exponent $\alpha$ and batch-size exponent $\gamma$ differ significantly from the original values.

2. **The power-law form is preserved**: Small scale does not break the power-law structure. $\eta^*$ is well described by a power law in $N$ and $D$ ($R^2 = 0.834$), and $B^*$ by a power law in $D$ alone ($R^2 = 0.950$).

3. **Structural claim reproduced**: Step Law's key structural claim that $B^*$ is independent of $N$ holds at small scale ($p(\log N) = 0.87$), though the growth of $B^*$ with $D$ is nearly twice as steep.

4. **Final verdict is H2**: At small scale, Step Law needs recalibration, but not a different functional class.

### Limitations

- **Single seed**: The experiments use a single seed (1337), so statistical intervals reflect variation across cells rather than initialization noise. Bootstrap CIs should be read as variation across cells, not full confidence intervals over seeds.
- **Limited range**: The main refit covers 0.25–2.03M parameters, well below Step Law's lower bound of 59M. The two $N = 5M$ cells serve only as diagnostic anchors.
- **The D-exponent $\beta$** for the learning rate deserves particular caution: it is statistically compatible with the original value but sits at the margin of significance.

### Future Directions

- Testing intermediate scales (e.g., 5M–59M) to map the transition between the small-scale and large-scale regimes
- Multi-seed experiments to establish genuine confidence intervals
- Exploring whether the recalibrated coefficients hold for other architectures, tokenizers, and data distributions
- Investigating the theoretical mechanisms behind the scale-dependent exponents

---

_Markdown view of https://picx.dev/p/zPJYQS, served by PicX — AI-generated visual whiteboard summaries of research papers._