Summary (Overview)

• Core Question: This paper empirically tests whether Step Law [Li et al., 2025] — a power-law formula for optimal learning rate $\eta^*$ and batch size $B^*$ calibrated on models between 59M and 1B parameters — transfers to small language models with $N < 59M$ parameters.
• Main Finding (H2 Accepted): The power-law functional form holds at small scale, but the coefficients differ significantly from the original Step Law. Direct transfer systematically overestimates the optimal learning rate by a median factor of ~4.0× (range 2.4–6.6×).
• Recalibrated Laws: The authors fit new coefficients: $\hat{\eta}^*(N, D) = 0.0985 N^{-0.508} D^{0.238}$ ($R^2 = 0.834$) and $\hat{B}^*(D) = 3.6 \times 10^{-4} D^{0.931}$ ($R^2 = 0.950$).
• Structural Claim Reproduced: Step Law's claim that $B^*$ is independent of $N$ holds at small scale ($p(\log N) = 0.87$), but the growth of $B^*$ with data budget $D$ is nearly twice as steep ($\gamma = 0.931$ vs. 0.571).
• Practical Recommendation: For small LLMs, divide the Step Law-predicted learning rate by approximately four, or use the recalibrated formula directly.

---

Introduction and Theoretical Foundation

Background and Motivation

Pre-training large language models is among the most expensive procedures in modern machine learning. Scaling laws [Kaplan et al., 2020; Hoffmann et al., 2022] characterize how loss depends on parameter count $N$ and data budget $D$, but are silent on optimization hyperparameters. Step Law [Li et al., 2025] fills this gap by providing:

$$\eta^*(N, D) = 1.79 \cdot N^{-0.713} \cdot D^{0.307}, \qquad B^*(D) = 0.58 \cdot D^{0.571} \tag{7}$$

Calibrated on ~3,700 runs at seven values of $N \in [59M, 1B]$, the region $N < 59M$ was never tested empirically by the authors, who explicitly defer generalization beyond the calibrated range to future work.

Why the Small-Scale Regime Matters

The region $N < 59M$ is practically significant for:

• Edge inference and mobile devices with tight memory and energy budgets
• Interpretability research, where small models are far easier to analyze mechanistically
• Single-GPU training, typical of educational and early-stage projects

There is also a principled reason to distrust naive extrapolation: power laws in machine learning routinely break down at the edges of their calibrated range, and reduced model capacity can change the geometry of the hyperparameter landscape.

Theoretical Foundation

The paper builds on:

• Gradient noise scale theory [McCandlish et al., 2018]: critical batch size $B^*$ is set by the ratio of gradient noise to gradient norm, which changes primarily with training duration $D$ rather than model width $N$.
• Maximal update parameterization (μP) [Yang et al., 2022]: an alternative theory-driven approach where optimal learning rate is invariant to model width, but requiring architectural modifications.
• Small language models [Eldan and Li, 2023]: models with 1–30M parameters trained on narrow, high-quality corpora produce coherent text and display rudimentary reasoning.

Research Gap

No prior work covers the region $N < 59M$ on a standard parameterization with a joint dependence of hyperparameters on both $N$ and $D$ — this is the gap addressed by the paper.

Table 1: Comparison with existing work

---

Methodology

Problem Setup

For fixed architecture $\mathcal{A}$, data distribution $\mathcal{D}$, parameter count $N$, and data budget $D$, the optimal hyperparameters are:

$$\eta^*, B^* = \arg \min_{\eta, B} \mathcal{L}_{\mathcal{A}, \mathcal{D}, N, D}(\eta, B) \tag{1}$$

Empirically, in log–log coordinates, the optima are described by the power-law family:

$$\eta^*(N, D) = c \cdot N^{\alpha} \cdot D^{\beta}, \qquad B^*(D) = d \cdot D^{\gamma} \tag{2}$$

Hypotheses

Three pairwise mutually exclusive and exhaustive hypotheses are tested:

• H1: Original Step Law coefficients hold directly for small models
• H2: Power-law form holds but with different coefficients
• H3: No power law describes the optima in this regime

Experimental Protocol

Models and data: nanoGPT-style architecture in standard (non-μP) parameterization, trained on roneneldan/TinyStories with a single in-domain BPE tokenizer (2,048-token vocabulary). Main scale range: $N \in \{255K, 519K, 997K, 2.03M\}$, with two diagnostic anchor cells at $N = 5M$.

Final grid: 29 unique $(N, D)$ cells and 935 analysis-ready runs. Main regression uses 25 cells (815 runs) in the working range $4 \leq D/N \leq 600$.

Table 2: Summary of the final grid

Training setup: Grid search over $(\eta, B)$ per cell; AdamW ($\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay 0.1, gradient clipping 1.0), bf16, warmup–cosine scheduler with warmup = 5% of budget and $\eta_{\min} = 10^{-5}$. Single seed (1337) used throughout.

Optimum Extraction

The main metric is smoothed training loss:

$$\mathcal{L}_{\text{smooth}}(\eta, B) = \frac{1}{K} \sum_{t = T - K + 1}^{T} \mathcal{L}_t(\eta, B), \quad K = 128 \tag{8}$$

Optima are extracted via a robust 2D-quadratic fit in log–log coordinates: within the window $\mathcal{L} \leq \mathcal{L}_{\min} + 0.15$, fit:

$$\mathcal{L} \approx q(\log \eta, \log B) \tag{9}$$

If the Hessian is positive definite and the vertex lies inside the explored box, the vertex is taken as $(\hat{\eta}, \hat{B})$; otherwise, the observed grid minimum is used.

Regression Fitting

Power-law coefficients are estimated via OLS in log–log coordinates:

$$\log \hat{\eta} = \log c - \alpha \log N + \beta \log D, \quad \log \hat{B} = \log d + \gamma \log D \tag{10}$$

---

Empirical Validation / Results

Direct Transfer of Step Law (H1 Rejected)

Step Law's original formula predicts a peak learning rate that is far too large. Across the 25 main-refit cells, the median ratio $\eta_{SL}/\hat{\eta} \approx 4.0\times$, with a range of 2.4–6.6×.

Table 3: Overestimation of optimal learning rate by Step Law formula

The overshoot decreases with growing N, consistent with a weaker N-dependence for $\eta^*$ than the original.

Recalibrated Coefficients (H2 Accepted)

On the 25 main cells, the final refit gives:

$$\hat{\eta}^*(N, D) = 0.0985 N^{-0.508} D^{0.238}, \quad R^2 = 0.834, \text{adj-}R^2 = 0.819 \tag{11}$$ $$\hat{B}^*(D) = 3.6 \times 10^{-4} D^{0.931}, \quad R^2 = 0.950 \tag{12}$$

Table 4: Final power-law exponents compared with Step Law's coefficients

Key observations:

• The N-exponent $\alpha = 0.508$ is significantly shallower than the original 0.713
• The D-exponent for $\eta^*$ ($\beta = 0.238$) is statistically compatible with the original 0.307 at the margin of significance
• The strongest discrepancy is in batch size: $\gamma = 0.931$ vs. 0.571

Batch Size Independence from N (Structural Claim Reproduced)

Testing the extended model $\log \hat{B} = \log d + a \log N + \gamma \log D$:

$$\hat{B}^* = 4.1 \times 10^{-4} N^{-0.013} D^{0.933}, \qquad R^2 = 0.950 \tag{13}$$

The coefficient on $\log N$ is statistically indistinguishable from zero: $p(\log N) = 0.87$. A hierarchical F-test finds no improvement from adding $\log N$ ($F = 0.03$, $p = 0.87$). The structural part of Step Law therefore reproduces, even though the exponent $\gamma$ on D differs substantially.

Stability Across Iterations

Table 5: Stability of coefficients across subsamples

The first and second iterations give a consistent picture: $\alpha$ stays around 0.51–0.55, $\gamma$ around 0.93–1.02, and the independence of $B^*$ from N is preserved. The most sensitive quantity is the D-exponent $\beta$ for the learning rate, ranging from 0.20 to 0.35 across subsamples.

Verdict Summary

Table 6: Final verdict on the three hypotheses

---

Theoretical and Practical Implications

Theoretical Implications

1. Power-law structure is fundamental: The fact that the power-law form holds at scales 30–250× smaller than the original calibration range suggests that the power-law relationship between optimal hyperparameters and $(N, D)$ is a structural property of language-model optimization, not an artifact of a particular scale regime.

2. Coefficients are scale-dependent: The significant differences in exponents (particularly $\alpha$ and $\gamma$) indicate that while the functional form transfers, the specific calibration does not. This has implications for any theory attempting to derive these exponents from first principles.

3. Gradient noise scale interpretation: The reproduction of $B^*$'s independence from $N$ is consistent with the gradient noise scale framework [McCandlish et al., 2018], where critical batch size is set by the ratio of gradient noise to gradient norm, which changes primarily with training duration rather than model width.

4. Comparison with μP: The shallower N-exponent ($\alpha = 0.508$ vs. 0.713) suggests that at small scale, the optimal learning rate is less sensitive to model width than at large scale, which has implications for hyperparameter transfer approaches.

Practical Implications

1. Direct heuristic: For small LLMs, divide the Step Law-predicted learning rate by approximately four, or use the recalibrated formula $\hat{\eta}^* \approx 0.1 N^{-0.51} D^{0.24}$.

2. Batch size guidance: The structural conclusion that $B^*$ is primarily set by the data budget $D$, not model scale $N$, holds — but the slope with respect to $D$ is substantially steeper in the small-scale regime ($\gamma = 0.931$ vs. 0.571), meaning batch size grows nearly linearly with data budget.

3. Reproducible benchmark: The paper releases code, configurations, run logs, tables of optima, and refit scripts as an open benchmark at https://github.com/kunikrubika05/step-law-small-scale.

Hardware Cost Analysis

The paper includes a cost analysis comparing GPUs for the experimental sweep. The cost of processing one million training tokens:

$$c_{1M} = \frac{\pi \cdot t}{3600 \cdot N_{\text{tok}}} \times 10^6 \tag{14}$$

where $\pi$ is the hourly GPU price, $t$ is wall-clock run time, and $N_{\text{tok}}$ is the number of training tokens.

Table 7: Cost comparison of identical run on A10 vs T4 GPU

The A10 is ~7.97× faster than the T4 at a 1.68× difference in hourly price, making it ~4.74× cheaper per token processed, with essentially identical validation loss.

---

Conclusion

Main Takeaways

1. Step Law does not transfer directly to small models: Direct application of the original coefficients systematically overestimates the optimal learning rate by a median factor of ~4× (range 2.4–6.6×). The N-exponent $\alpha$ and batch-size exponent $\gamma$ differ significantly from the original values.

2. The power-law form is preserved: Small scale does not break the power-law structure. $\eta^*$ is well described by a power law in $N$ and $D$ ($R^2 = 0.834$), and $B^*$ by a power law in $D$ alone ($R^2 = 0.950$).

3. Structural claim reproduced: Step Law's key structural claim that $B^*$ is independent of $N$ holds at small scale ($p(\log N) = 0.87$), though the growth of $B^*$ with $D$ is nearly twice as steep.

4. Final verdict is H2: At small scale, Step Law needs recalibration, but not a different functional class.

Limitations

• Single seed: The experiments use a single seed (1337), so statistical intervals reflect variation across cells rather than initialization noise. Bootstrap CIs should be read as variation across cells, not full confidence intervals over seeds.
• Limited range: The main refit covers 0.25–2.03M parameters, well below Step Law's lower bound of 59M. The two $N = 5M$ cells serve only as diagnostic anchors.
• The D-exponent $\beta$ for the learning rate deserves particular caution: it is statistically compatible with the original value but sits at the margin of significance.

Future Directions

• Testing intermediate scales (e.g., 5M–59M) to map the transition between the small-scale and large-scale regimes
• Multi-seed experiments to establish genuine confidence intervals
• Exploring whether the recalibrated coefficients hold for other architectures, tokenizers, and data distributions
• Investigating the theoretical mechanisms behind the scale-dependent exponents