Summary (Overview)

• This paper presents an extensive empirical study of scaling laws for learning rate (LR) and batch size in pretraining dense decoder-only LLMs, spanning models from 47M to 1.7B parameters and up to 300B tokens.
• The authors derive jointly optimal scaling laws for LR and batch size, as well as marginal scaling laws for optimal LR at arbitrary (fixed) batch sizes—a practical scenario when hardware constraints limit batch size.
• Using a Warmup-Stable-Decay (WSD) LR schedule, they show that LR annealing's loss improvement is not constant across model sizes, token budgets, LRs, and batch sizes, and that optimal hyperparameters shift between stable and decay phases.
• They demonstrate that the Chinchilla scaling law fails to capture the interaction between model size $N$ and data size $D$, while the recently proposed Skaling law provides substantially better fits and extrapolation to held-out 1.7B models (RMSE: 0.0056 vs. 0.0174).
• The complete collection of pretraining runs, checkpoints, and evaluations is open-sourced to foster reproducible scaling law research.

---

Introduction and Theoretical Foundation

Motivation. Pretraining LLMs at scale requires informed decisions about how models, data, and training configurations should grow. Two critical components are:

1. Learning rate and batch size selection—affects training efficiency and the accuracy of derived scaling laws.
2. Scaling laws for loss prediction—used to predict performance at larger scales, derive compute-optimal boundaries, and compare models/datasets.

Prior work. Early work focused on the critical batch size (McCandlish et al., 2018; Merrill et al., 2025; Bergsma et al., 2025). More recent approaches investigate how optimal batch size scales with parameters, data, and compute (Hu et al., 2024; DeepSeek-AI, 2024; Li et al., 2025b; Rütte et al., 2026). LR scaling has been studied via muP (Yang et al., 2021) and through direct scaling-law fits (Shen et al., 2024; Porian et al., 2025).

Open questions addressed:

• How do optimal LR and batch size scale jointly and marginally with model size $N$ and data budget $D$?
• Does LR annealing shift optimal hyperparameters, and is its benefit constant across configurations?
• Can recently proposed scaling forms (e.g., Skaling) better capture the $N$–$D$ interaction than the classical Chinchilla law?

---

Methodology

Data. Training uses the high-quality subset of the Nemotron-CC dataset (Su et al., 2025), with the GPT-NeoX-20B tokenizer. Experiments span up to 300B tokens. Validation loss is measured on a held-out set of 204,800 sequences (0.838B tokens).

Models. Dense decoder-only transformers from 47.6M to 1.713B parameters, with:

• Vocabulary size 50304, FFN expansion factor 4, GLU activations
• QK normalization, attention bias terms, tied input-output embeddings
• Fixed head dimension 64, geometrically spaced parameter counts

Training configuration. AdamW optimizer with a WSD (trapezoidal) LR schedule. Intermediate decays at $D = 6, 12, 20, 30, 50, 80, 120, 200$ BT, decaying over the final 20% of each budget to a final LR of $10^{-5}$. Fixed warmup of 2000 steps. Training on AMD Instinct MI250X GPUs with Megatron-LM.

Hyperparameter grid.

• Batch sizes $b$ log₂-spaced in $[2^4, 2^{10}]$
• Learning rates $\eta \in \{0.00025, 0.0005, 0.001, 0.002, 0.004\}$
• Combinations where warmup exceeds 30% of $D$ are excluded

Loss smoothing. To obtain reliable optima estimates, the loss surface is approximated by a quadratic function of $\eta$ and $b$:

$$L(\eta, b | N, D) = \beta_{0} + \beta_{1} \eta + \beta_{2} \eta^{2} + \beta_{3} b + \beta_{4} b^{2} + \beta_{5} \eta b + \epsilon, \quad \forall N, D.\tag{1}$$

Optimal hyperparameters are predicted via:

$$\eta^{\star}, b^{\star} | N, D = \underset{\eta, b}{\arg\min} \hat{L}(\eta, b | N, D), \qquad \forall N, D.\tag{2}$$

Conditional optima (fixing one hyperparameter) are obtained from:

$$\eta^{\star} | b, N, D = \underset{\eta}{\arg\min} \hat{L}(\eta, b | N, D); \qquad b^{\star} | \eta, N, D = \underset{b}{\arg\min} \hat{L}(\eta, b | N, D).\tag{3}$$

---

Empirical Validation / Results

3.2 Jointly Optimal LR and Batch Size

Power-law models are fit in log-space:

$$\log b^{\star}(N, D) = \alpha_{1} + \beta_{1} \log N + \gamma_{1} \log D + \varepsilon_{1},\tag{4}$$ $$\log \eta^{\star}(N, D) = \alpha_{2} + \beta_{2} \log N + \gamma_{2} \log D + \varepsilon_{2}.\tag{5}$$

Batch size: Explains 93.6% of variance. Strong positive scaling with $D$ (exponent 0.46), weak but significant positive effect of $N$ (exponent 0.145):

$$\hat{b}^{\star} | N, D = 0.000099 \, N^{0.145} D^{0.460}.\tag{6}$$

Learning rate: Explains 79.2% of variance. Strong negative scaling with $N$ (exponent −0.417), weaker positive dependence on $D$ (exponent 0.0862, bootstrap p-value 0.0055):

$$\hat{\eta}^{\star} | N, D = 0.371396 \, N^{-0.417} D^{0.0862}.\tag{7}$$

Comparison with prior work:

Held-out validation (1.7B model): MAPEs of 28% (η) and 10% (b) against smoothed optima, with all predictions within parametric prediction intervals.

3.3 Optimal LR at Arbitrary Batch Size

The full model with interaction terms:

$$\begin{array}{c} \log \eta^{\star} = \alpha_{0} + \alpha_{1} \log b + \\ \beta_{0} \log N + \beta_{1} \log b \log N + \\ \gamma_{0} \log D + \gamma_{1} \log b \log D + \\ \delta_{0} \log N \log D + \delta_{1} \log b \log N \log D + \epsilon. \end{array}\tag{8}$$

Key findings:

• Optimal LR scales strongly with $N$ ($\hat{\beta}_0 \approx -0.54$) and more weakly with $D$ ($\hat{\gamma}_0 \approx -0.25$)
• Larger batch sizes shift optimal LR upward (exponent 0.80 on $b$)
• Opposite D-trend vs. joint scaling: when $b$ is fixed, larger $D$ decreases optimal LR; when jointly scaled, larger $D$ increases it

Reduced model after removing non-significant terms:

$$\log \eta^{\star} = -6.60 + 0.80 \log b - 0.53 \log N - 0.24 \log D - 0.06 \log N \log D + 0.08 \log D \log b.\tag{9}$$

4 LR Annealing Effects

Loss improvement is configuration-dependent (Figure 8):

• Larger $\eta$ and smaller $b$ → larger loss improvements from decay (more optimization noise benefits from annealing)
• Improvement increases later in training (larger $D$)
• Larger models $N$ show consistently larger improvements

Optimal hyperparameters shift after decay:

• $L_{decay}$ favors larger learning rates (up to 4× larger than stable-phase optima)
• Stable-phase optima consistently sit at the smallest LR in the grid
• This suggests that higher LR (or smaller $b$) during stable phase aids exploration, which is then corrected by annealing

5 Scaling Laws for Loss

Chinchilla law (independent power-law contributions):

$$L(N, D) = E + A N^{-\alpha} + B D^{-\beta}.\tag{10}$$

Skaling law (couples $N$ and $D$ via interaction exponent $k$):

$$L(N, D) = E + \left(A N^{-\alpha} + B D^{-\beta}\right)^{k}.\tag{11}$$

Fitted parameters:

Key results:

• Chinchilla fails at extreme $N$ and $D$ values; residuals show clear $N$–$D$ interaction
• Skaling's interaction exponent $k \approx 0.41$ matches Videau et al. [2026]
• Skaling extrapolates to held-out 1.7B with 3× lower RMSE (0.0056 vs. 0.0174)

Compute-optimal scaling (derived from Eq. 11):

$$L^{\star}(C) = 1.030 + 60.894 \, C^{-0.084}.\tag{12}$$

6 Downstream Performance

• Consistent decrease in downstream error (DCLM-CORE suite) with increasing compute across all model scales
• Smallest models plateau quickly (capacity-limited); larger models follow the overall compute-performance trend
• OpenEuroLLM models show better compute efficiency than Pythia and SmolLM2 at matched compute

---

Theoretical and Practical Implications

Theoretical implications:

• Joint vs. marginal scaling differ fundamentally: the sign of the $D$-exponent for optimal LR flips depending on whether $b$ is jointly optimized or fixed. This highlights the need to distinguish these settings when deriving or applying scaling laws.
• Interaction between $N$ and $D$ is real and significant: the Chinchilla law's separability assumption is violated; the Skaling law's interaction exponent provides a principled way to capture undertrained and overtrained regimes.
• LR annealing is not a constant shift: the benefit of decay depends systematically on $\eta$, $b$, $N$, and $D$, challenging the assumption (Rütte et al., 2026) that optima are unaffected by annealing.

Practical implications:

• Batch-size-constrained training: when hardware dictates a batch size away from the joint optimum, Equation 9 provides the correct LR prescription.
• WSD schedule design: the finding that decay favors larger stable-phase LRs suggests a practical recipe: use larger LR during stable phase, then anneal to correct for optimization noise.
• Compute savings: understanding annealing effects could enable cheaper scaling-law derivation without full decay runs (decay runs are ~78.6% of total compute in this setup).
• Open-sourced resources: the full collection of runs, checkpoints, and evaluations enables reproducible scaling-law research.

---

Conclusion

This study establishes a first scaling baseline for OpenEuroLLM models, covering optimal LR and batch size scaling (both joint and marginal), the effect of LR annealing under WSD schedules, and loss scaling laws.

Main takeaways:

1. Optimal batch size scales as $b^{\star} \propto N^{0.145} D^{0.460}$; optimal LR as $\eta^{\star} \propto N^{-0.417} D^{0.0862}$.
2. When batch size is fixed, optimal LR scales as $\eta^{\star} \propto b^{0.80} N^{-0.53} D^{-0.24}$ (with interaction terms), showing an opposite $D$-trend versus joint scaling.
3. LR decay benefits vary systematically across configurations, and optimal hyperparameters shift between stable and decay phases—larger LRs are favored when annealing is applied.
4. The Skaling law with interaction exponent $k \approx 0.41$ substantially outperforms Chinchilla for extrapolation to held-out scales.

Limitations and future directions:

• Extrapolation beyond ~1 order of magnitude in scale remains uncertain
• Scaling exponents vary substantially across studies (datasets, architectures, configurations), limiting transferability
• Future work: modeling the evolution of LR-decay benefit to enable cheaper scaling-law derivation without annealing; exploring non-parametric approaches beyond the Skaling form