Summary (Overview)

• Core Finding: The paper uncovers "ELR collapse" in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR), defined as $\eta_k^{\mathrm{eff}} := \eta_k / \|\mathbf{W}_k\|_F$.
• Quantitative Precision: When ELR schedules are matched across runs with substantially different LRs and norms, loss trajectories collapse with mean discrepancies of only a few $\times 10^{-3}$—well below seed-to-seed variation (~$1.1 \times 10^{-2}$ to $1.6 \times 10^{-2}$) and the $10^{-2}$-scale differences used in optimizer studies.
• Mechanistic Mediation: Weight decay and Hyperball (explicit norm constraint) affect loss dynamics primarily through the ELR schedules they induce, not as independent controls. LR-only interventions that match target ELR recover target loss trajectories with errors of $4.8 \times 10^{-3}$ and $1.2 \times 10^{-3}$.
• Scaling Law Transfer: Replacing LR with ELR in functional scaling laws (FSL) enables transfer across norm-control methods, reducing out-of-distribution prediction error by 11.83×.
• Explanatory Power: The ELR framework explains "delayed acceleration"—the phenomenon where norm-controlled runs initially have higher loss but later overtake baselines—as a temporal separation between when learning gains are acquired (early) and when they are revealed (late).

---

Introduction and Theoretical Foundation

Background and Motivation

Training dynamics in LLM pretraining are governed by hyperparameters like LR and batch size, but norm control provides another axis—encompassing implicit mechanisms (weight decay) and explicit constraints (Hyperball, SSO). Norm control improves training stability and hyperparameter transfer but also reshapes loss dynamics. This raises the central question:

Does norm control introduce an independent degree of freedom governing loss dynamics?

Theoretical Basis: Scale Invariance and ELR

The ELR concept originates from scale-invariant objectives where $\mathcal{L}(c\mathbf{w}) = \mathcal{L}(\mathbf{w})$ for all $c > 0$. For such objectives:

• For gradient descent: ELR is $\eta_k / \|\mathbf{W}_k\|^2$
• For scale-insensitive optimizers (signSGD, Adam): ELR is $\eta_k / \|\mathbf{W}_k\|$

However, the authors emphasize that transformers are not exactly scale invariant. In a pre-norm residual block:

$$\mathbf{h}^{\ell+1} = \mathbf{h}^{\ell} + \operatorname{FFN}\left(\operatorname{Norm}\left(\mathbf{h}^{\ell}\right)\right)$$

rescaling $\mathbf{h}^\ell$ affects identity and residual branches differently. Learnable normalization gains, nonhomogeneous activations, and embedding/output layers introduce further scale dependence. Thus, ELR collapse is not implied by exact reparameterization symmetry.

Key Definitions

The effective learning rate at step $k$:

$$\eta_k^{\mathrm{eff}} := \frac{\eta_k}{\|\mathbf{W}_k\|_F}, \tag{1}$$

Loss residual and mean collapse error:

$$r_k := L_k - L_k^{\mathrm{ref}}, \qquad \Delta_{\mathrm{coll}} := \frac{1}{|\mathcal{T}|} \sum_{k \in \mathcal{T}} |r_k|, \tag{2}$$

---

Methodology

ELR Matching Protocol

For a reference run A with LR $\{\eta_k^A\}$ and norm evolution $R_k^A := \|\mathbf{W}_k^A\|_F$, inducing ELR schedule $\gamma_k := \eta_k^A / R_k^A$:

1. Choose a different LR schedule $\{\eta_k^B\}$ for run B
2. Set target norm $R_k^B := \eta_k^B / \gamma_k$
3. Enforce $\|\mathbf{W}_k^B\|_F = R_k^B$ at every optimization step

This ensures runs A and B have different LR and norm schedules but identical ELR schedules.

Norm Control Methods

Decoupled weight decay:

$$\mathbf{W}_{k+1} = (1 - \eta_k \lambda) \mathbf{W}_k - \eta_k \mathbf{U}_k$$

Hyperball (explicit norm constraint):

$$\mathbf{W}_{k+1} = \operatorname{Norm}_R(\mathbf{W}_k - \eta_k \operatorname{Norm}_R(\mathbf{U}_k)), \qquad \operatorname{Norm}_R(\mathbf{Q}) := R \frac{\mathbf{Q}}{\|\mathbf{Q}\|_F}$$

Experimental Configurations

• Models: Llama (dense), Qwen3 (dense), Qwen3-MoE, Kimi Delta Attention (linear attention), 100M–1B parameters
• Datasets: FineWeb, C4, OpenWebText
• Optimizers: AdamW, Muon, Signum
• Ablations: QK-Norm removal, fixed vs. learnable RMSNorm gains, sinusoidal LR–norm modulation at varying frequencies

Functional Scaling Laws (FSL)

Given LR schedule $\boldsymbol{\eta} = (\eta_0, \dots, \eta_N)$, the intrinsic training time is $t_k := \sum_{j=0}^k \eta_j$. FSL models:

$$L_k = L_{\infty} + S(t_k) + N_k, \qquad N_k = \sum_{j=0}^k K(t_k - t_j) \eta_j^2, \tag{3}$$

where $S(t) \asymp t^{-s}$ and $K(t) \asymp t^{-\gamma}$. elr-FSL replaces $\eta_j$ with $\eta_j^{\mathrm{eff}}$ throughout.

---

Empirical Validation / Results

Central Result: ELR Collapse

Across 26 ELR-matched comparisons, the median collapse error is $2.5 \times 10^{-3}$, with all comparisons below $5 \times 10^{-3}$. Representative results:

$$\begin{array}{rl} \mathrm{Llama(124M):} & \Delta_{\mathrm{coll}} = (1.8, 2.5, 2.6) \times 10^{-3}, \\ \mathrm{Qwen3-MoE(586M):} & \Delta_{\mathrm{coll}} = (3.0, 4.1, 3.3) \times 10^{-3}. \end{array}$$

Conditions for High-Precision Collapse

1. QK-Norm: Removing QK-Norm increases collapse error from $2.3 \times 10^{-3}$ to $5.2 \times 10^{-3}$

2. Learnable RMSNorm gains: Fixing gains (making parameterization more scale invariant) increases error to $1.84 \times 10^{-2}$—a ~3.5× degradation

3. Timescale of LR–norm variation: Modulating LR and norm jointly with sinusoidal perturbations preserves ELR but degrades collapse as frequency increases:

Mediation through ELR

• Weight decay (λ=0.1 target vs. λ=0 with adapted LR): collapse error of $4.8 \times 10^{-3}$
• Hyperball (MuonH target vs. MuonW with adapted LR): collapse error of $1.2 \times 10^{-3}$ (evaluated from step 7.5k due to initial transient)

Scaling Law Transfer

Delayed Acceleration

The ELR mechanism explains why weight decay initially yields higher loss but later overtakes baselines:

• Without weight decay, norm growth causes ELR to decay faster than nominal LR
• Weight decay sustains larger ELR longer, accumulating effective training time $t_k^{\mathrm{eff}} = \sum_{j=0}^k \eta_j^{\mathrm{eff}}$ more rapidly
• The signal-learning advantage is acquired early but revealed late as noise injection diminishes

---

Theoretical and Practical Implications

Theoretical Significance

1. ELR as a low-dimensional coordinate: Despite the high-dimensional, coupled dynamics of Transformer training, loss dynamics admit an accurate low-dimensional description through ELR—analogous to how density (mass/volume ratio) governs macroscopic physical behavior.

2. Not a consequence of scale invariance: The counterintuitive finding that learnable RMSNorm gains improve collapse despite making parameterization less scale invariant rules out static symmetry as the explanation, suggesting a dynamical compensation mechanism with finite response timescale.

3. Conditional validity: ELR collapse is precise but not exact—precision depends on normalization design and the timescale of LR–norm variation.

Practical Implications

1. ELR-first pretraining: The ELR schedule should be the design object; LR scheduling and norm control are mechanisms for realizing it. This separates two conflated questions: what effective trajectory should training follow and how should that trajectory be realized.

2. Cross-scale transfer: Hyperparameter transfer becomes a two-stage problem: (a) determine the desired ELR schedule for the target scale, (b) choose a stable LR–norm realization. This replaces method-specific heuristics with a common dynamical target.

3. Norm control as ELR shaping: Norm control is useful not because a particular norm is intrinsically preferable, but because it expands the set of realizable ELR schedules. The explicit norm-control experiment shows that increasing parameter norm more rapidly late in training can improve final loss by producing more favorable ELR decay.

---

Conclusion

Main Takeaways

1. ELR collapse is a precise macroscopic law: LR and parameter norm govern loss dynamics primarily through their ratio, with collapse errors of a few $\times 10^{-3}$ across optimizers, architectures, datasets, and model scales.

2. Norm control acts through ELR: Weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce, not as independent controls.

3. ELR enables scaling law transfer: Parameterizing FSL with ELR instead of LR enables transfer across norm-control methods, reducing OOD prediction error by 11.83×.

4. Delayed acceleration is explained: The late-emerging benefit of norm control reflects temporal separation—gains are acquired early but revealed late as noise diminishes.

Limitations and Future Directions

• Scope limited to loss dynamics: Matched loss trajectories need not imply matched parameters, representations, or downstream performance
• Microscopic mechanism remains open: A satisfactory theory should explain why such an accurate low-dimensional description exists and why precision depends on normalization design and variation timescale
• Future work: Determine how the desired ELR schedule changes with model size, batch size, data budget, and training horizon; stability, numerical precision, and transferability remain additional constraints on ELR schedule realization