# Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

> Learning rate and parameter norm govern language model pretraining loss dynamics almost entirely through their ratio, the effective learning rate, collapsing loss trajectories across diverse training configurations.

- **Source:** [arXiv](https://arxiv.org/abs/2608.24814)
- **Published:** 2026-09-05
- **Permalink:** https://picx.dev/p/06lXva
- **Whiteboard:** https://picx.dev/p/06lXva/image

## Summary

## Summary (Overview)

- **Core Finding**: The paper uncovers "ELR collapse" in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the **effective learning rate (ELR)**, defined as $\eta_k^{\mathrm{eff}} := \eta_k / \|\mathbf{W}_k\|_F$.
- **Quantitative Precision**: When ELR schedules are matched across runs with substantially different LRs and norms, loss trajectories collapse with mean discrepancies of only a few $\times 10^{-3}$—well below seed-to-seed variation (~$1.1 \times 10^{-2}$ to $1.6 \times 10^{-2}$) and the $10^{-2}$-scale differences used in optimizer studies.
- **Mechanistic Mediation**: Weight decay and Hyperball (explicit norm constraint) affect loss dynamics primarily through the ELR schedules they induce, not as independent controls. LR-only interventions that match target ELR recover target loss trajectories with errors of $4.8 \times 10^{-3}$ and $1.2 \times 10^{-3}$.
- **Scaling Law Transfer**: Replacing LR with ELR in functional scaling laws (FSL) enables transfer across norm-control methods, reducing out-of-distribution prediction error by 11.83×.
- **Explanatory Power**: The ELR framework explains "delayed acceleration"—the phenomenon where norm-controlled runs initially have higher loss but later overtake baselines—as a temporal separation between when learning gains are acquired (early) and when they are revealed (late).

---

## Introduction and Theoretical Foundation

### Background and Motivation

Training dynamics in LLM pretraining are governed by hyperparameters like LR and batch size, but **norm control** provides another axis—encompassing implicit mechanisms (weight decay) and explicit constraints (Hyperball, SSO). Norm control improves training stability and hyperparameter transfer but also reshapes loss dynamics. This raises the central question:

> Does norm control introduce an independent degree of freedom governing loss dynamics?

### Theoretical Basis: Scale Invariance and ELR

The ELR concept originates from **scale-invariant objectives** where $\mathcal{L}(c\mathbf{w}) = \mathcal{L}(\mathbf{w})$ for all $c > 0$. For such objectives:
- For gradient descent: ELR is $\eta_k / \|\mathbf{W}_k\|^2$
- For scale-insensitive optimizers (signSGD, Adam): ELR is $\eta_k / \|\mathbf{W}_k\|$

However, the authors emphasize that **transformers are not exactly scale invariant**. In a pre-norm residual block:

$$
\mathbf{h}^{\ell+1} = \mathbf{h}^{\ell} + \operatorname{FFN}\left(\operatorname{Norm}\left(\mathbf{h}^{\ell}\right)\right)
$$

rescaling $\mathbf{h}^\ell$ affects identity and residual branches differently. Learnable normalization gains, nonhomogeneous activations, and embedding/output layers introduce further scale dependence. Thus, **ELR collapse is not implied by exact reparameterization symmetry**.

### Key Definitions

The **effective learning rate** at step $k$:

$$
\eta_k^{\mathrm{eff}} := \frac{\eta_k}{\|\mathbf{W}_k\|_F}, \tag{1}
$$

**Loss residual and mean collapse error**:

$$
r_k := L_k - L_k^{\mathrm{ref}}, \qquad \Delta_{\mathrm{coll}} := \frac{1}{|\mathcal{T}|} \sum_{k \in \mathcal{T}} |r_k|, \tag{2}
$$

---

## Methodology

### ELR Matching Protocol

For a reference run A with LR $\{\eta_k^A\}$ and norm evolution $R_k^A := \|\mathbf{W}_k^A\|_F$, inducing ELR schedule $\gamma_k := \eta_k^A / R_k^A$:

1. Choose a different LR schedule $\{\eta_k^B\}$ for run B
2. Set target norm $R_k^B := \eta_k^B / \gamma_k$
3. Enforce $\|\mathbf{W}_k^B\|_F = R_k^B$ at every optimization step

This ensures runs A and B have different LR and norm schedules but identical ELR schedules.

### Norm Control Methods

**Decoupled weight decay**:

$$
\mathbf{W}_{k+1} = (1 - \eta_k \lambda) \mathbf{W}_k - \eta_k \mathbf{U}_k
$$

**Hyperball** (explicit norm constraint):

$$
\mathbf{W}_{k+1} = \operatorname{Norm}_R(\mathbf{W}_k - \eta_k \operatorname{Norm}_R(\mathbf{U}_k)), \qquad \operatorname{Norm}_R(\mathbf{Q}) := R \frac{\mathbf{Q}}{\|\mathbf{Q}\|_F}
$$

### Experimental Configurations

- **Models**: Llama (dense), Qwen3 (dense), Qwen3-MoE, Kimi Delta Attention (linear attention), 100M–1B parameters
- **Datasets**: FineWeb, C4, OpenWebText
- **Optimizers**: AdamW, Muon, Signum
- **Ablations**: QK-Norm removal, fixed vs. learnable RMSNorm gains, sinusoidal LR–norm modulation at varying frequencies

### Functional Scaling Laws (FSL)

Given LR schedule $\boldsymbol{\eta} = (\eta_0, \dots, \eta_N)$, the intrinsic training time is $t_k := \sum_{j=0}^k \eta_j$. FSL models:

$$
L_k = L_{\infty} + S(t_k) + N_k, \qquad N_k = \sum_{j=0}^k K(t_k - t_j) \eta_j^2, \tag{3}
$$

where $S(t) \asymp t^{-s}$ and $K(t) \asymp t^{-\gamma}$. **elr-FSL** replaces $\eta_j$ with $\eta_j^{\mathrm{eff}}$ throughout.

---

## Empirical Validation / Results

### Central Result: ELR Collapse

Across 26 ELR-matched comparisons, the **median collapse error is $2.5 \times 10^{-3}$**, with all comparisons below $5 \times 10^{-3}$. Representative results:

$$
\begin{array}{rl}
\mathrm{Llama(124M):} & \Delta_{\mathrm{coll}} = (1.8, 2.5, 2.6) \times 10^{-3}, \\
\mathrm{Qwen3-MoE(586M):} & \Delta_{\mathrm{coll}} = (3.0, 4.1, 3.3) \times 10^{-3}.
\end{array}
$$

### Conditions for High-Precision Collapse

1. **QK-Norm**: Removing QK-Norm increases collapse error from $2.3 \times 10^{-3}$ to $5.2 \times 10^{-3}$

2. **Learnable RMSNorm gains**: Fixing gains (making parameterization *more* scale invariant) increases error to $1.84 \times 10^{-2}$—a ~3.5× degradation

3. **Timescale of LR–norm variation**: Modulating LR and norm jointly with sinusoidal perturbations preserves ELR but degrades collapse as frequency increases:

| Modulation Cycles | Collapse Error |
|-------------------|----------------|
| 2 | $2.8 \times 10^{-3}$ |
| 32 | $7.5 \times 10^{-3}$ |

### Mediation through ELR

- **Weight decay** (λ=0.1 target vs. λ=0 with adapted LR): collapse error of $4.8 \times 10^{-3}$
- **Hyperball** (MuonH target vs. MuonW with adapted LR): collapse error of $1.2 \times 10^{-3}$ (evaluated from step 7.5k due to initial transient)

### Scaling Law Transfer

| Evaluation | # Runs | lr-FSL RMSE ↓ | elr-FSL RMSE ↓ | RMSE ratio ↑ |
|------------|--------|---------------|----------------|--------------|
| Fit | 4 | 0.0183 | 0.0131 | 1.40× |
| ID: held-out | 4 | 0.0239 | 0.0133 | 1.80× |
| OOD: Hyperball | 2 | 0.2508 | 0.0212 | **11.83×** |

### Delayed Acceleration

The ELR mechanism explains why weight decay initially yields higher loss but later overtakes baselines:
- Without weight decay, norm growth causes ELR to decay faster than nominal LR
- Weight decay sustains larger ELR longer, accumulating effective training time $t_k^{\mathrm{eff}} = \sum_{j=0}^k \eta_j^{\mathrm{eff}}$ more rapidly
- The signal-learning advantage is **acquired early but revealed late** as noise injection diminishes

---

## Theoretical and Practical Implications

### Theoretical Significance

1. **ELR as a low-dimensional coordinate**: Despite the high-dimensional, coupled dynamics of Transformer training, loss dynamics admit an accurate low-dimensional description through ELR—analogous to how density (mass/volume ratio) governs macroscopic physical behavior.

2. **Not a consequence of scale invariance**: The counterintuitive finding that learnable RMSNorm gains *improve* collapse despite making parameterization *less* scale invariant rules out static symmetry as the explanation, suggesting a dynamical compensation mechanism with finite response timescale.

3. **Conditional validity**: ELR collapse is precise but not exact—precision depends on normalization design and the timescale of LR–norm variation.

### Practical Implications

1. **ELR-first pretraining**: The ELR schedule should be the design object; LR scheduling and norm control are mechanisms for realizing it. This separates two conflated questions: *what effective trajectory should training follow* and *how should that trajectory be realized*.

2. **Cross-scale transfer**: Hyperparameter transfer becomes a two-stage problem: (a) determine the desired ELR schedule for the target scale, (b) choose a stable LR–norm realization. This replaces method-specific heuristics with a common dynamical target.

3. **Norm control as ELR shaping**: Norm control is useful not because a particular norm is intrinsically preferable, but because it expands the set of realizable ELR schedules. The explicit norm-control experiment shows that increasing parameter norm *more rapidly* late in training can *improve* final loss by producing more favorable ELR decay.

---

## Conclusion

### Main Takeaways

1. **ELR collapse is a precise macroscopic law**: LR and parameter norm govern loss dynamics primarily through their ratio, with collapse errors of a few $\times 10^{-3}$ across optimizers, architectures, datasets, and model scales.

2. **Norm control acts through ELR**: Weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce, not as independent controls.

3. **ELR enables scaling law transfer**: Parameterizing FSL with ELR instead of LR enables transfer across norm-control methods, reducing OOD prediction error by 11.83×.

4. **Delayed acceleration is explained**: The late-emerging benefit of norm control reflects temporal separation—gains are acquired early but revealed late as noise diminishes.

### Limitations and Future Directions

- **Scope limited to loss dynamics**: Matched loss trajectories need not imply matched parameters, representations, or downstream performance
- **Microscopic mechanism remains open**: A satisfactory theory should explain why such an accurate low-dimensional description exists and why precision depends on normalization design and variation timescale
- **Future work**: Determine how the desired ELR schedule changes with model size, batch size, data budget, and training horizon; stability, numerical precision, and transferability remain additional constraints on ELR schedule realization

---

_Markdown view of https://picx.dev/p/06lXva, served by PicX — AI-generated visual whiteboard summaries of research papers._