Summary (Overview)

• This paper presents the first systematic study of how the singular value spectrum of Muon's momentum matrices evolves during training and scales with model size, across GPT-2-style models ranging from 77M to 2.8B parameters.
• The authors discover that after a short burn-in period, singular value quantiles of the momentum buffer stabilize at values determined by layer type and model size, following remarkably clean power laws in model size with layer-dependent exponents.
• Scaling exponents vary dramatically across depth: mid-early to mid-late layers scale mildly (~$M^{-0.25}$), while some final layers scale aggressively (up to $M^{-0.96}$), meaning uniform Newton–Schulz (NS) configurations become suboptimal at scale.
• A controlled experiment shows that orthonormalizing roughly the top 50% of singular directions is sufficient to recover full Muon performance, while the top 10% is insufficient.
• The paper provides a practical, layer-aware recipe for choosing the minimum NS configuration needed at any target scale, with a concrete case study extrapolating to 300B parameters.

---

Introduction and Theoretical Foundation

Background and Motivation

Pre-training large language models (LLMs) consumes millions of GPU hours, making optimizer choice a central design decision. While AdamW has long been the standard optimizer, orthonormalized-update optimizers such as Muon have recently gained prominence, with state-of-the-art models like Kimi-K2, GLM-5, and DeepSeek-V4 adopting Muon. At scale, Muon achieves twice the compute efficiency of AdamW.

The Newton–Schulz (NS) Iteration

Muon performs approximate orthonormalization of momentum matrices using the Newton–Schulz (NS) iteration, which repeatedly applies an odd polynomial to push singular values toward 1. The key limitation: since each polynomial is odd, $f(0) = 0$, so directions with sufficiently small singular values fail to be orthonormalized.

The canonical NS polynomial is:

$$p(x) = 2x - 1.5x^3 + 0.5x^5$$

applied $n = 5$ times, i.e., $f = p^{\circ 5}$. For $\sigma \leq 0.003$, one can verify numerically that $f(\sigma) \leq 0.1$, meaning directions with singular values below roughly 0.003 remain essentially unorthonormalized.

Key Theoretical Insight

The NS procedure reduces to a one-dimensional problem: find a scalar composition $f = p_n \circ \cdots \circ p_1$ such that $f(\sigma) \approx 1$ for every singular value $\sigma \in [0,1]$. For any matrix with SVD $A = USV^{\top}$, an odd polynomial satisfies:

$$p(A) = U p(S) V^{\top}$$

so singular vectors are exactly preserved, and only singular values are modified.

---

Methodology

Experimental Setup

The authors pretrain seven GPT-2-style models ranging from 77M to 2.8B parameters with Muon, each trained for a Chinchilla-optimal number of tokens:

Tracking Protocol

For each model, the authors monitor layers at relative depth checkpoints $\lfloor{N/4}, lfloor{2N/4}, lfloor{3N/4}, N$ (mid-early, mid, mid-late, final). Within each layer, they track all six momentum matrices (four attention projections Q, K, V, O and two MLP projections), recording singular value quantiles for $q \in \{0.1, 0.25, 0.5, 0.75, 0.9\}$ at every training step.

Rank-p Orthonormal Updates Experiment

To determine how many singular directions must be orthonormalized, the authors introduce rank-$p$ orthonormal updates:

$$O = U_{:,1:k} V_{:,1:k}^{\top}, \qquad k = \lfloor{\min(m,n) \cdot p}rfloor$$

with $p \in \{0.1, 0.25, 0.5, 0.9\}$.

Scaling Law Fitting

For each model size and layer type, stabilization values are estimated by averaging quantiles over training steps 1300–1500, then plotted against model size on a log-log scale to fit power laws.

---

Empirical Validation / Results

Stabilization of Singular Value Quantiles

A consistent phenomenon emerges across all model sizes and layer types: after a short transient phase, the quantiles stabilize at values that persist for the remainder of training. The shape of the transient differs by matrix type:

• Q and K matrices: sharp decrease followed by recovery before stabilizing
• V, O, and MLP matrices: monotonic increase from the start before stabilizing

Full Spectrum Structure

The full spectrum of normalized singular values exhibits two consistent features:

1. A single outlier singular value dominates, often an order of magnitude or more larger than the rest
2. Once the outlier is removed, the bulk is concentrated near zero, with count decaying roughly exponentially as singular value grows

Scaling Law Exponents

The fitted power laws show remarkably clean log-log linear relationships with layer-dependent exponents:

This means increasing model size by a factor of 32 decreases stabilization values by only ~2x for mid layers, but nearly linearly for final layers.

Rank-p Update Results

• $p = 0.9$: essentially indistinguishable from full Muon
• $p = 0.5$: only minor performance gap
• $p = 0.25$: 10–20% less token-efficient than full Muon
• $p = 0.1$: ~50% less efficient, impractical

The gap is confirmed not to be an artifact of early-training dynamics via controlled switching experiments.

Case Study: Extrapolating to 300B Scale

Mid-late Q projection (exponent -0.27):

$$5 \times 10^{-3} \times 100^{-0.27} \approx 1.4 \times 10^{-3}$$

This sits above the NanoGPT 5-step failure regime — standard 5-step NS remains sufficient.

Final O projection (exponent -0.66):

$$10^{-3} \times 100^{-0.66} \approx 5 \times 10^{-5}$$

This falls inside the NanoGPT failure regime — requires the 10-step composition used by DeepSeek-V4.

---

Theoretical and Practical Implications

Theoretical Significance

The discovery that singular value quantiles follow clean power laws in model size with layer-dependent exponents reveals an underlying structure in how Muon's spectral dynamics scale. The wide range of exponents (from -0.25 to -0.96) demonstrates that uniform NS configurations are unavoidably suboptimal at scale — the layers needing the most accurate orthonormalization are precisely those whose singular values shrink fastest with model size.

Practical Recipe

The scaling laws enable a principled, layer-aware approach to NS configuration:

1. Fit the laws at accessible scales (e.g., up to 2.8B)
2. Extrapolate to target frontier scale for each layer type
3. Select per-layer NS configurations that keep the relevant quantile above the failure threshold

For a 300B-scale run, this means:

• Use the 5-step NanoGPT coefficients for the majority of layers
• Apply the 10-step composition (as in DeepSeek-V4) to final layers

Self-Consistency Guarantee

The authors establish a clean correspondence: as long as NS orthonormalizes at least the top 50% of directions, the induced quantile dynamics closely track full Muon, making the fitted scaling laws self-consistent in this regime.

---

Conclusion

Main Takeaways

• After a short burn-in, momentum singular value quantiles stabilize at values following clean power laws in model size with layer-dependent exponents
• Exponents range from -0.25 (mid layers) to -0.96 (final MLP projection)
• A uniform NS configuration across all layers is suboptimal at scale
• The paper turns the opaque design choice of NS accuracy into a quantitative, layer-aware decision

Future Directions

1. Extending to other architectures: Mixture-of-Experts models and other optimizers relying on iterative matrix-function approximations (e.g., Shampoo and descendants)
2. Designing NS coefficients tuned to empirical singular value distributions of each layer to further reduce orthonormalization cost at frontier scale

The authors note that the stabilization phenomenon and specific exponents may be specific to GPT-2-style language models trained with Muon, and quantifying the underestimation effect for NS configurations that only orthonormalize the top 25% of directions at scale remains open work requiring expensive per-step SVD computations.