# Spectral Scaling Laws of Muon

> Muon momentum singular value quantiles stabilize during training and follow layer-dependent power laws in model size, making uniform Newton-Schulz configurations suboptimal at scale.

- **Source:** [arXiv](https://arxiv.org/abs/2606.04058)
- **Published:** 2026-08-29
- **Permalink:** https://picx.dev/p/TMAPT5
- **Whiteboard:** https://picx.dev/p/TMAPT5/image

## Summary

## Summary (Overview)

- This paper presents the **first systematic study** of how the singular value spectrum of Muon's momentum matrices evolves during training and scales with model size, across GPT-2-style models ranging from **77M to 2.8B parameters**.
- The authors discover that after a short burn-in period, singular value quantiles of the momentum buffer **stabilize at values** determined by layer type and model size, following **remarkably clean power laws** in model size with **layer-dependent exponents**.
- Scaling exponents vary dramatically across depth: mid-early to mid-late layers scale mildly (~$M^{-0.25}$), while some final layers scale aggressively (up to $M^{-0.96}$), meaning uniform Newton–Schulz (NS) configurations become **suboptimal at scale**.
- A controlled experiment shows that orthonormalizing roughly the **top 50% of singular directions** is sufficient to recover full Muon performance, while the top 10% is insufficient.
- The paper provides a **practical, layer-aware recipe** for choosing the minimum NS configuration needed at any target scale, with a concrete case study extrapolating to 300B parameters.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Pre-training large language models (LLMs) consumes millions of GPU hours, making optimizer choice a central design decision. While AdamW has long been the standard optimizer, **orthonormalized-update optimizers such as Muon** have recently gained prominence, with state-of-the-art models like Kimi-K2, GLM-5, and DeepSeek-V4 adopting Muon. At scale, Muon achieves **twice the compute efficiency** of AdamW.

### The Newton–Schulz (NS) Iteration

Muon performs approximate orthonormalization of momentum matrices using the **Newton–Schulz (NS) iteration**, which repeatedly applies an odd polynomial to push singular values toward 1. The key limitation: since each polynomial is odd, $f(0) = 0$, so directions with sufficiently small singular values **fail to be orthonormalized**.

The canonical NS polynomial is:

$$
p(x) = 2x - 1.5x^3 + 0.5x^5
$$

applied $n = 5$ times, i.e., $f = p^{\circ 5}$. For $\sigma \leq 0.003$, one can verify numerically that $f(\sigma) \leq 0.1$, meaning directions with singular values below roughly 0.003 remain essentially unorthonormalized.

### Key Theoretical Insight

The NS procedure reduces to a one-dimensional problem: find a scalar composition $f = p_n \circ \cdots \circ p_1$ such that $f(\sigma) \approx 1$ for every singular value $\sigma \in [0,1]$. For any matrix with SVD $A = USV^{\top}$, an odd polynomial satisfies:

$$
p(A) = U p(S) V^{\top}
$$

so singular vectors are exactly preserved, and only singular values are modified.

---

## Methodology

### Experimental Setup

The authors pretrain **seven GPT-2-style models** ranging from 77M to 2.8B parameters with Muon, each trained for a Chinchilla-optimal number of tokens:

| Model | Params | Model Dim | # Layers | # Heads | Seq Len |
|-------|--------|-----------|----------|---------|---------|
| GPT2-77M | 77M | 512 | 8 | 8 | 512 |
| GPT2-160M | 160M | 768 | 12 | 12 | 512 |
| GPT2-354M | 354M | 1024 | 20 | 16 | 512 |
| GPT2-600M | 600M | 1280 | 24 | 20 | 512 |
| GPT2-1.2B | 1.2B | 1792 | 26 | 28 | 1024 |
| GPT2-1.6B | 1.6B | 2048 | 28 | 32 | 1024 |
| GPT2-2.8B | 2.8B | 2560 | 32 | 40 | 1024 |

### Tracking Protocol

For each model, the authors monitor layers at relative depth checkpoints $\lfloor{N/4}, lfloor{2N/4}, lfloor{3N/4}, N$ (mid-early, mid, mid-late, final). Within each layer, they track all **six momentum matrices** (four attention projections Q, K, V, O and two MLP projections), recording singular value quantiles for $q \in \{0.1, 0.25, 0.5, 0.75, 0.9\}$ at every training step.

### Rank-p Orthonormal Updates Experiment

To determine how many singular directions must be orthonormalized, the authors introduce **rank-$p$ orthonormal updates**:

$$
O = U_{:,1:k} V_{:,1:k}^{\top}, \qquad k = \lfloor{\min(m,n) \cdot p}rfloor
$$

with $p \in \{0.1, 0.25, 0.5, 0.9\}$.

### Scaling Law Fitting

For each model size and layer type, stabilization values are estimated by **averaging quantiles over training steps 1300–1500**, then plotted against model size on a log-log scale to fit power laws.

---

## Empirical Validation / Results

### Stabilization of Singular Value Quantiles

A consistent phenomenon emerges across all model sizes and layer types: **after a short transient phase, the quantiles stabilize** at values that persist for the remainder of training. The shape of the transient differs by matrix type:
- **Q and K matrices**: sharp decrease followed by recovery before stabilizing
- **V, O, and MLP matrices**: monotonic increase from the start before stabilizing

### Full Spectrum Structure

The full spectrum of normalized singular values exhibits two consistent features:
1. **A single outlier singular value** dominates, often an order of magnitude or more larger than the rest
2. Once the outlier is removed, the **bulk is concentrated near zero**, with count decaying roughly exponentially as singular value grows

### Scaling Law Exponents

The fitted power laws show remarkably clean log-log linear relationships with **layer-dependent exponents**:

| Layer Type | Scaling Exponent |
|---|---|
| Mid-early through mid-late layers | ~$-0.25$ |
| Final MLP projection | $-0.96$ |

This means increasing model size by a factor of 32 decreases stabilization values by only ~2x for mid layers, but nearly linearly for final layers.

### Rank-p Update Results

- $p = 0.9$: essentially indistinguishable from full Muon
- $p = 0.5$: only minor performance gap
- $p = 0.25$: 10–20% less token-efficient than full Muon
- $p = 0.1$: ~50% less efficient, impractical

The gap is confirmed not to be an artifact of early-training dynamics via controlled switching experiments.

### Case Study: Extrapolating to 300B Scale

**Mid-late Q projection** (exponent -0.27):
$$
5 \times 10^{-3} \times 100^{-0.27} \approx 1.4 \times 10^{-3}
$$
This sits above the NanoGPT 5-step failure regime — standard 5-step NS remains sufficient.

**Final O projection** (exponent -0.66):
$$
10^{-3} \times 100^{-0.66} \approx 5 \times 10^{-5}
$$
This falls inside the NanoGPT failure regime — requires the 10-step composition used by DeepSeek-V4.

---

## Theoretical and Practical Implications

### Theoretical Significance

The discovery that singular value quantiles follow clean power laws in model size with **layer-dependent exponents** reveals an underlying structure in how Muon's spectral dynamics scale. The wide range of exponents (from -0.25 to -0.96) demonstrates that **uniform NS configurations are unavoidably suboptimal at scale** — the layers needing the most accurate orthonormalization are precisely those whose singular values shrink fastest with model size.

### Practical Recipe

The scaling laws enable a principled, layer-aware approach to NS configuration:
1. **Fit the laws** at accessible scales (e.g., up to 2.8B)
2. **Extrapolate** to target frontier scale for each layer type
3. **Select per-layer NS configurations** that keep the relevant quantile above the failure threshold

For a 300B-scale run, this means:
- Use the **5-step NanoGPT coefficients** for the majority of layers
- Apply the **10-step composition** (as in DeepSeek-V4) to final layers

### Self-Consistency Guarantee

The authors establish a clean correspondence: as long as NS orthonormalizes at least the top 50% of directions, the induced quantile dynamics closely track full Muon, making the fitted scaling laws **self-consistent** in this regime.

---

## Conclusion

### Main Takeaways

- After a short burn-in, momentum singular value quantiles stabilize at values following **clean power laws in model size** with layer-dependent exponents
- Exponents range from **-0.25 (mid layers) to -0.96 (final MLP projection)**
- A **uniform NS configuration across all layers is suboptimal at scale**
- The paper turns the opaque design choice of NS accuracy into a **quantitative, layer-aware decision**

### Future Directions

1. **Extending to other architectures**: Mixture-of-Experts models and other optimizers relying on iterative matrix-function approximations (e.g., Shampoo and descendants)
2. **Designing NS coefficients tuned to empirical singular value distributions** of each layer to further reduce orthonormalization cost at frontier scale

The authors note that the stabilization phenomenon and specific exponents may be specific to GPT-2-style language models trained with Muon, and quantifying the underestimation effect for NS configurations that only orthonormalize the top 25% of directions at scale remains open work requiring expensive per-step SVD computations.

---

_Markdown view of https://picx.dev/p/TMAPT5, served by PicX — AI-generated visual whiteboard summaries of research papers._