# Equal Steps, Unequal Curvature: Where Muon's Update Goes Wrong

> Muon's equal step lengths across singular directions over-step high-curvature top directions, and splitting QKV orthogonalization plus a spectral tilt improves loss by 0.046 nats.

- **Source:** [arXiv](https://arxiv.org/abs/2611.24644)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/GquJAY
- **Whiteboard:** https://picx.dev/p/GquJAY/image

## Summary

# Equal Steps, Unequal Curvature: Where Muon's Update Goes Wrong

**Authors:** Biswa Sengupta, Jinhua Wang, Léo Maxime Brunswic (LLM Suite Team, JPMorgan Chase & Co.)

---

## Summary (Overview)

- **Core finding:** Muon's optimizer assigns equal step lengths across all singular directions of the momentum matrix, which systematically **over-steps high-curvature top singular directions** relative to the tail of the spectrum.
- **Neuron-level imbalance:** In fused QKV matrices, per-neuron step sizes vary dramatically (coefficient of variation 0.43) due to leverage-score effects, while square matrices show near-uniform steps.
- **Key interventions:** Splitting Q, K, V orthogonalization separately yields **−0.0250 nats** loss improvement; a rotation-equivariant spectral tilt $\sigma^{-1/4}$ during cooldown adds **−0.0120 nats**; combined with NorMuon, gains reach **−0.0158 nats** (baseline needs ~4.3% more tokens to match).
- **Negative results:** Input whitening, exact polar factors, and top-heavy decay all fail—none address the measured miscalibration.
- **Formal verification:** All geometric claims are machine-checked in Lean 4, including a criterion for when same-norm reallocations can improve Muon's step.

---

## Introduction and Theoretical Foundation

### Background

Muon (Jordan et al., 2024b) has become the reference optimizer for transformer hidden matrices at small and medium scale. For a weight matrix with momentum $N = U\Sigma V^\top$, the update direction $\text{polar}(N) = UV^\top$ is the steepest-descent direction under the spectral norm.

### Two Hidden Allocations

Muon's step fixes two distributions **without consulting the loss**:

1. **Across singular directions:** Every direction $u_i v_i^\top$ receives a step of the same length, regardless of gradient signal $\sigma_i$ or loss curvature.
2. **Across neurons:** A tall polar factor has row norms equal to square roots of leverage scores of $N$'s column space.

### Theoretical Foundation

The Muon variant studied uses sum-form Nesterov input with a quintic Newton–Schulz map:

$$
B_t = \beta B_{t-1} + G_t, \qquad N_t = G_t + \beta B_t,
$$
$$
W_{t+1} = (1 - \eta_t \lambda) W_t - \eta_t s \cdot \text{NS}_5(N_t) \tag{1}
$$

with shape factor $s = \sqrt{\max(1, m/n)}$. $\text{NS}_5$ normalizes $N$ by its Frobenius norm and applies five iterations of $X \mapsto aX + bXX^\top X + c(XX^\top)^2X$ with $(a,b,c) = (3.4445, -4.775, 2.0315)$.

**Proposition 1 (Polar row norms are leverage scores):** Let $O \in \mathbb{R}^{m \times n}$ with $O^\top O = I_n$ ($m \geq n$). Then $\sum_i \|O_{i,:}\|^2 = n$, but individual $\|O_{i,:}\|^2$ (leverage scores) can be arbitrarily uneven: one row can have norm 1 and another 0.

**Proposition 2 (Block-wise polar steps):** Splitting $W = [W_1; \ldots; W_k]$ into blocks, $[\text{polar}(N_1); \ldots; \text{polar}(N_k)]$ maximizes $\langle N, D \rangle$ over $\|D\|_{\text{blk}} \leq 1$; the joint polar factor can be strictly worse.

---

## Methodology

### Curvature-Resolved Anatomy

The authors partition Muon's next step into groups by **singular-value bands** or **neuron sextiles**, then compute second-order optimal multipliers:

$$
q(c) = -g^\top c + \frac{1}{2}c^\top Hc, \tag{2}
$$
$$
g_b = \langle \nabla \hat{L}, D_b \rangle, \qquad H_{bb'} = \langle D_b, \nabla^2 \hat{L} D_{b'} \rangle
$$

where $\hat{L}$ is the loss on held-out tokens. Muon corresponds to $c = \mathbf{1}$; the stationary point is $c^\star = H^{-1}g$.

### Scale vs. Shape Decomposition

**Theorem 3 (Exact scale/shape split):** Let $H$ be symmetric positive definite, $c^\star = H^{-1}g$, and $\gamma^\star = \mathbf{1}^\top g / \mathbf{1}^\top H\mathbf{1}$. Then:

$$
q(\mathbf{1}) - q(c^\star) = \underbrace{\frac{1}{2}(1 - \gamma^\star)^2 \mathbf{1}^\top H\mathbf{1}}_{\text{step size}} + \underbrace{\frac{1}{2}(\gamma^\star \mathbf{1} - c^\star)^\top H(\gamma^\star \mathbf{1} - c^\star)}_{\text{shape}}
$$

The shape term is strictly positive unless $c^\star$ is flat.

### Same-Norm Reallocation Criterion

**Proposition 4:** Let $N \succ 0$, $r = H\mathbf{1} - g$ (the model gradient at Muon's step), and $p$ the component of $r$ orthogonal to $N\mathbf{1}$. Then:
- (i) If $Hp \neq 0$, some $c$ with $c^\top N c = \mathbf{1}^\top N\mathbf{1}$ has $q(c) < q(\mathbf{1})$.
- (ii) If $r = \lambda N\mathbf{1}$ and $H - \lambda N \succeq 0$ for some $\lambda$, no such $c$ exists.
- (iii) The shape term does not decide between (i) and (ii).

### Experimental Setup

- **Model:** 124.5M-parameter pre-norm decoder (12 layers, width 768, 12 heads, tied embeddings)
- **Data:** 167.8M-token FineWeb-Edu sample, sequence length 512, 24,576 tokens/step
- **Schedule:** Warmup–constant–cooldown (128 warmup steps, linear decay over last 25%)
- **Controls:** Per-step norm matching, tuned baselines (interior-optimum learning rates), paired seeds (42–44)

---

## Empirical Validation / Results

### Finding F1: Muon Over-Steps Top Singular Directions

At every checkpoint, the per-band multiplier rises monotonically from top band to tail:

| Checkpoint | Tail-to-Top Ratio |
|---|---|
| Step 1,024 | 48× |
| Cooldown start | 38× |
| Cooldown end | 190× |

**Mechanism:** At step 1,024, the top band's curvature $H_{11}$ is 29× that of the next band, but gradient alignment $g_1$ is only 9× larger. **Curvature falls off faster than signal with rank.**

### Finding F2: Same-Norm Reallocation Helps Until Late Cooldown

At Muon's own update norm:
- **Step 1,024:** Best same-norm step predicted to change one-step loss by −0.00589 vs. −0.00079 for Muon
- **Cooldown start:** −0.00488 vs. +0.00026
- **Cooldown end:** Only −0.00009 possible gain (top band no longer over-stepped, $c_1 = 0.98$)

**Measured confirmation:** Halving ranks 1–8 improves one-step loss by −4.4 and −3.2 ($\times 10^{-3}$) at matched norm.

### Finding F3: Neuron Imbalance Concentrated in Fused QKV

| Matrix Type | Coefficient of Variation |
|---|---|
| Fused 2304×768 QKV | 0.43 |
| MLP expand/contract | 0.18 |
| Square attention-output | 0.02 |

### Main Results (100.7M tokens, 124M model)

| Optimizer | Loss | Δ vs. Muon | Δ vs. split-QKV |
|---|---|---|---|
| Muon (fused QKV) | 3.9427 | — | +0.0250 |
| NorMuon (fused QKV) | 3.9277 | −0.0150 | +0.0100 |
| Square blocks | 3.9187 | −0.0239 | +0.0011 |
| **Split-QKV Muon** | **3.9176** | **−0.0250** | — |
| + cooldown tilt | 3.9057 | −0.0370 | −0.0120 |
| + NorMuon + tilt | 3.9018 | −0.0408 | −0.0158 |
| + NorMuon + tilt + Schatten-1 decay | 3.8966 | −0.0460 | −0.0210 |

### Scale Test (355M parameters)

NorMuon + tilt changes final loss by **−0.0188 nats** [−0.0202, −0.0175] (three seeds), comparable to the 124M effect.

### Negative Results

| Intervention | Effect |
|---|---|
| Input whitening | +0.021 to +0.060 nats (harmful) |
| Extra Newton–Schulz iterations (exact polar) | −0.0020 (within noise) |
| Schatten-4 decay (top-heavy) | +0.0177 (harmful) |
| Schatten-1 decay (bulk pruning) | −0.0057 (helps) |

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Joint-curvature diagnostic:** First measurement of full cross-group curvature of Muon's step by singular band and neuron group, including cross-group terms that isolated probes omit.
2. **Proved criteria:** Exact scale/shape decomposition (Theorem 3) and same-norm reallocation criterion (Proposition 4), both machine-checked in Lean 4.
3. **Spectral tilt implementation:** A matmul-only, rotation-equivariant tilt using Chebyshev polynomial approximation via Clenshaw's recurrence, avoiding per-step eigendecomposition.

### Practical Implications

- **QKV splitting is the dominant neuron-level gain:** Explains all of the square-block variant's benefit; methods evaluated against fused-QKV Muon may claim gains that splitting provides at zero cost.
- **Spectral tilt timing matters:** Largest benefit from mid-training to cooldown start; decaying learning rate eventually corrects the over-stepping naturally.
- **Additive corrections:** Neuron rebalancing (NorMuon) and spectral tilt combine additively, suggesting they act on independent mechanisms.
- **Decay geometry:** Schatten-1 (bulk-pruning) decay helps while Schatten-4 (top-pruning) hurts, indicating the useful role is cleaning the bulk, not capping spectral norm.

### What Formal Verification Did and Did Not Buy

**Bought:** Bookkeeping (the $1/\sqrt{k}$ factor for wide matrices), symmetry guarantees, correction of reasoning (shape term compares different norms), and a change of direction (finite-step certificate).

**Did not buy:** Prediction of training benefit. Whitened Muon is provably equivariant yet hurts; joint polar factor is provably suboptimal for block norm yet MLP blocks don't help.

---

## Conclusion

Measured against curvature, both of Muon's hidden allocations are miscalibrated:
- Its **spectrum over-steps the top**, which a same-norm reallocation can exploit until late in the cooldown
- Its **per-neuron steps are most uneven** in fused QKV matrices

Corrections aimed at these help additively; principled changes that miss them do not.

### Limitations and Future Directions

- Single data distribution; 124M and 355M parameters; at most 100.7M tokens
- Diagnostic is local, second-order, one-step ahead, from a single run
- Cooldown length and final learning rate not tuned for baseline
- QKV mechanism remains a hypothesis
- Folding the tilt into Newton–Schulz coefficients (as DynMuon does) may remove most overhead

---

_Markdown view of https://picx.dev/p/GquJAY, served by PicX — AI-generated visual whiteboard summaries of research papers._
