Equal Steps, Unequal Curvature: Where Muon's Update Goes Wrong

Authors: Biswa Sengupta, Jinhua Wang, Léo Maxime Brunswic (LLM Suite Team, JPMorgan Chase & Co.)


Summary (Overview)

  • Core finding: Muon's optimizer assigns equal step lengths across all singular directions of the momentum matrix, which systematically over-steps high-curvature top singular directions relative to the tail of the spectrum.
  • Neuron-level imbalance: In fused QKV matrices, per-neuron step sizes vary dramatically (coefficient of variation 0.43) due to leverage-score effects, while square matrices show near-uniform steps.
  • Key interventions: Splitting Q, K, V orthogonalization separately yields −0.0250 nats loss improvement; a rotation-equivariant spectral tilt σ−1/4\sigma^{-1/4} during cooldown adds −0.0120 nats; combined with NorMuon, gains reach −0.0158 nats (baseline needs ~4.3% more tokens to match).
  • Negative results: Input whitening, exact polar factors, and top-heavy decay all fail—none address the measured miscalibration.
  • Formal verification: All geometric claims are machine-checked in Lean 4, including a criterion for when same-norm reallocations can improve Muon's step.

Introduction and Theoretical Foundation

Background

Muon (Jordan et al., 2024b) has become the reference optimizer for transformer hidden matrices at small and medium scale. For a weight matrix with momentum N=UΣV⊤N = U\Sigma V^\top, the update direction polar(N)=UV⊤\text{polar}(N) = UV^\top is the steepest-descent direction under the spectral norm.

Two Hidden Allocations

Muon's step fixes two distributions without consulting the loss:

  1. Across singular directions: Every direction uivi⊤u_i v_i^\top receives a step of the same length, regardless of gradient signal σi\sigma_i or loss curvature.
  2. Across neurons: A tall polar factor has row norms equal to square roots of leverage scores of NN's column space.

Theoretical Foundation

The Muon variant studied uses sum-form Nesterov input with a quintic Newton–Schulz map:

Bt=βBt−1+Gt,Nt=Gt+βBt,B_t = \beta B_{t-1} + G_t, \qquad N_t = G_t + \beta B_t, Wt+1=(1−ηtλ)Wt−ηts⋅NS5(Nt)(1)W_{t+1} = (1 - \eta_t \lambda) W_t - \eta_t s \cdot \text{NS}_5(N_t) \tag{1}

with shape factor s=max⁡(1,m/n)s = \sqrt{\max(1, m/n)}. NS5\text{NS}_5 normalizes NN by its Frobenius norm and applies five iterations of X↦aX+bXX⊤X+c(XX⊤)2XX \mapsto aX + bXX^\top X + c(XX^\top)^2X with (a,b,c)=(3.4445,−4.775,2.0315)(a,b,c) = (3.4445, -4.775, 2.0315).

Proposition 1 (Polar row norms are leverage scores): Let O∈Rm×nO \in \mathbb{R}^{m \times n} with O⊤O=InO^\top O = I_n (m≥nm \geq n). Then ∑i∥Oi,:∥2=n\sum_i \|O_{i,:}\|^2 = n, but individual ∥Oi,:∥2\|O_{i,:}\|^2 (leverage scores) can be arbitrarily uneven: one row can have norm 1 and another 0.

Proposition 2 (Block-wise polar steps): Splitting W=[W1;…;Wk]W = [W_1; \ldots; W_k] into blocks, [polar(N1);…;polar(Nk)][\text{polar}(N_1); \ldots; \text{polar}(N_k)] maximizes ⟨N,D⟩\langle N, D \rangle over ∥D∥blk≤1\|D\|_{\text{blk}} \leq 1; the joint polar factor can be strictly worse.


Methodology

Curvature-Resolved Anatomy

The authors partition Muon's next step into groups by singular-value bands or neuron sextiles, then compute second-order optimal multipliers:

q(c)=−g⊤c+12c⊤Hc,(2)q(c) = -g^\top c + \frac{1}{2}c^\top Hc, \tag{2} gb=⟨∇L^,Db⟩,Hbb′=⟨Db,∇2L^Db′⟩g_b = \langle \nabla \hat{L}, D_b \rangle, \qquad H_{bb'} = \langle D_b, \nabla^2 \hat{L} D_{b'} \rangle

where L^\hat{L} is the loss on held-out tokens. Muon corresponds to c=1c = \mathbf{1}; the stationary point is c⋆=H−1gc^\star = H^{-1}g.

Scale vs. Shape Decomposition

Theorem 3 (Exact scale/shape split): Let HH be symmetric positive definite, c⋆=H−1gc^\star = H^{-1}g, and γ⋆=1⊤g/1⊤H1\gamma^\star = \mathbf{1}^\top g / \mathbf{1}^\top H\mathbf{1}. Then:

q(1)−q(c⋆)=12(1−γ⋆)21⊤H1⏟step size+12(γ⋆1−c⋆)⊤H(γ⋆1−c⋆)⏟shapeq(\mathbf{1}) - q(c^\star) = \underbrace{\frac{1}{2}(1 - \gamma^\star)^2 \mathbf{1}^\top H\mathbf{1}}_{\text{step size}} + \underbrace{\frac{1}{2}(\gamma^\star \mathbf{1} - c^\star)^\top H(\gamma^\star \mathbf{1} - c^\star)}_{\text{shape}}

The shape term is strictly positive unless c⋆c^\star is flat.

Same-Norm Reallocation Criterion

Proposition 4: Let N≻0N \succ 0, r=H1−gr = H\mathbf{1} - g (the model gradient at Muon's step), and pp the component of rr orthogonal to N1N\mathbf{1}. Then:

  • (i) If Hp≠0Hp \neq 0, some cc with c⊤Nc=1⊤N1c^\top N c = \mathbf{1}^\top N\mathbf{1} has q(c)<q(1)q(c) < q(\mathbf{1}).
  • (ii) If r=λN1r = \lambda N\mathbf{1} and H−λN⪰0H - \lambda N \succeq 0 for some λ\lambda, no such cc exists.
  • (iii) The shape term does not decide between (i) and (ii).

Experimental Setup

  • Model: 124.5M-parameter pre-norm decoder (12 layers, width 768, 12 heads, tied embeddings)
  • Data: 167.8M-token FineWeb-Edu sample, sequence length 512, 24,576 tokens/step
  • Schedule: Warmup–constant–cooldown (128 warmup steps, linear decay over last 25%)
  • Controls: Per-step norm matching, tuned baselines (interior-optimum learning rates), paired seeds (42–44)

Empirical Validation / Results

Finding F1: Muon Over-Steps Top Singular Directions

At every checkpoint, the per-band multiplier rises monotonically from top band to tail:

CheckpointTail-to-Top Ratio
Step 1,02448×
Cooldown start38×
Cooldown end190×

Mechanism: At step 1,024, the top band's curvature H11H_{11} is 29× that of the next band, but gradient alignment g1g_1 is only 9× larger. Curvature falls off faster than signal with rank.

Finding F2: Same-Norm Reallocation Helps Until Late Cooldown

At Muon's own update norm:

  • Step 1,024: Best same-norm step predicted to change one-step loss by −0.00589 vs. −0.00079 for Muon
  • Cooldown start: −0.00488 vs. +0.00026
  • Cooldown end: Only −0.00009 possible gain (top band no longer over-stepped, c1=0.98c_1 = 0.98)

Measured confirmation: Halving ranks 1–8 improves one-step loss by −4.4 and −3.2 (×10−3\times 10^{-3}) at matched norm.

Finding F3: Neuron Imbalance Concentrated in Fused QKV

Matrix TypeCoefficient of Variation
Fused 2304×768 QKV0.43
MLP expand/contract0.18
Square attention-output0.02

Main Results (100.7M tokens, 124M model)

OptimizerLossΔ vs. MuonΔ vs. split-QKV
Muon (fused QKV)3.9427—+0.0250
NorMuon (fused QKV)3.9277−0.0150+0.0100
Square blocks3.9187−0.0239+0.0011
Split-QKV Muon3.9176−0.0250—
+ cooldown tilt3.9057−0.0370−0.0120
+ NorMuon + tilt3.9018−0.0408−0.0158
+ NorMuon + tilt + Schatten-1 decay3.8966−0.0460−0.0210

Scale Test (355M parameters)

NorMuon + tilt changes final loss by −0.0188 nats [−0.0202, −0.0175] (three seeds), comparable to the 124M effect.

Negative Results

InterventionEffect
Input whitening+0.021 to +0.060 nats (harmful)
Extra Newton–Schulz iterations (exact polar)−0.0020 (within noise)
Schatten-4 decay (top-heavy)+0.0177 (harmful)
Schatten-1 decay (bulk pruning)−0.0057 (helps)

Theoretical and Practical Implications

Theoretical Contributions

  1. Joint-curvature diagnostic: First measurement of full cross-group curvature of Muon's step by singular band and neuron group, including cross-group terms that isolated probes omit.
  2. Proved criteria: Exact scale/shape decomposition (Theorem 3) and same-norm reallocation criterion (Proposition 4), both machine-checked in Lean 4.
  3. Spectral tilt implementation: A matmul-only, rotation-equivariant tilt using Chebyshev polynomial approximation via Clenshaw's recurrence, avoiding per-step eigendecomposition.

Practical Implications

  • QKV splitting is the dominant neuron-level gain: Explains all of the square-block variant's benefit; methods evaluated against fused-QKV Muon may claim gains that splitting provides at zero cost.
  • Spectral tilt timing matters: Largest benefit from mid-training to cooldown start; decaying learning rate eventually corrects the over-stepping naturally.
  • Additive corrections: Neuron rebalancing (NorMuon) and spectral tilt combine additively, suggesting they act on independent mechanisms.
  • Decay geometry: Schatten-1 (bulk-pruning) decay helps while Schatten-4 (top-pruning) hurts, indicating the useful role is cleaning the bulk, not capping spectral norm.

What Formal Verification Did and Did Not Buy

Bought: Bookkeeping (the 1/k1/\sqrt{k} factor for wide matrices), symmetry guarantees, correction of reasoning (shape term compares different norms), and a change of direction (finite-step certificate).

Did not buy: Prediction of training benefit. Whitened Muon is provably equivariant yet hurts; joint polar factor is provably suboptimal for block norm yet MLP blocks don't help.


Conclusion

Measured against curvature, both of Muon's hidden allocations are miscalibrated:

  • Its spectrum over-steps the top, which a same-norm reallocation can exploit until late in the cooldown
  • Its per-neuron steps are most uneven in fused QKV matrices

Corrections aimed at these help additively; principled changes that miss them do not.

Limitations and Future Directions

  • Single data distribution; 124M and 355M parameters; at most 100.7M tokens
  • Diagnostic is local, second-order, one-step ahead, from a single run
  • Cooldown length and final learning rate not tuned for baseline
  • QKV mechanism remains a hypothesis
  • Folding the tilt into Newton–Schulz coefficients (as DynMuon does) may remove most overhead

Related papers