Equal Steps, Unequal Curvature: Where Muon's Update Goes Wrong
Authors: Biswa Sengupta, Jinhua Wang, Léo Maxime Brunswic (LLM Suite Team, JPMorgan Chase & Co.)
Summary (Overview)
- Core finding: Muon's optimizer assigns equal step lengths across all singular directions of the momentum matrix, which systematically over-steps high-curvature top singular directions relative to the tail of the spectrum.
- Neuron-level imbalance: In fused QKV matrices, per-neuron step sizes vary dramatically (coefficient of variation 0.43) due to leverage-score effects, while square matrices show near-uniform steps.
- Key interventions: Splitting Q, K, V orthogonalization separately yields −0.0250 nats loss improvement; a rotation-equivariant spectral tilt during cooldown adds −0.0120 nats; combined with NorMuon, gains reach −0.0158 nats (baseline needs ~4.3% more tokens to match).
- Negative results: Input whitening, exact polar factors, and top-heavy decay all fail—none address the measured miscalibration.
- Formal verification: All geometric claims are machine-checked in Lean 4, including a criterion for when same-norm reallocations can improve Muon's step.
Introduction and Theoretical Foundation
Background
Muon (Jordan et al., 2024b) has become the reference optimizer for transformer hidden matrices at small and medium scale. For a weight matrix with momentum , the update direction is the steepest-descent direction under the spectral norm.
Two Hidden Allocations
Muon's step fixes two distributions without consulting the loss:
- Across singular directions: Every direction receives a step of the same length, regardless of gradient signal or loss curvature.
- Across neurons: A tall polar factor has row norms equal to square roots of leverage scores of 's column space.
Theoretical Foundation
The Muon variant studied uses sum-form Nesterov input with a quintic Newton–Schulz map:
with shape factor . normalizes by its Frobenius norm and applies five iterations of with .
Proposition 1 (Polar row norms are leverage scores): Let with (). Then , but individual (leverage scores) can be arbitrarily uneven: one row can have norm 1 and another 0.
Proposition 2 (Block-wise polar steps): Splitting into blocks, maximizes over ; the joint polar factor can be strictly worse.
Methodology
Curvature-Resolved Anatomy
The authors partition Muon's next step into groups by singular-value bands or neuron sextiles, then compute second-order optimal multipliers:
where is the loss on held-out tokens. Muon corresponds to ; the stationary point is .
Scale vs. Shape Decomposition
Theorem 3 (Exact scale/shape split): Let be symmetric positive definite, , and . Then:
The shape term is strictly positive unless is flat.
Same-Norm Reallocation Criterion
Proposition 4: Let , (the model gradient at Muon's step), and the component of orthogonal to . Then:
- (i) If , some with has .
- (ii) If and for some , no such exists.
- (iii) The shape term does not decide between (i) and (ii).
Experimental Setup
- Model: 124.5M-parameter pre-norm decoder (12 layers, width 768, 12 heads, tied embeddings)
- Data: 167.8M-token FineWeb-Edu sample, sequence length 512, 24,576 tokens/step
- Schedule: Warmup–constant–cooldown (128 warmup steps, linear decay over last 25%)
- Controls: Per-step norm matching, tuned baselines (interior-optimum learning rates), paired seeds (42–44)
Empirical Validation / Results
Finding F1: Muon Over-Steps Top Singular Directions
At every checkpoint, the per-band multiplier rises monotonically from top band to tail:
| Checkpoint | Tail-to-Top Ratio |
|---|---|
| Step 1,024 | 48× |
| Cooldown start | 38× |
| Cooldown end | 190× |
Mechanism: At step 1,024, the top band's curvature is 29× that of the next band, but gradient alignment is only 9× larger. Curvature falls off faster than signal with rank.
Finding F2: Same-Norm Reallocation Helps Until Late Cooldown
At Muon's own update norm:
- Step 1,024: Best same-norm step predicted to change one-step loss by −0.00589 vs. −0.00079 for Muon
- Cooldown start: −0.00488 vs. +0.00026
- Cooldown end: Only −0.00009 possible gain (top band no longer over-stepped, )
Measured confirmation: Halving ranks 1–8 improves one-step loss by −4.4 and −3.2 () at matched norm.
Finding F3: Neuron Imbalance Concentrated in Fused QKV
| Matrix Type | Coefficient of Variation |
|---|---|
| Fused 2304×768 QKV | 0.43 |
| MLP expand/contract | 0.18 |
| Square attention-output | 0.02 |
Main Results (100.7M tokens, 124M model)
| Optimizer | Loss | Δ vs. Muon | Δ vs. split-QKV |
|---|---|---|---|
| Muon (fused QKV) | 3.9427 | — | +0.0250 |
| NorMuon (fused QKV) | 3.9277 | −0.0150 | +0.0100 |
| Square blocks | 3.9187 | −0.0239 | +0.0011 |
| Split-QKV Muon | 3.9176 | −0.0250 | — |
| + cooldown tilt | 3.9057 | −0.0370 | −0.0120 |
| + NorMuon + tilt | 3.9018 | −0.0408 | −0.0158 |
| + NorMuon + tilt + Schatten-1 decay | 3.8966 | −0.0460 | −0.0210 |
Scale Test (355M parameters)
NorMuon + tilt changes final loss by −0.0188 nats [−0.0202, −0.0175] (three seeds), comparable to the 124M effect.
Negative Results
| Intervention | Effect |
|---|---|
| Input whitening | +0.021 to +0.060 nats (harmful) |
| Extra Newton–Schulz iterations (exact polar) | −0.0020 (within noise) |
| Schatten-4 decay (top-heavy) | +0.0177 (harmful) |
| Schatten-1 decay (bulk pruning) | −0.0057 (helps) |
Theoretical and Practical Implications
Theoretical Contributions
- Joint-curvature diagnostic: First measurement of full cross-group curvature of Muon's step by singular band and neuron group, including cross-group terms that isolated probes omit.
- Proved criteria: Exact scale/shape decomposition (Theorem 3) and same-norm reallocation criterion (Proposition 4), both machine-checked in Lean 4.
- Spectral tilt implementation: A matmul-only, rotation-equivariant tilt using Chebyshev polynomial approximation via Clenshaw's recurrence, avoiding per-step eigendecomposition.
Practical Implications
- QKV splitting is the dominant neuron-level gain: Explains all of the square-block variant's benefit; methods evaluated against fused-QKV Muon may claim gains that splitting provides at zero cost.
- Spectral tilt timing matters: Largest benefit from mid-training to cooldown start; decaying learning rate eventually corrects the over-stepping naturally.
- Additive corrections: Neuron rebalancing (NorMuon) and spectral tilt combine additively, suggesting they act on independent mechanisms.
- Decay geometry: Schatten-1 (bulk-pruning) decay helps while Schatten-4 (top-pruning) hurts, indicating the useful role is cleaning the bulk, not capping spectral norm.
What Formal Verification Did and Did Not Buy
Bought: Bookkeeping (the factor for wide matrices), symmetry guarantees, correction of reasoning (shape term compares different norms), and a change of direction (finite-step certificate).
Did not buy: Prediction of training benefit. Whitened Muon is provably equivariant yet hurts; joint polar factor is provably suboptimal for block norm yet MLP blocks don't help.
Conclusion
Measured against curvature, both of Muon's hidden allocations are miscalibrated:
- Its spectrum over-steps the top, which a same-norm reallocation can exploit until late in the cooldown
- Its per-neuron steps are most uneven in fused QKV matrices
Corrections aimed at these help additively; principled changes that miss them do not.
Limitations and Future Directions
- Single data distribution; 124M and 355M parameters; at most 100.7M tokens
- Diagnostic is local, second-order, one-step ahead, from a single run
- Cooldown length and final learning rate not tuned for baseline
- QKV mechanism remains a hypothesis
- Folding the tilt into Newton–Schulz coefficients (as DynMuon does) may remove most overhead
Related papers
- InfuseLoop: Cross-Trajectory Latent Injection for Scalable Parallel Looped Transformers
InfuseLoop injects cross-trajectory latent states into parallel looped transformers, eliminating training spikes and enabling quality gains at 8 loops with minimal overhead.
- BaseRSI: Recursive Self-Improvement of a Base Vision-Language Model via Training-Free Harness Internalization
BaseRSI lets a vision-language model improve its own weights via training-free per-block activation edits, gaining +11.7 points in-distribution and +6.1 out-of-distribution across ten benchmarks.
- BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
BuggyMLE generates 6,600+ bug-injected ML repositories and a structural complexity metric (CCI) predicting solve rates with R²=0.951, enabling difficulty-graded benchmarks where a 9B model rivals frontier agents.