Summary (Overview)

• Proposes Musec (MomentUm SpEctral Clipping): A new optimizer that replaces Muon's spectral flattening (setting all singular values to 1) with spectral clipping (only clipping singular values exceeding a threshold D), preserving the underlying spectral structure of the momentum matrix.
• Provides first convergence guarantees for Muon-type algorithms in nonconvex nonsmooth settings: Achieves $\mathcal{O}(r^{3/2}\delta^{-1}\epsilon^{-3})$ oracle complexity for finding Goldstein stationary points, matching the optimal rate for stochastic first-order methods in this regime.
• Develops Soft Musec: An efficient SVD-free implementation using a smooth saturation function $h(w,D) = \frac{Dw}{\sqrt{w^2 + D^2}}$ approximated via coupled Newton–Schulz iterations (only 5 iterations in bfloat16).
• Demonstrates empirical stability improvements: Soft Musec maintains stable convergence across learning rates from 0.01 to 0.8 where Muon and MuonClip diverge, while matching their performance under well-tuned configurations.
• Offers a principled interpretation: Musec's update solves a Frobenius-norm-penalized quadratic subproblem subject to a spectral-norm constraint, complementing Muon's interpretation as unconstrained steepest descent under the spectral norm.

Introduction and Theoretical Foundation

Background and Motivation

Muon has emerged as a highly effective matrix-aware optimizer for large language model training, achieving superior convergence compared to Adam/AdamW. It has been adopted in several large-scale commercial LLM training efforts, including Kimi K2, GLM-5, and DeepSeek-V4. However, Muon suffers from training instability manifested by:

• Exploding weight norms
• Divergent attention logits
• Loss spikes

Root cause: Muon replaces the singular values of its momentum matrix with approximately one (spectral flattening), which injects substantial update magnitude into directions with relatively small singular values, amplifying weakly represented directions.

Limitations of Existing Approaches

Theoretical Gap

Jiang et al. (2026a) recently showed that Muon does not converge in the general nonconvex nonsmooth setting, highlighting the need for principled modifications to its update rule.

Methodology

Problem Formulation

The paper considers the stochastic optimization problem:

$$\min_{\boldsymbol{W} \in \mathbb{R}^{m \times n}} f(\boldsymbol{W}) = \mathbb{E}_{\xi \sim \mathcal{P}}[F(\boldsymbol{W}; \xi)]$$

Key assumptions:

• $F(\boldsymbol{W};\xi)$ is $L$-Lipschitz continuous in its first argument
• $f(\boldsymbol{W})$ is $\rho$-weakly convex
• Stochastic oracle $G(\boldsymbol{W};\xi)$ is an unbiased estimator of a Clarke subgradient with bounded variance $\sigma^2$

The Musec Algorithm

The key difference from Muon lies in the update step. Instead of orthogonalization (setting all singular values to 1), Musec applies spectral clipping:

$$\boldsymbol{M}_n = \boldsymbol{U}_n \boldsymbol{S}_n \boldsymbol{V}_n^{\top}, \quad \text{where} \quad \boldsymbol{S}_n = \text{Clip}(\hat{\boldsymbol{S}}_n, D)$$

The clipping operator is defined entrywise: $\text{clip}(w, D) = \min\{w, D\}$

Unlike Muon's polar step (which maps all nonzero singular values to one), Musec only truncates singular values exceeding threshold $D$ while preserving smaller ones.

Musec as Constrained Steepest Descent

Proposition 4.1: The Musec update solves:

$$\min_{\boldsymbol{\Delta}_l} \left[ \langle \boldsymbol{G}_l, \boldsymbol{\Delta}_l \rangle + \frac{\lambda}{2}\|\boldsymbol{\Delta}_l\|_F^2 \right], \quad \text{s.t.} \quad \|\boldsymbol{\Delta}_l\|_2 \leq D/\lambda$$

which yields $\boldsymbol{\Delta}_l = -\eta \cdot \boldsymbol{U}_l \hat{\boldsymbol{S}}_l \boldsymbol{V}_l^{\top}$ where $\eta = 1/\lambda$ and $\hat{\boldsymbol{S}}_l = \text{Clip}(\boldsymbol{S}_l, D)$.

Convergence Analysis

Key Lemma 4.2 (Single-epoch progress bound): For $\gamma = \beta/\eta$ and $\eta \leq 1/\rho$ where $\beta \leq 1/8$:

$$\mathbb{E}[f(\boldsymbol{W}_T^{(k)}) - f(\boldsymbol{W}_0^{(k)})] \leq -\mathbb{E}\left[\frac{\beta D T}{\gamma}\|\overline{\boldsymbol{H}}^{(k)}\|_F + \sum_{t=0}^{T-1}\frac{\beta}{8\gamma}\|\boldsymbol{M}_t^{(k)}\|_F^2\right] + \frac{\beta D \sigma \sqrt{T}}{\gamma} + \left(\frac{\beta T}{\gamma} + \frac{1}{\gamma}\right)D^2 + \frac{\beta^2 L^2 T}{\gamma} + \frac{rD^2}{\gamma}$$

Theorem 4.3 (Convergence guarantee): With $\gamma = \beta/\eta$ and $\eta \leq 1/\rho$ where $\beta \leq 1/8$:

$$\mathbb{E}\left[\frac{1}{K}\sum_{k=1}^{K}\text{dist}(\mathbf{0}, \partial_\delta f(\overline{\boldsymbol{W}}^{(k)}))\right] \leq \frac{\sigma}{\sqrt{T}} + \left(1 + \frac{2r}{\beta T}\right)D + \frac{\beta L^2}{D} + \frac{\gamma \Delta_f}{\beta D T K}$$

Corollary 4.4 (Oracle complexity): With optimal parameter choices, Musec requires at most:

$$N = \mathcal{O}\left(\frac{r^{3/4}}{\delta} + \frac{\delta^2 \rho^3}{r^{3/2}} + \frac{r^{3/2}(\sigma^3 + L^6 + \Delta_f^3)}{\delta \epsilon^3}\right)$$

oracle calls to find a $(\delta, \epsilon)$-Goldstein stationary point. When $\rho \leq \mathcal{O}(r\delta^{-1}\epsilon^{-1})$, the dominant term reduces to $\mathcal{O}(r^{3/2}\delta^{-1}\epsilon^{-3})$, matching the optimal dependence on accuracy parameters.

Soft Musec: Efficient Implementation

To avoid expensive SVD computations, hard clipping is replaced with a smooth saturation function:

$$h(w, D) = \frac{Dw}{\sqrt{w^2 + D^2}}$$

which yields the soft spectral clipping operator:

$$H(\boldsymbol{M}, D) = D(\boldsymbol{M}\boldsymbol{M}^{\top} + D^2\boldsymbol{I})^{-1/2}\boldsymbol{M}$$

The matrix inverse square root $(\boldsymbol{M}\boldsymbol{M}^{\top} + D^2\boldsymbol{I})^{-1/2}$ is approximated using coupled Newton–Schulz iterations (only matrix multiplications, GPU-friendly). The addition of $D^2\boldsymbol{I}$ shifts eigenvalues away from zero, ensuring numerical stability. Only 5 Newton–Schulz iterations suffice in bfloat16 arithmetic.

Empirical Validation / Results

Experimental Setup

• Models: NanoGPT-Small (491M), NanoGPT-Medium (613M), NanoGPT-Wide (1.63B parameters)
• Datasets: FineWeb, OpenWebText, C4
• Baselines: Muon, MuonClip, SPECTRA (with SGDM base)
• Hyperparameters: Weight decay tuned over {0.096, 0.3, 0.6, 1.2}; clipping threshold D tuned over {0.05, 0.125, 0.25, 0.5, 0.75, 1.0}

Stability Across Learning Rates

Results on FineWeb (Figure 1 in paper):

• All methods achieve comparable validation loss at smaller learning rates
• As learning rate increases: Soft Musec maintains stable convergence; Muon and MuonClip degrade
• At the largest learning rate: Muon fails to converge across all model sizes; MuonClip fails on Small and Medium, converges to substantially worse loss on Wide
• Soft Musec and SPECTRA remain stable across all configurations
• Soft Musec achieves performance comparable to SPECTRA with slight improvements in several settings

Training Dynamics

Training NanoGPT-Medium at learning rate 0.2 (Figure 2 in paper):

• All methods exhibit a transient loss spike near step 400 (coincides with scheduled learning rate increase of 52%, doubled batch size, and widened attention window)
• Soft Musec and SPECTRA recover quickly and smoothly from the spike
• Muon recovers slowly and converges to substantially higher final validation loss
• MuonClip improves over Muon but displays noticeable oscillations

Weight Norm Analysis

Spectral norm tracking during NanoGPT-Medium training on FineWeb at $\eta = 0.2$ (Figure 3):

This confirms that optimizer-level spectral clipping provides uniform stabilization across the entire model, whereas MuonClip only regulates query-key matrices.

Theoretical and Practical Implications

Theoretical Implications

1. First convergence guarantee for Muon-type algorithms in nonconvex nonsmooth settings: While Jiang et al. (2026a) showed Muon with exact polar updates does not converge, Musec achieves convergence through explicit spectral clipping.

2. Optimal complexity: The dependence on accuracy parameters ($\delta^{-1}\epsilon^{-3}$) matches the optimal stochastic first-order method (Cutkosky et al., 2023). The factor $r^{3/2}$ arises from the matrix-valued structure, improving on the $r^2$ factor of prior Muon convergence analyses under smoothness.

3. Complementary to concurrent work: Li & Tsuchiya (2026) proved convergence via implicit smoothing from Newton–Schulz iterations; Musec offers convergence via explicit spectral clipping.

Practical Implications

1. Architecture-agnostic stabilization: Unlike MuonClip (which targets QK weights only), Musec operates at the optimizer level, stabilizing all weight matrices uniformly without architectural modifications.

2. Wider usable learning rate range: Soft Musec remains stable at learning rates where Muon and MuonClip diverge, potentially enabling faster training through higher learning rates.

3. Safe integration: Soft Musec matches Muon's performance under well-tuned configurations while providing a safety margin against divergence, making it suitable for large-scale LLM training where failed runs are extremely costly.

Conclusion

The paper introduces Musec, which replaces Muon's spectral flattening with spectral clipping to stabilize training while preserving fast convergence. Key contributions:

1. Theoretical: First convergence guarantees for spectral clipping-based Muon variants in nonconvex nonsmooth optimization, with optimal $\mathcal{O}(r^{3/2}\delta^{-1}\epsilon^{-3})$ complexity matching the optimal stochastic first-order method.

2. Algorithmic: Development of Soft Musec, an efficient SVD-free implementation using smooth spectral saturation via coupled Newton–Schulz iterations.

3. Empirical: Consistent stability improvements over existing Muon variants across multiple datasets (FineWeb, OpenWebText, C4) and model sizes (491M to 1.63B parameters), with stable convergence in settings where Muon and MuonClip diverge.

Future directions suggested by the work include:

• Further investigation of spectral clipping as an optimizer-agnostic wrapper (as explored concurrently by SPECTRA)
• Application of Musec to other matrix-aware optimizers beyond Muon
• Extension of the theoretical analysis to other smoothness conditions or problem structures