# Musec: MomentUm SpEctral Clipping for Stable Muon-type Training

> Musec replaces Muon's spectral flattening with spectral clipping, achieving the first convergence guarantees for Muon-type optimizers in nonconvex nonsmooth settings with optimal complexity while stabilizing training across learning rates.

- **Source:** [arXiv](https://arxiv.org/abs/2609.11655)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/pWlpPO
- **Whiteboard:** https://picx.dev/p/pWlpPO/image

## Summary

## Summary (Overview)

- **Proposes Musec (MomentUm SpEctral Clipping)**: A new optimizer that replaces Muon's spectral flattening (setting all singular values to 1) with spectral clipping (only clipping singular values exceeding a threshold D), preserving the underlying spectral structure of the momentum matrix.
- **Provides first convergence guarantees for Muon-type algorithms in nonconvex nonsmooth settings**: Achieves $\mathcal{O}(r^{3/2}\delta^{-1}\epsilon^{-3})$ oracle complexity for finding Goldstein stationary points, matching the optimal rate for stochastic first-order methods in this regime.
- **Develops Soft Musec**: An efficient SVD-free implementation using a smooth saturation function $h(w,D) = \frac{Dw}{\sqrt{w^2 + D^2}}$ approximated via coupled Newton–Schulz iterations (only 5 iterations in bfloat16).
- **Demonstrates empirical stability improvements**: Soft Musec maintains stable convergence across learning rates from 0.01 to 0.8 where Muon and MuonClip diverge, while matching their performance under well-tuned configurations.
- **Offers a principled interpretation**: Musec's update solves a Frobenius-norm-penalized quadratic subproblem subject to a spectral-norm constraint, complementing Muon's interpretation as unconstrained steepest descent under the spectral norm.

## Introduction and Theoretical Foundation

### Background and Motivation

Muon has emerged as a highly effective matrix-aware optimizer for large language model training, achieving superior convergence compared to Adam/AdamW. It has been adopted in several large-scale commercial LLM training efforts, including Kimi K2, GLM-5, and DeepSeek-V4. However, Muon suffers from **training instability** manifested by:

- Exploding weight norms
- Divergent attention logits
- Loss spikes

**Root cause**: Muon replaces the singular values of its momentum matrix with approximately one (spectral flattening), which injects substantial update magnitude into directions with relatively small singular values, amplifying weakly represented directions.

### Limitations of Existing Approaches

| Method | Approach | Limitation |
|--------|----------|------------|
| Logit soft-capping (Gemma et al., 2024) | Bounds attention scores after computation | Does not prevent underlying query-key dot products from growing |
| QK-Norm (Dehghani et al., 2023) | Normalizes query/key vectors | Incompatible with MLA architectures |
| MuonClip (Kimi et al., 2025) | Weight clipping on QK matrices | Architecture-specific; leaves VO and MLP weights unaddressed |

### Theoretical Gap

Jiang et al. (2026a) recently showed that **Muon does not converge** in the general nonconvex nonsmooth setting, highlighting the need for principled modifications to its update rule.

## Methodology

### Problem Formulation

The paper considers the stochastic optimization problem:

$$
\min_{\boldsymbol{W} \in \mathbb{R}^{m \times n}} f(\boldsymbol{W}) = \mathbb{E}_{\xi \sim \mathcal{P}}[F(\boldsymbol{W}; \xi)]
$$

**Key assumptions:**
- $F(\boldsymbol{W};\xi)$ is $L$-Lipschitz continuous in its first argument
- $f(\boldsymbol{W})$ is $\rho$-weakly convex
- Stochastic oracle $G(\boldsymbol{W};\xi)$ is an unbiased estimator of a Clarke subgradient with bounded variance $\sigma^2$

### The Musec Algorithm

The key difference from Muon lies in the update step. Instead of orthogonalization (setting all singular values to 1), Musec applies spectral clipping:

$$\boldsymbol{M}_n = \boldsymbol{U}_n \boldsymbol{S}_n \boldsymbol{V}_n^{\top}, \quad \text{where} \quad \boldsymbol{S}_n = \text{Clip}(\hat{\boldsymbol{S}}_n, D)$$

The clipping operator is defined entrywise: $\text{clip}(w, D) = \min\{w, D\}$

Unlike Muon's polar step (which maps all nonzero singular values to one), Musec only truncates singular values exceeding threshold $D$ while preserving smaller ones.

### Musec as Constrained Steepest Descent

**Proposition 4.1**: The Musec update solves:

$$
\min_{\boldsymbol{\Delta}_l} \left[ \langle \boldsymbol{G}_l, \boldsymbol{\Delta}_l \rangle + \frac{\lambda}{2}\|\boldsymbol{\Delta}_l\|_F^2 \right], \quad \text{s.t.} \quad \|\boldsymbol{\Delta}_l\|_2 \leq D/\lambda
$$

which yields $\boldsymbol{\Delta}_l = -\eta \cdot \boldsymbol{U}_l \hat{\boldsymbol{S}}_l \boldsymbol{V}_l^{\top}$ where $\eta = 1/\lambda$ and $\hat{\boldsymbol{S}}_l = \text{Clip}(\boldsymbol{S}_l, D)$.

### Convergence Analysis

**Key Lemma 4.2** (Single-epoch progress bound): For $\gamma = \beta/\eta$ and $\eta \leq 1/\rho$ where $\beta \leq 1/8$:

$$
\mathbb{E}[f(\boldsymbol{W}_T^{(k)}) - f(\boldsymbol{W}_0^{(k)})] \leq -\mathbb{E}\left[\frac{\beta D T}{\gamma}\|\overline{\boldsymbol{H}}^{(k)}\|_F + \sum_{t=0}^{T-1}\frac{\beta}{8\gamma}\|\boldsymbol{M}_t^{(k)}\|_F^2\right] + \frac{\beta D \sigma \sqrt{T}}{\gamma} + \left(\frac{\beta T}{\gamma} + \frac{1}{\gamma}\right)D^2 + \frac{\beta^2 L^2 T}{\gamma} + \frac{rD^2}{\gamma}
$$

**Theorem 4.3** (Convergence guarantee): With $\gamma = \beta/\eta$ and $\eta \leq 1/\rho$ where $\beta \leq 1/8$:

$$
\mathbb{E}\left[\frac{1}{K}\sum_{k=1}^{K}\text{dist}(\mathbf{0}, \partial_\delta f(\overline{\boldsymbol{W}}^{(k)}))\right] \leq \frac{\sigma}{\sqrt{T}} + \left(1 + \frac{2r}{\beta T}\right)D + \frac{\beta L^2}{D} + \frac{\gamma \Delta_f}{\beta D T K}
$$

**Corollary 4.4** (Oracle complexity): With optimal parameter choices, Musec requires at most:

$$
N = \mathcal{O}\left(\frac{r^{3/4}}{\delta} + \frac{\delta^2 \rho^3}{r^{3/2}} + \frac{r^{3/2}(\sigma^3 + L^6 + \Delta_f^3)}{\delta \epsilon^3}\right)
$$

oracle calls to find a $(\delta, \epsilon)$-Goldstein stationary point. When $\rho \leq \mathcal{O}(r\delta^{-1}\epsilon^{-1})$, the dominant term reduces to $\mathcal{O}(r^{3/2}\delta^{-1}\epsilon^{-3})$, matching the optimal dependence on accuracy parameters.

### Soft Musec: Efficient Implementation

To avoid expensive SVD computations, hard clipping is replaced with a smooth saturation function:

$$h(w, D) = \frac{Dw}{\sqrt{w^2 + D^2}}$$

which yields the soft spectral clipping operator:

$$H(\boldsymbol{M}, D) = D(\boldsymbol{M}\boldsymbol{M}^{\top} + D^2\boldsymbol{I})^{-1/2}\boldsymbol{M}$$

The matrix inverse square root $(\boldsymbol{M}\boldsymbol{M}^{\top} + D^2\boldsymbol{I})^{-1/2}$ is approximated using coupled Newton–Schulz iterations (only matrix multiplications, GPU-friendly). The addition of $D^2\boldsymbol{I}$ shifts eigenvalues away from zero, ensuring numerical stability. Only **5 Newton–Schulz iterations** suffice in bfloat16 arithmetic.

## Empirical Validation / Results

### Experimental Setup

- **Models**: NanoGPT-Small (491M), NanoGPT-Medium (613M), NanoGPT-Wide (1.63B parameters)
- **Datasets**: FineWeb, OpenWebText, C4
- **Baselines**: Muon, MuonClip, SPECTRA (with SGDM base)
- **Hyperparameters**: Weight decay tuned over {0.096, 0.3, 0.6, 1.2}; clipping threshold D tuned over {0.05, 0.125, 0.25, 0.5, 0.75, 1.0}

### Stability Across Learning Rates

Results on FineWeb (Figure 1 in paper):

- **All methods achieve comparable validation loss** at smaller learning rates
- **As learning rate increases**: Soft Musec maintains stable convergence; Muon and MuonClip degrade
- **At the largest learning rate**: Muon fails to converge across all model sizes; MuonClip fails on Small and Medium, converges to substantially worse loss on Wide
- **Soft Musec and SPECTRA remain stable** across all configurations
- Soft Musec achieves performance comparable to SPECTRA with slight improvements in several settings

### Training Dynamics

Training NanoGPT-Medium at learning rate 0.2 (Figure 2 in paper):

- All methods exhibit a transient loss spike near step 400 (coincides with scheduled learning rate increase of 52%, doubled batch size, and widened attention window)
- **Soft Musec and SPECTRA recover quickly and smoothly** from the spike
- **Muon recovers slowly** and converges to substantially higher final validation loss
- **MuonClip improves over Muon** but displays noticeable oscillations

### Weight Norm Analysis

Spectral norm tracking during NanoGPT-Medium training on FineWeb at $\eta = 0.2$ (Figure 3):

| Method | QK norms | VO norms | MLP norms |
|--------|----------|----------|-----------|
| Muon | 300–400 (severe inflation, large oscillations) | 300–400 | 300–400 |
| MuonClip | Partially mitigated | Elevated | Elevated |
| Soft Musec | Below 10 | Below 10 | Below 10 |
| SPECTRA | Below 10 | Below 10 | Below 10 |

This confirms that **optimizer-level spectral clipping provides uniform stabilization across the entire model**, whereas MuonClip only regulates query-key matrices.

## Theoretical and Practical Implications

### Theoretical Implications

1. **First convergence guarantee for Muon-type algorithms in nonconvex nonsmooth settings**: While Jiang et al. (2026a) showed Muon with exact polar updates does not converge, Musec achieves convergence through explicit spectral clipping.

2. **Optimal complexity**: The dependence on accuracy parameters ($\delta^{-1}\epsilon^{-3}$) matches the optimal stochastic first-order method (Cutkosky et al., 2023). The factor $r^{3/2}$ arises from the matrix-valued structure, improving on the $r^2$ factor of prior Muon convergence analyses under smoothness.

3. **Complementary to concurrent work**: Li & Tsuchiya (2026) proved convergence via implicit smoothing from Newton–Schulz iterations; Musec offers convergence via explicit spectral clipping.

### Practical Implications

1. **Architecture-agnostic stabilization**: Unlike MuonClip (which targets QK weights only), Musec operates at the optimizer level, stabilizing all weight matrices uniformly without architectural modifications.

2. **Wider usable learning rate range**: Soft Musec remains stable at learning rates where Muon and MuonClip diverge, potentially enabling faster training through higher learning rates.

3. **Safe integration**: Soft Musec matches Muon's performance under well-tuned configurations while providing a safety margin against divergence, making it suitable for large-scale LLM training where failed runs are extremely costly.

## Conclusion

The paper introduces **Musec**, which replaces Muon's spectral flattening with spectral clipping to stabilize training while preserving fast convergence. Key contributions:

1. **Theoretical**: First convergence guarantees for spectral clipping-based Muon variants in nonconvex nonsmooth optimization, with optimal $\mathcal{O}(r^{3/2}\delta^{-1}\epsilon^{-3})$ complexity matching the optimal stochastic first-order method.

2. **Algorithmic**: Development of **Soft Musec**, an efficient SVD-free implementation using smooth spectral saturation via coupled Newton–Schulz iterations.

3. **Empirical**: Consistent stability improvements over existing Muon variants across multiple datasets (FineWeb, OpenWebText, C4) and model sizes (491M to 1.63B parameters), with stable convergence in settings where Muon and MuonClip diverge.

**Future directions** suggested by the work include:
- Further investigation of spectral clipping as an optimizer-agnostic wrapper (as explored concurrently by SPECTRA)
- Application of Musec to other matrix-aware optimizers beyond Muon
- Extension of the theoretical analysis to other smoothness conditions or problem structures

---

_Markdown view of https://picx.dev/p/pWlpPO, served by PicX — AI-generated visual whiteboard summaries of research papers._