Summary (Overview)

• First last-iterate guarantees for practical finite-step Muon: This paper establishes the first last-iterate gradient convergence results for Muon with finitely many quintic Newton–Schulz (NS) iterations on general smooth, lower-bounded nonconvex objectives, without structural assumptions like Polyak–Łojasiewicz or star-convexity.

• Deterministic setting: Without input stabilization (ε = 0), Muon can cycle indefinitely even on a smooth convex quadratic (Proposition 3.1). With positive stabilization (ε > 0), last-iterate gradient convergence holds for any fixed momentum coefficient β < 1 and sufficiently small constant learning rate (Theorem 3.2), with square-summable gradients.

• Stochastic setting: Persistent bounded, unbiased noise prevents convergence with fixed learning rates and momentum, even with positive stabilization (Proposition 4.1). Combining decreasing learning rates with increasing momentum yields almost-sure and mean-square last-iterate gradient convergence (Theorem 4.2).

• Rate guarantees: A smooth counterexample shows no pointwise polynomial rate is possible in either regime (Proposition 5.1). The paper establishes density-one convergence rates, guaranteeing polynomial gradient decay for all but a vanishing fraction of iterations (Theorem 5.3).

• Key technical contributions: A coupled descent–tracking Lyapunov analysis, momentum-based noise filtering with time-varying hyperparameters, and a summability-to-density counting argument for quantitative rates.

---

Introduction and Theoretical Foundation

Background and Motivation

Muon has emerged as a successful alternative to AdamW for large-scale language-model training, with scaling experiments on Moonlight reporting roughly twice the training-compute efficiency of AdamW, and the Muon-derived MuonClip optimizer used to pretrain the trillion-parameter Kimi K2 model [17, 20]. Muon exploits the matrix structure of neural-network parameters by applying a spectral transformation to momentum-based updates.

In its idealized formulation, Muon uses the polar factor of the momentum matrix as a steepest-descent direction under a spectral-norm constraint [1, 32]. Practical implementations instead use a small, fixed number of quintic Newton–Schulz (NS) iterations, typically with a positive input-normalization stabilizer [14, 15, 27].

Key Theoretical Issues

Existing analyses of Muon establish average- or best-iterate stationarity guarantees for smooth nonconvex objectives [5, 12, 16, 18, 23, 26, 30, 32, 33, 35]. Such guarantees ensure a nearly stationary iterate can be found, but they do not rule out large gradients at individual iterations, even late in training. This motivates studying last-iterate stationarity:

$$\|\nabla F(W_t)\| \longrightarrow 0 \quad \text{as } t \to \infty$$

Two distinct challenges arise:

1. Instability from normalized updates: Even with exact gradients, normalized updates with constant learning rate can exhibit persistent oscillations near stationary points.
2. Persistent fluctuations from stochastic gradients: Mini-batch gradient estimates introduce noise that persists when batch size is fixed.

Problem Setup

Consider a single matrix parameter $W \in \mathbb{R}^{m \times n}$ and a continuously differentiable loss $F: \mathbb{R}^{m \times n} \to \mathbb{R}$. The standing assumption is:

Assumption 2.1 (Global smoothness and lower boundedness): There exist $L > 0$ and $F_{\inf} \in \mathbb{R}$ such that for all $X, Y \in \mathbb{R}^{m \times n}$:

$$F(X) \geq F_{\inf}, \quad \|\nabla F(X) - \nabla F(Y)\| \leq L\|X - Y\|$$

---

Methodology

The Finite-Step Muon Update

Given deterministic sequences $\eta_t > 0$, $\beta_t \in [0, 1)$, and $\theta_t \in [0, 1]$, the finite-step Muon update is:

$$M_t = \beta_t M_{t-1} + (1 - \beta_t) G_t \tag{2.1}$$ $$N_t = \theta_t M_t + (1 - \theta_t) G_t \tag{2.2}$$ $$D_t = \Phi_\varepsilon(N_t)$$ $$W_{t+1} = W_t - \eta_t D_t \tag{2.3}$$

where $G_t = g_t + \zeta_t$ is a possibly perturbed gradient ($g_t = \nabla F(W_t)$).

For fixed $J \geq 1$ and $\varepsilon \geq 0$, $\Phi_\varepsilon(N)$ is the output $Z_J$ of the Newton–Schulz iteration:

$$Z_0 = \frac{N}{\|N\|_F + \varepsilon}, \quad Z_{j+1} = aZ_j + bZ_j Z_j^\top Z_j + c(Z_j Z_j^\top)^2 Z_j \tag{2.4}$$

with the practical quintic coefficients:

$$(a, b, c) = (3.4445, -4.7750, 2.0315) \tag{2.5}$$

Key Spectral Properties (Lemma 2.2)

For each fixed finite $J \geq 1$, $0 < \ell_J \leq u_J < \infty$. For any matrix $N$ with $R = \|N\|_F$:

$$\ell_J \frac{R}{R + \varepsilon} \leq \|\Phi_\varepsilon(N)\|_F \leq u_J \frac{R}{R + \varepsilon} \leq u_J \tag{2.6}$$ $$\langle N, \Phi_\varepsilon(N)\rangle_F \geq \ell_J \frac{R^2}{R + \varepsilon} \tag{2.7}$$

If $\varepsilon > 0$, additionally:

$$\langle N, \Phi_\varepsilon(N)\rangle_F \geq \frac{\varepsilon}{u_J}\|\Phi_\varepsilon(N)\|_F^2 \tag{2.8}$$

The sector inequality (2.8) is the key driver of the fixed-step Lyapunov analysis.

Analysis Regimes

1. Deterministic setting: Exact gradients ($G_t = g_t$), constant learning rate $\eta$, fixed momentum $\beta$.
2. Stochastic setting: Conditionally unbiased, bounded-variance noise with decreasing learning rates and increasing momentum.

---

Empirical Validation / Results

Deterministic Setting: Stabilization Enables Fixed-Step Convergence

Proposition 3.1 (A parameter two-cycle when ε = 0): Take $F(w) = 0.01w^2$, exact gradients, $\varepsilon = 0$, fixed $\eta > 0$, $\beta \in [0, 1)$, and $\theta \in [0, 1]$. Let $c_J = \varphi_J(1) > 0$ and $\delta = \eta c_J$. With $w_1 = \delta/2$ and $M_0 = 0$, the iterates satisfy:

$$w_t = (-1)^{t-1}\frac{\delta}{2}, \quad |\nabla F(w_t)| = 0.01\delta > 0 \quad (t \geq 1)$$

The two-cycle occurs for every fixed $\beta < 1$, independently of the Nesterov-style mixing parameter.

Theorem 3.2 (Fixed-step full-sequence stationarity: exact gradients): Suppose Assumption 2.1 holds. In the deterministic setting where $G_t = g_t$, fix finite $J \geq 1$, $\varepsilon > 0$, $\beta \in [0, 1)$, and $\theta \in [0, 1]$, and let the constant step size satisfy:

$$0 < \eta \leq \frac{\varepsilon(1 - \beta)}{4Lu_J} = \frac{\kappa\alpha}{4L} \tag{3.2}$$

Then:

$$\sum_{t=1}^{\infty} \|D_t\|^2 < \infty, \quad \sum_{t=1}^{\infty} \|M_t - g_t\|^2 < \infty, \quad \sum_{t=1}^{\infty} \|g_t\|^2 < \infty \tag{3.3}$$

In particular, $\|\nabla F(W_t)\| \to 0$.

Key Lyapunov Analysis

The proof combines two complementary lemmas:

Lemma 3.3 (Objective descent):

$$F(W_{t+1}) - F(W_t) \leq -\left(\frac{\eta\kappa}{2} - \frac{L\eta^2}{2}\right)\|D_t\|^2 + \frac{\eta}{2\kappa}\|e_t\|^2 \tag{3.4}$$

Lemma 3.4 (Momentum tracking):

$$\|e_{t+1}\|^2 \leq \frac{1 + \beta}{2}\|e_t\|^2 + \frac{\beta(1 + \beta)L^2\eta^2}{1 - \beta}\|D_t\|^2 \tag{3.5}$$

Combining these with the Lyapunov function $V_t := F(W_t) + \lambda\|e_t\|^2$ where $\lambda := \frac{2\eta}{\kappa(1-\beta)}$ yields:

$$V_{t+1} - V_t \leq -\frac{\eta\kappa}{4}\|D_t\|^2 - \frac{\eta}{2\kappa}\|e_t\|^2 \tag{3.6}$$

Stochastic Setting: Persistent Noise

Proposition 4.1 (Persistent symmetric noise at a fixed step): Let $F(w) = 0.01w^2$ and $G_t = 0.02w_t + \zeta_t$, where $\zeta_t$ are independent equiprobable variables in $\{+\sigma, -\sigma\}$, with $\sigma > 0$. Fix $\eta > 0$, $\beta \in [0, 1)$, $\theta \in [0, 1]$, and $\varepsilon \geq 0$. Then:

$$\limsup_{t \to \infty} \mathbb{E}|\nabla F(w_t)| \geq \frac{0.02\eta d_0}{4} > 0 \tag{4.1}$$

The gradient does not converge to zero in probability.

Theorem 4.2 (Full-sequence stationarity under persistent noise): Suppose Assumption 2.1 and the noise condition (4.3) hold. Fix finite $J \geq 1$ and $\varepsilon \geq 0$; use the schedules:

$$\eta_t = \eta_0 t^{-p}, \quad \beta_t = 1 - \alpha_0 t^{-q}, \quad \eta_0 > 0, \quad 0 < \alpha_0 \leq 1 \tag{4.4}$$

with exponents in the admissible region:

$$\frac{3}{4} < p \leq 1, \quad 2(1 - p) < q < 2p - 1 \tag{4.5}$$

Then:

$$\|g_t\| \to 0 \text{ a.s.}, \quad \mathbb{E}\|g_t\|^2 \to 0 \tag{4.6}$$ $$\sum_{t=1}^{\infty} \eta_t \psi_\varepsilon(\|g_t\|) < \infty, \quad \sum_{t=1}^{\infty} \eta_t \|g_t\|^2 < \infty \text{ a.s.} \tag{4.7}$$

Density-One Rates

Proposition 5.1 (No polynomial pointwise rate in either regime): A smooth objective $F$ is constructed with sparse gradient peaks such that for both regimes, $\limsup_{t \to \infty} t^\nu |F'(w_t)| = +\infty$ for every $\nu > 0$, despite all convergence assumptions holding.

Definition 5.2 (Density-one rate): For a nonnegative sequence $h_t$ and a positive reference rate $\rho_t$, write $h_t = o_{\text{dens}}(\rho_t)$ if:

$$\forall c > 0, \quad \lim_{T \to \infty} \frac{1}{T} \#\{1 \leq t \leq T : h_t > c\rho_t\} = 0$$

Theorem 5.3 (Density-one gradient rates for finite-step Muon):

• Under Theorem 3.2 assumptions (deterministic, fixed-step): $\|g_t\| = o_{\text{dens}}(t^{-1/2})$
• Under Theorem 4.2 assumptions with $p < 1$:
  • For $\varepsilon > 0$: $\|g_t\| = o_{\text{dens}}(t^{-(1-p)/2})$
  • For $\varepsilon = 0$: $\|g_t\| = o_{\text{dens}}(t^{-(1-p)})$

As $p \downarrow 3/4$, one can achieve $\|g_t\| = o_{\text{dens}}(t^{-1/8+\delta})$ for $\varepsilon > 0$ and $\|g_t\| = o_{\text{dens}}(t^{-1/4+\delta})$ for $\varepsilon = 0$.

---

Theoretical and Practical Implications

Theoretical Contributions

1. First last-iterate guarantees for practical Muon: This work closes a gap between average-/best-iterate analyses and last-iterate convergence for finite-step Muon on general nonconvex objectives.

2. Role of input stabilization: The paper precisely characterizes when stabilization is necessary (constant-step deterministic setting) and when it is not (diminishing-step setting), clarifying the geometry of the finite-NS map.

3. Momentum as noise filter: The balanced schedule analysis reveals the fundamental trade-off: momentum averaging reduces noise at rate $t^{-q/2}$ while gradient drift introduces lag at rate $t^{q-p}$. The optimal balance $q = 2p/3$ gives tracking error $O(t^{-p/3})$.

4. Density-one rates as a new metric: The paper introduces a practical notion of convergence rate that captures trajectory-level behavior while accommodating sparse exceptional iterations.

Practical Implications

• Stabilization is essential for fixed-learning-rate training: practitioners should maintain positive $\varepsilon$ in the NS input normalization.
• For stochastic training, decreasing learning rates with increasing momentum is necessary for last-iterate convergence; the admissible region $3/4 < p \leq 1$ with $2(1-p) < q < 2p-1$ provides explicit hyperparameter guidance.
• Density-one rates provide a realistic quantitative view of training progress: gradient decay at polynomial rates holds for asymptotically all late iterations, even when occasional gradient spikes occur.

Comparison with Existing Work

---

Conclusion

The paper identifies two distinct convergence obstructions for finite-step Muon:

1. Constant-step cycling without input stabilization ($\varepsilon = 0$), demonstrated via a two-cycle on a smooth convex quadratic.
2. Persistent-noise floor under fixed learning rates and momentum, even with positive stabilization.

These motivate two complementary convergence mechanisms:

• Positive stabilization combined with a coupled descent–tracking Lyapunov principle yields convergence with exact gradients, fixed steps, and arbitrary fixed momentum below one.
• Coordinated diminishing steps and increasing momentum extend full-sequence stationarity to persistent unbiased bounded-variance stochastic noise.

The same convergence regimes provide gradient-energy bounds supporting density-one convergence rates, which quantify gradient decay at asymptotically almost all late iterations of a single run. Sparse-peak constructions explain the distinction from pointwise polynomial bounds.

Future directions suggested by this work include:

• Relaxing the conservative step-size condition in Theorem 3.2
• Extending to multiple matrix parameters with weight decay
• Investigating tighter rates or lower bounds for the density-one guarantees
• Understanding the optimal balance between learning rate decay and momentum increase in practice