# Last-Iterate Stationarity of Muon with Finitely Many Newton-Schulz Iterations > Finite-step Muon achieves last-iterate gradient convergence on general nonconvex objectives via input stabilization with fixed steps, or via decreasing learning rates paired with increasing momentum under stochastic noise. - **Source:** [arXiv](https://arxiv.org/abs/2610.34158) - **Published:** 2026-10-31 - **Permalink:** https://picx.dev/p/azP9kR - **Whiteboard:** https://picx.dev/p/azP9kR/image ## Summary ## Summary (Overview) - **First last-iterate guarantees for practical finite-step Muon**: This paper establishes the first last-iterate gradient convergence results for Muon with finitely many quintic Newton–Schulz (NS) iterations on general smooth, lower-bounded nonconvex objectives, without structural assumptions like Polyak–Łojasiewicz or star-convexity. - **Deterministic setting**: Without input stabilization (ε = 0), Muon can cycle indefinitely even on a smooth convex quadratic (Proposition 3.1). With positive stabilization (ε > 0), last-iterate gradient convergence holds for any fixed momentum coefficient β < 1 and sufficiently small constant learning rate (Theorem 3.2), with square-summable gradients. - **Stochastic setting**: Persistent bounded, unbiased noise prevents convergence with fixed learning rates and momentum, even with positive stabilization (Proposition 4.1). Combining decreasing learning rates with increasing momentum yields almost-sure and mean-square last-iterate gradient convergence (Theorem 4.2). - **Rate guarantees**: A smooth counterexample shows no pointwise polynomial rate is possible in either regime (Proposition 5.1). The paper establishes density-one convergence rates, guaranteeing polynomial gradient decay for all but a vanishing fraction of iterations (Theorem 5.3). - **Key technical contributions**: A coupled descent–tracking Lyapunov analysis, momentum-based noise filtering with time-varying hyperparameters, and a summability-to-density counting argument for quantitative rates. --- ## Introduction and Theoretical Foundation ### Background and Motivation Muon has emerged as a successful alternative to AdamW for large-scale language-model training, with scaling experiments on Moonlight reporting roughly twice the training-compute efficiency of AdamW, and the Muon-derived MuonClip optimizer used to pretrain the trillion-parameter Kimi K2 model [17, 20]. Muon exploits the matrix structure of neural-network parameters by applying a spectral transformation to momentum-based updates. In its idealized formulation, Muon uses the **polar factor** of the momentum matrix as a steepest-descent direction under a spectral-norm constraint [1, 32]. Practical implementations instead use a small, fixed number of **quintic Newton–Schulz (NS) iterations**, typically with a positive input-normalization stabilizer [14, 15, 27]. ### Key Theoretical Issues Existing analyses of Muon establish **average- or best-iterate** stationarity guarantees for smooth nonconvex objectives [5, 12, 16, 18, 23, 26, 30, 32, 33, 35]. Such guarantees ensure a nearly stationary iterate can be found, but they do not rule out large gradients at individual iterations, even late in training. This motivates studying **last-iterate stationarity**: $$\|\nabla F(W_t)\| \longrightarrow 0 \quad \text{as } t \to \infty$$ Two distinct challenges arise: 1. **Instability from normalized updates**: Even with exact gradients, normalized updates with constant learning rate can exhibit persistent oscillations near stationary points. 2. **Persistent fluctuations from stochastic gradients**: Mini-batch gradient estimates introduce noise that persists when batch size is fixed. ### Problem Setup Consider a single matrix parameter $W \in \mathbb{R}^{m \times n}$ and a continuously differentiable loss $F: \mathbb{R}^{m \times n} \to \mathbb{R}$. The standing assumption is: **Assumption 2.1 (Global smoothness and lower boundedness)**: There exist $L > 0$ and $F_{\inf} \in \mathbb{R}$ such that for all $X, Y \in \mathbb{R}^{m \times n}$: $$F(X) \geq F_{\inf}, \quad \|\nabla F(X) - \nabla F(Y)\| \leq L\|X - Y\|$$ --- ## Methodology ### The Finite-Step Muon Update Given deterministic sequences $\eta_t > 0$, $\beta_t \in [0, 1)$, and $\theta_t \in [0, 1]$, the finite-step Muon update is: $$M_t = \beta_t M_{t-1} + (1 - \beta_t) G_t \tag{2.1}$$ $$N_t = \theta_t M_t + (1 - \theta_t) G_t \tag{2.2}$$ $$D_t = \Phi_\varepsilon(N_t)$$ $$W_{t+1} = W_t - \eta_t D_t \tag{2.3}$$ where $G_t = g_t + \zeta_t$ is a possibly perturbed gradient ($g_t = \nabla F(W_t)$). For fixed $J \geq 1$ and $\varepsilon \geq 0$, $\Phi_\varepsilon(N)$ is the output $Z_J$ of the Newton–Schulz iteration: $$Z_0 = \frac{N}{\|N\|_F + \varepsilon}, \quad Z_{j+1} = aZ_j + bZ_j Z_j^\top Z_j + c(Z_j Z_j^\top)^2 Z_j \tag{2.4}$$ with the practical quintic coefficients: $$(a, b, c) = (3.4445, -4.7750, 2.0315) \tag{2.5}$$ ### Key Spectral Properties (Lemma 2.2) For each fixed finite $J \geq 1$, $0 < \ell_J \leq u_J < \infty$. For any matrix $N$ with $R = \|N\|_F$: $$\ell_J \frac{R}{R + \varepsilon} \leq \|\Phi_\varepsilon(N)\|_F \leq u_J \frac{R}{R + \varepsilon} \leq u_J \tag{2.6}$$ $$\langle N, \Phi_\varepsilon(N)\rangle_F \geq \ell_J \frac{R^2}{R + \varepsilon} \tag{2.7}$$ If $\varepsilon > 0$, additionally: $$\langle N, \Phi_\varepsilon(N)\rangle_F \geq \frac{\varepsilon}{u_J}\|\Phi_\varepsilon(N)\|_F^2 \tag{2.8}$$ The sector inequality (2.8) is the key driver of the fixed-step Lyapunov analysis. ### Analysis Regimes 1. **Deterministic setting**: Exact gradients ($G_t = g_t$), constant learning rate $\eta$, fixed momentum $\beta$. 2. **Stochastic setting**: Conditionally unbiased, bounded-variance noise with decreasing learning rates and increasing momentum. --- ## Empirical Validation / Results ### Deterministic Setting: Stabilization Enables Fixed-Step Convergence **Proposition 3.1 (A parameter two-cycle when ε = 0)**: Take $F(w) = 0.01w^2$, exact gradients, $\varepsilon = 0$, fixed $\eta > 0$, $\beta \in [0, 1)$, and $\theta \in [0, 1]$. Let $c_J = \varphi_J(1) > 0$ and $\delta = \eta c_J$. With $w_1 = \delta/2$ and $M_0 = 0$, the iterates satisfy: $$w_t = (-1)^{t-1}\frac{\delta}{2}, \quad |\nabla F(w_t)| = 0.01\delta > 0 \quad (t \geq 1)$$ The two-cycle occurs for every fixed $\beta < 1$, independently of the Nesterov-style mixing parameter. **Theorem 3.2 (Fixed-step full-sequence stationarity: exact gradients)**: Suppose Assumption 2.1 holds. In the deterministic setting where $G_t = g_t$, fix finite $J \geq 1$, $\varepsilon > 0$, $\beta \in [0, 1)$, and $\theta \in [0, 1]$, and let the constant step size satisfy: $$0 < \eta \leq \frac{\varepsilon(1 - \beta)}{4Lu_J} = \frac{\kappa\alpha}{4L} \tag{3.2}$$ Then: $$\sum_{t=1}^{\infty} \|D_t\|^2 < \infty, \quad \sum_{t=1}^{\infty} \|M_t - g_t\|^2 < \infty, \quad \sum_{t=1}^{\infty} \|g_t\|^2 < \infty \tag{3.3}$$ In particular, $\|\nabla F(W_t)\| \to 0$. ### Key Lyapunov Analysis The proof combines two complementary lemmas: **Lemma 3.3 (Objective descent)**: $$F(W_{t+1}) - F(W_t) \leq -\left(\frac{\eta\kappa}{2} - \frac{L\eta^2}{2}\right)\|D_t\|^2 + \frac{\eta}{2\kappa}\|e_t\|^2 \tag{3.4}$$ **Lemma 3.4 (Momentum tracking)**: $$\|e_{t+1}\|^2 \leq \frac{1 + \beta}{2}\|e_t\|^2 + \frac{\beta(1 + \beta)L^2\eta^2}{1 - \beta}\|D_t\|^2 \tag{3.5}$$ Combining these with the Lyapunov function $V_t := F(W_t) + \lambda\|e_t\|^2$ where $\lambda := \frac{2\eta}{\kappa(1-\beta)}$ yields: $$V_{t+1} - V_t \leq -\frac{\eta\kappa}{4}\|D_t\|^2 - \frac{\eta}{2\kappa}\|e_t\|^2 \tag{3.6}$$ ### Stochastic Setting: Persistent Noise **Proposition 4.1 (Persistent symmetric noise at a fixed step)**: Let $F(w) = 0.01w^2$ and $G_t = 0.02w_t + \zeta_t$, where $\zeta_t$ are independent equiprobable variables in $\{+\sigma, -\sigma\}$, with $\sigma > 0$. Fix $\eta > 0$, $\beta \in [0, 1)$, $\theta \in [0, 1]$, and $\varepsilon \geq 0$. Then: $$\limsup_{t \to \infty} \mathbb{E}|\nabla F(w_t)| \geq \frac{0.02\eta d_0}{4} > 0 \tag{4.1}$$ The gradient does not converge to zero in probability. **Theorem 4.2 (Full-sequence stationarity under persistent noise)**: Suppose Assumption 2.1 and the noise condition (4.3) hold. Fix finite $J \geq 1$ and $\varepsilon \geq 0$; use the schedules: $$\eta_t = \eta_0 t^{-p}, \quad \beta_t = 1 - \alpha_0 t^{-q}, \quad \eta_0 > 0, \quad 0 < \alpha_0 \leq 1 \tag{4.4}$$ with exponents in the admissible region: $$\frac{3}{4} < p \leq 1, \quad 2(1 - p) < q < 2p - 1 \tag{4.5}$$ Then: $$\|g_t\| \to 0 \text{ a.s.}, \quad \mathbb{E}\|g_t\|^2 \to 0 \tag{4.6}$$ $$\sum_{t=1}^{\infty} \eta_t \psi_\varepsilon(\|g_t\|) < \infty, \quad \sum_{t=1}^{\infty} \eta_t \|g_t\|^2 < \infty \text{ a.s.} \tag{4.7}$$ ### Density-One Rates **Proposition 5.1 (No polynomial pointwise rate in either regime)**: A smooth objective $F$ is constructed with sparse gradient peaks such that for both regimes, $\limsup_{t \to \infty} t^\nu |F'(w_t)| = +\infty$ for every $\nu > 0$, despite all convergence assumptions holding. **Definition 5.2 (Density-one rate)**: For a nonnegative sequence $h_t$ and a positive reference rate $\rho_t$, write $h_t = o_{\text{dens}}(\rho_t)$ if: $$\forall c > 0, \quad \lim_{T \to \infty} \frac{1}{T} \#\{1 \leq t \leq T : h_t > c\rho_t\} = 0$$ **Theorem 5.3 (Density-one gradient rates for finite-step Muon)**: - Under Theorem 3.2 assumptions (deterministic, fixed-step): $\|g_t\| = o_{\text{dens}}(t^{-1/2})$ - Under Theorem 4.2 assumptions with $p < 1$: - For $\varepsilon > 0$: $\|g_t\| = o_{\text{dens}}(t^{-(1-p)/2})$ - For $\varepsilon = 0$: $\|g_t\| = o_{\text{dens}}(t^{-(1-p)})$ As $p \downarrow 3/4$, one can achieve $\|g_t\| = o_{\text{dens}}(t^{-1/8+\delta})$ for $\varepsilon > 0$ and $\|g_t\| = o_{\text{dens}}(t^{-1/4+\delta})$ for $\varepsilon = 0$. --- ## Theoretical and Practical Implications ### Theoretical Contributions 1. **First last-iterate guarantees for practical Muon**: This work closes a gap between average-/best-iterate analyses and last-iterate convergence for finite-step Muon on general nonconvex objectives. 2. **Role of input stabilization**: The paper precisely characterizes when stabilization is necessary (constant-step deterministic setting) and when it is not (diminishing-step setting), clarifying the geometry of the finite-NS map. 3. **Momentum as noise filter**: The balanced schedule analysis reveals the fundamental trade-off: momentum averaging reduces noise at rate $t^{-q/2}$ while gradient drift introduces lag at rate $t^{q-p}$. The optimal balance $q = 2p/3$ gives tracking error $O(t^{-p/3})$. 4. **Density-one rates as a new metric**: The paper introduces a practical notion of convergence rate that captures trajectory-level behavior while accommodating sparse exceptional iterations. ### Practical Implications - **Stabilization is essential** for fixed-learning-rate training: practitioners should maintain positive $\varepsilon$ in the NS input normalization. - **For stochastic training**, decreasing learning rates with increasing momentum is necessary for last-iterate convergence; the admissible region $3/4 < p \leq 1$ with $2(1-p) < q < 2p-1$ provides explicit hyperparameter guidance. - **Density-one rates** provide a realistic quantitative view of training progress: gradient decay at polynomial rates holds for asymptotically all late iterations, even when occasional gradient spikes occur. ### Comparison with Existing Work | Work | Type | Method | Nonconvex | Convergence | |------|------|--------|-----------|-------------| | Li–Hong [18] | Stoch. | Polar | ✓ | Avg. | | Kim–Oh [16] | Stoch. | Finite-NS (Taylor) | ✓ | Avg. | | De Oliveira et al. [7] | Det. | Soft Muon | PL | Last / linear obj. | | Chang et al. [3] | Stoch. | Muon–MVR | ✓ (avg.)/PL(last) | Avg. / last obj. | | **This work** | **Det.** | **Practical** | **✓** | **Last** | | **This work** | **Stoch.** | **Practical** | **✓** | **Last** | --- ## Conclusion The paper identifies two distinct convergence obstructions for finite-step Muon: 1. **Constant-step cycling** without input stabilization ($\varepsilon = 0$), demonstrated via a two-cycle on a smooth convex quadratic. 2. **Persistent-noise floor** under fixed learning rates and momentum, even with positive stabilization. These motivate two complementary convergence mechanisms: - **Positive stabilization** combined with a **coupled descent–tracking Lyapunov principle** yields convergence with exact gradients, fixed steps, and arbitrary fixed momentum below one. - **Coordinated diminishing steps and increasing momentum** extend full-sequence stationarity to persistent unbiased bounded-variance stochastic noise. The same convergence regimes provide gradient-energy bounds supporting **density-one convergence rates**, which quantify gradient decay at asymptotically almost all late iterations of a single run. Sparse-peak constructions explain the distinction from pointwise polynomial bounds. **Future directions** suggested by this work include: - Relaxing the conservative step-size condition in Theorem 3.2 - Extending to multiple matrix parameters with weight decay - Investigating tighter rates or lower bounds for the density-one guarantees - Understanding the optimal balance between learning rate decay and momentum increase in practice --- _Markdown view of https://picx.dev/p/azP9kR, served by PicX — AI-generated visual whiteboard summaries of research papers._