New Axis for Optimizer Benchmarking and Theorization of Muon
Highlights of This Issue
This issue advances both main lines of the optimizer system by one step: one is to continue expanding the methodological axis of optimizer benchmarking, and the other is to provide unprecedented theorization for Muon-type updates at the statistical and quadratic form levels. A Statistical Viewpoint on Adam, Shampoo and Muon (https://arxiv.org/abs/2610.24839) for the first time unifies the three major optimizers into the statistical framework of "whitened momentum methods," and provides a Bayesian interpretation of the Muon update direction as the MAP of a Matrix Langevin distribution, deriving new methods from it; A Quadratic Lens on Muon's Polar Update (https://arxiv.org/abs/2610.30280) provides exact one-step and trajectory-level analysis of PolarGD on matrix quadratic forms, reinterpreting finite Newton–Schulz iterations from "approximation error" to "beneficial damping." Another piece of the theoretical puzzle comes from How Does Orthogonalization Adapt to the Neural Network Hessian Structure? (https://arxiv.org/abs/2610.28975), which proves from the asymptotic theory of the gradient outer product that Muon's implicit preconditioner tends to be diagonally dominant as the hidden dimension grows, thereby matching the block-diagonal structure of the Hessian—three works together advance Muon from "effective in engineering" to the accountability level of "why it works."
The methodology of optimizer benchmarking is also tightening. When Does Muon Actually Win? (https://arxiv.org/abs/2610.21555) for the first time uses a pairwise protocol to simultaneously control the three axes of data, wall-clock time, and tuning budget, revealing that the winner flips with the axis being equalized, directly questioning the common practice of reporting only compute-matched comparisons—this forms a three-way extension with the existing evaluation methodology in this direction (batch axis in Issue 9, training duration axis in Issue 7). On the pretraining objective side, ECHO (https://arxiv.org/abs/2610.24766) refactors multi-token prediction from an auxiliary module into causal self-recurrence, allowing multi-horizon supervision to directly act on the backbone; Looped Uniform Diffusion Language Models (https://arxiv.org/abs/2610.30066) for the first time introduces a looped denoiser into uniform diffusion language models, achieving the validation NLL of the non-looped baseline with about three times fewer FLOPs under the iso-FLOP protocol, continuing the compute-matched requirement in this direction for the debate on data efficiency between diffusion and autoregressive models.
It is worth noting that most of the optimizer theory work in this issue holds under initialization or quadratic settings (Quadratic Lens stops at 90M diagnostics, How Does Orthogonalization holds only in expectation), and its transfer to large-scale training still needs self-assessment; in contrast, When Does Muon Actually Win explicitly acknowledges that its small-scale single-GPU conclusions do not constitute large-scale rankings, and readers should focus on the pairwise budget protocol itself rather than performance numbers.
Community and Dynamics
- Reinforcement Learning on Pre-Training Data (https://aclanthology.org/2026.acl-long.506/) directly applies reinforcement learning to pretraining data: it proposes a next-segment reasoning objective, using a generative reward model to score whether the predicted segment is semantically equivalent to the true continuation as the reward, constituting a concrete instance of the "RL as pretraining objective" route. On Qwen3-8B-Base, continued pretraining +4.6%, post-training +3.4%, and segment-level rewards are better than token-level rewards for RPT/RLP, providing a reference alongside the MTP/self-distillation alternative pretraining signals long tracked in this direction.
Open Questions
- The MND proposed in A Statistical Viewpoint has not been compared horizontally with spectral-type optimizers (SOAP, Muon spectral shaping family). If the "whitened momentum" framework can cover methods with spectral preconditioning, can it constitute a single statistical model unifying all mainstream optimizers?
- When Does Muon Actually Win shows that on small-scale language models, Muon consistently wins data efficiency but loses wall-clock time (optimizer steps account for 56–70% of training time). After switching to optimized orthogonalization kernels or multi-GPU implementations, will the break-even point on the time axis shift significantly to the right, thereby flipping the conclusion at larger scales?
- ECHO's dynamic depth sampling halves the overhead of multi-horizon supervision and can extrapolate beyond training depth. Does this sampling distribution remain compute-optimal at longer horizons and larger models, and can the benefits of inference depth extrapolation and speculative decoding be stacked?
- A Quadratic Lens reinterprets finite Newton–Schulz as damped polar decomposition. Does this "damping is beneficial" reading still hold for Muon variants that modify Newton-Schulz iterations, such as spectral clipping (e.g., Musec in Issue 7), or is it altered by the clipping mechanism?
Papers in this issue
Whitening gradients with estimated covariance unifies Adam, Shampoo, and Muon, yielding MND that outperforms Shampoo and MND-Muon that improves Muon's stability.
Editor's noteFor the first time unifies Adam/Shampoo/Muon into the statistical framework of "whitened momentum methods," and provides a Bayesian interpretation that the Muon update direction is the MAP of a Matrix Langevin distribution. Based on this, it proposes MND and MND-Muon: jointly estimating the mean and Kronecker factors, extending polar decomposition to momentum in anisotropic whitening coordinates. On 124M/353M, MND relative to Shampoo and MND-Muon relative to Muon achieve final loss and stability gains, and report full learning rate grids and spike information. The unified perspective has the potential to replace the current state of analyzing each optimizer separately, and is an important reference for optimizer benchmarking and stability in this direction. The limitation is that it has not been compared with spectral-type optimizers.
ECHO reuses the backbone's final layer for multi-token prediction, matching DeepSeek's auxiliary MTP performance at half the training cost while enabling depth extrapolation for speculative decoding.
Editor's noteRefactors multi-token prediction from an auxiliary module into causal self-recurrence: reusing the backbone's last layer and output head to recursively predict future horizons, so that multi-horizon supervision directly acts on the backbone rather than being absorbed by the auxiliary module, and introduces dynamic depth sampling (DDS). At 10B MoE and 600B token multi-scale validation, depth 3 still shows continuous improvement (DeepSeek-style MTP has saturated). Compared to the latent-space variant of LoopMTP in Issue 3, this work elevates MTP from an "auxiliary objective" to a "parameter-shared self-recurrent main objective," and DDS provides a new low-cost approach for long-horizon supervision. Note that the benefits of DDS in the pretraining phase only appear after SFT, so trade-offs need to be considered.
Muon's PolarGD is gradient descent implicitly preconditioned by the iterate's covariance, not the Hessian, so Newton-Schulz's GD-like damping of tiny singular values is a feature, not a bug.
Editor's noteFor the first time provides exact one-step and trajectory-level analysis of PolarGD (the core update of Muon) on matrix quadratic forms, proving that its implicit preconditioner is determined by the current iteration covariance rather than the Hessian, and revealing that finite Newton–Schulz behaves as damped polar decomposition near rank deficiency, thus converging faster than exact polar decomposition—reinterpreting finite steps from approximation error to a beneficial damping mechanism. Compared to the stability phase diagram of Muon Sublates the Edge of Stability in Issue 9 and the zero-cost spectral estimation in Issue 8, this paper provides an exact quadratic theory for polar decomposition updates. The limitation is that the scale stops at 90M diagnostics, and transfer to large scale requires self-assessment.
Muon's orthogonalization preconditioner matches neural-network Hessian structure because the gradient self outer-product becomes diagonally dominant as hidden dimensions grow, with explicit asymptotic rates across canonical models.
Editor's noteFor the first time proves from the asymptotic theory of the gradient outer product E[GG^T] that Muon's implicit preconditioner tends to be diagonally dominant as the hidden dimension grows, thereby matching the row-block diagonal dominant structure of the neural network Hessian, and gives an explicit scaling rate for attention output weights (Θ(d_v d_0^3)). Compared to the static spectral scaling law of Spectral Scaling Laws of Muon in Issue 9 and the per-rank measurement of Spectral Allocation in Issue 4, this paper elevates the diagonal dominance of the gradient Gram from empirical observation to a verifiable asymptotic theory. The limitation is that it holds only at initialization and in expectation, does not cover training dynamics, and does not report extrapolation residuals or stability.
Selective looping in diffusion language models cuts training compute by up to 3.3x while also improving inference perplexity, with adaptive looping preserving quality using 26-35% fewer iterations.
Editor's noteFor the first time systematically introduces selective looping into uniform diffusion language models, achieving the validation NLL of the non-looped UDLM with 2.8–3.3× fewer training FLOPs under the iso-FLOP protocol, and proposes two training-time loop-count sampling distributions (FLOP-proportional is effective, noise-proportional is not). Through theoretical decomposition of retention probability and contextual prediction, it locates the loop gains mainly from improved contextual prediction of replaced tokens, directly responding to the compute-matched requirement in the debate on data efficiency between masked diffusion and autoregressive models. The scale stops at 170M, so cross-scale extrapolation requires caution.
Muon beats AdamW on data efficiency across all workloads but loses on language models under wall-clock time due to its costly orthogonalization, while winning on vision.
Editor's noteFor the first time uses a pairwise protocol to simultaneously control the three axes of data, wall-clock time, and tuning budget to compare Muon and AdamW, finding that the winner flips with the axis being equalized: at small scale, Muon consistently wins data efficiency (test loss is lower in all 20 pairwise comparisons), but loses on the wall-clock axis for language tasks, with optimizer steps accounting for 56–70% of training time. Compared to the batch axis established by The Best Optimizer Depends on Batch Size in Issue 9, this paper extends the optimizer ranking reversal into a three-way budget trade-off, being the latest axis of pairwise benchmarking protocol. Note that its scale is only 17.7–33.6M and it states that it does not constitute a large-scale ranking; the value lies in the protocol design rather than performance conclusions.
