When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark

Summary (Overview)

• This paper presents a controlled benchmark comparing Muon (a matrix-momentum optimizer with Newton–Schulz orthogonalization) against AdamW across four workloads: two causal language models (17.7M and 33.6M parameters) and two vision transformers (2.70M parameters) trained on balanced and 100:1 long-tailed CIFAR-100.
• Muon wins every equal-data comparison: validation-loss AUC is reduced by 3.25–22.44% at equal data, and test loss is lower in all 20 paired seed comparisons.
• The time-budget ranking reverses by domain: Muon improves time-normalized loss AUC by 19.68–22.31% on vision, while AdamW is better by 3.38–6.96% on language.
• The reference Muon optimizer call is expensive: consuming 56.1–69.6% of training time (vs. 6.1–9.1% for AdamW), with Muon throughput at only 34–47% of AdamW's.
• Muon has higher tuning regret: with one randomly available candidate from frozen 23-candidate pools, expected validation-AUC regret is 13.66% for Muon vs. 7.52% for AdamW.

Introduction and Theoretical Foundation

Background

The paper addresses a fundamental question in optimizer benchmarking: efficiency can mean different things—fewer training tokens, shorter runtime, or less tuning effort. These rankings can diverge when optimizers have different update costs and tuning sensitivity.

Optimizer Foundations

AdamW combines coordinate-wise adaptation with decoupled weight decay:

• Coordinate-wise adaptive learning rates
• Decoupled weight decay (Loshchilov & Hutter, 2019)
• Mature, highly optimized implementations

Muon takes a fundamentally different approach:

• Forms a matrix momentum update
• Approximately orthogonalizes it through Newton–Schulz iteration (Jordan et al., 2024; Bernstein & Newhouse, 2024)
• Large language-model experiments report favorable scaling and lower training loss per unit of compute

The Core Theoretical Mechanism

For a matrix gradient $G_t$, Muon maintains momentum and applies five steps of the quintic Newton–Schulz map to a Frobenius-normalized matrix. Writing $M_t$ for the Nesterov momentum candidate and $M_t = U\Sigma V^\top$, the resulting direction approximates $UV^\top$:

$$X_{k+1} = a X_k + b (X_k X_k^\top) X_k + c (X_k X_k^\top)^2 X_k, \quad (a, b, c) = (3.4445, -4.7750, 2.0315). \tag{1}$$

This is a practical inexact linear minimization oracle whose degenerate case has recently been analyzed (Qian & Richtárik, 2026).

Routing Strategy

Muon is applied to hidden attention and feed-forward matrices (including fused QKV projections), while embeddings, normalization parameters, biases, and output heads use AdamW—following the reference routing (Jordan et al., 2024). Muon updates approximately 26.7% and 42.2% of parameters in language-micro and language-small respectively.

Methodology

Workloads

Experimental Design

Development phase: One seed, 23 candidates per workload and strategy (accumulated through initial grid, boundary extension, and five-point audit). Selection minimizes data-normalized validation-loss AUC.

Full-horizon runs: Seeds 101–105, with both data- and time-budget comparisons using data-AUC-selected configurations.

Test isolation: All training, selection, and trajectory analyses were frozen before accessing official test sets. The final set contains 40 primary evaluations (2 optimizers × 4 workloads × 5 seeds) and 24 prespecified secondary evaluations.

Estimands

For optimizer $o$, let $L_o(x)$ denote interpolated validation loss against processed data or measured training seconds. For a paired common horizon $H$:

$$\mathrm{AUC}_o(H) = \frac{1}{H} \int_0^H L_o(x) \, dx, \quad \mathrm{gain} = 1 - \frac{\mathrm{AUC}_{Muon}}{\mathrm{AUC}_{AdamW}}. \tag{2}$$

A stable crossover is the first recorded point after which Muon remains at least 1% below AdamW through the common data horizon. The seed is the replication unit; 10,000 paired bootstrap resamples provide within-workload intervals.

Empirical Validation / Results

Data Efficiency (Equal-Data Comparison)

Muon lowers validation-loss AUC at equal data on every workload:

All 20 paired trajectories reach a stable 1%-margin crossover. Mean crossover occurs at 4.8% and 3.0% of language horizons; vision crossovers occur at the first 1% evaluation point.

Time Efficiency (Equal-Time Comparison)

The wall-clock accounting divides the workloads:

Optimizer-Step Cost

Table 1: Mean systems measurements over five seeds. The throughput ratio is Muon/AdamW. Optimizer shares cover the full optimizer call, including auxiliary AdamW in the reference Muon implementation.

Peak allocated memory is 0.6–2.4% lower for Muon, but this metric alone does not identify the cause of the slowdown.

Frozen Test Evaluation

Muon lowers test loss in every paired seed:

Across the four equally weighted workloads, mean relative test-loss reduction is 12.33% (stratified bootstrap 95% interval [11.84, 12.83]%).

Tuning-Budget Regret

Expected relative validation-AUC regret with restricted access to frozen candidate pools:

Variants and Approximations

• Late switch: Switching from AdamW to Muon after 25% of training reduces test loss by 3.97–23.40% relative to AdamW, but full-horizon Muon achieves lower loss on all workloads.
• Variance-Muon (exploratory, 3 seeds): improves language models and balanced vision but increases long-tailed test loss by 11.71%.
• Fewer Newton–Schulz steps (3 vs. 5): saves 21.87% training time with 0.74% AUC penalty.
• Restricted matrix routing: saves 28–45% with penalties of 1.13–2.76%.

Theoretical and Practical Implications

Break-Even Analysis

Let $q_o$ denote measured throughput for optimizer $o$. At time $T$, its trajectory has processed approximately $q_o T$ data units, so a time-budget comparison couples the loss curve to the implementation that produces it. Here $q_{Muon}/q_{AdamW}$ ranges from 0.339 to 0.468.

The key insight: The larger vision gains overcome the throughput gap; the smaller language gains do not. Faster orthogonalization would raise $q_{Muon}$ and move the break-even point even if the loss trajectory per token is unchanged.

Tuning Sensitivity

Within the frozen candidate pools, restricting the tuning budget increases Muon's expected regret more than AdamW's. This suggests Muon may have greater tuning sensitivity—a practical consideration for users with limited compute for hyperparameter search.

Scale and Implementation Caveats

The authors are explicit about scope limitations:

• Language models use widths 256/384 with 4,096 tokens per optimizer update on one A100
• The relative cost of orthogonalization depends on matrix dimensions, token batch, and kernel implementation
• Auxiliary AdamW covers most language-model parameters, limiting comparisons with architectures dominated by Muon-eligible matrices
• Large-scale studies report favorable Muon compute efficiency (Liu et al., 2025; Khona et al., 2026); in an 8B model with tensor parallelism, Khaled et al. (2025) report a much smaller throughput penalty

Recommendations for Credible Optimizer Comparisons

The paper argues that credible comparisons should:

1. Pair initializations across optimizers
2. Equalize tuning opportunities
3. Report both data- and time-normalized curves
4. Attribute optimizer cost separately
5. Freeze test evaluation before access

Conclusion

The paper demonstrates that "who wins" depends critically on the budget being measured:

• Under a data budget: Muon consistently wins across all four workloads, with gains ranging from 3.25% to 22.44% in validation AUC and up to 26.03% test-loss reduction.
• Under a wall-clock budget: The reference Muon implementation's high per-step cost (56.1–69.6% of training time) reverses the ranking on language models, where gains are modest, but not on vision, where gains are large enough to overcome the throughput penalty.
• Under a tuning budget: Muon exhibits higher expected regret (13.66% vs. 7.52% with one candidate), suggesting greater sensitivity to hyperparameter selection.

Future Directions

The authors identify several open questions:

• Selection stability across multiple development seeds
• Additional architectures, imbalance constructions, and longer training horizons
• Optimized kernels and multi-accelerator implementations
• The role of adaptive step scaling in reducing tuning sensitivity (Demidovich et al., 2026)

The paper's central methodological contribution is its paired, budget-aware benchmark framework—pairing initializations, equalizing tuning opportunities, reporting data- and time-normalized curves, attributing optimizer cost, and freezing test evaluation—which provides a template for future optimizer comparisons beyond the specific AdamW-vs-Muon question.