# When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark

> Muon beats AdamW on data efficiency across all workloads but loses on language models under wall-clock time due to its costly orthogonalization, while winning on vision.

- **Source:** [arXiv](https://arxiv.org/abs/2610.21555)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/X9aH6g
- **Whiteboard:** https://picx.dev/p/X9aH6g/image

## Summary

# When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark

## Summary (Overview)

- This paper presents a controlled benchmark comparing **Muon** (a matrix-momentum optimizer with Newton–Schulz orthogonalization) against **AdamW** across four workloads: two causal language models (17.7M and 33.6M parameters) and two vision transformers (2.70M parameters) trained on balanced and 100:1 long-tailed CIFAR-100.
- **Muon wins every equal-data comparison**: validation-loss AUC is reduced by 3.25–22.44% at equal data, and test loss is lower in all 20 paired seed comparisons.
- **The time-budget ranking reverses by domain**: Muon improves time-normalized loss AUC by 19.68–22.31% on vision, while AdamW is better by 3.38–6.96% on language.
- **The reference Muon optimizer call is expensive**: consuming 56.1–69.6% of training time (vs. 6.1–9.1% for AdamW), with Muon throughput at only 34–47% of AdamW's.
- **Muon has higher tuning regret**: with one randomly available candidate from frozen 23-candidate pools, expected validation-AUC regret is 13.66% for Muon vs. 7.52% for AdamW.

## Introduction and Theoretical Foundation

### Background

The paper addresses a fundamental question in optimizer benchmarking: **efficiency can mean different things**—fewer training tokens, shorter runtime, or less tuning effort. These rankings can diverge when optimizers have different update costs and tuning sensitivity.

### Optimizer Foundations

**AdamW** combines coordinate-wise adaptation with decoupled weight decay:
- Coordinate-wise adaptive learning rates
- Decoupled weight decay (Loshchilov & Hutter, 2019)
- Mature, highly optimized implementations

**Muon** takes a fundamentally different approach:
- Forms a matrix momentum update
- Approximately orthogonalizes it through **Newton–Schulz iteration** (Jordan et al., 2024; Bernstein & Newhouse, 2024)
- Large language-model experiments report favorable scaling and lower training loss per unit of compute

### The Core Theoretical Mechanism

For a matrix gradient $G_t$, Muon maintains momentum and applies five steps of the quintic Newton–Schulz map to a Frobenius-normalized matrix. Writing $M_t$ for the Nesterov momentum candidate and $M_t = U\Sigma V^\top$, the resulting direction approximates $UV^\top$:

$$X_{k+1} = a X_k + b (X_k X_k^\top) X_k + c (X_k X_k^\top)^2 X_k, \quad (a, b, c) = (3.4445, -4.7750, 2.0315). \tag{1}$$

This is a practical inexact linear minimization oracle whose degenerate case has recently been analyzed (Qian & Richtárik, 2026).

### Routing Strategy

Muon is applied to hidden attention and feed-forward matrices (including fused QKV projections), while embeddings, normalization parameters, biases, and output heads use AdamW—following the reference routing (Jordan et al., 2024). Muon updates approximately 26.7% and 42.2% of parameters in language-micro and language-small respectively.

## Methodology

### Workloads

| Workload | Architecture | Parameters | Training Budget |
|----------|-------------|------------|-----------------|
| Language-micro | 6 layers, width 256, decoder-only | 17.7M | 5,000 steps, 100M tokens FineWeb-Edu |
| Language-small | 8 layers, width 384, decoder-only | 33.6M | 5,000 steps, 100M tokens FineWeb-Edu |
| Vision-balanced | 6-layer ViT, width 192, 4×4 patches | 2.70M | 100 epochs, 45,000 balanced CIFAR-100 |
| Vision-long-tail | Same ViT | 2.70M | 100 epochs, 9,754 examples, 100:1 imbalance |

### Experimental Design

**Development phase:** One seed, 23 candidates per workload and strategy (accumulated through initial grid, boundary extension, and five-point audit). Selection minimizes data-normalized validation-loss AUC.

**Full-horizon runs:** Seeds 101–105, with both data- and time-budget comparisons using data-AUC-selected configurations.

**Test isolation:** All training, selection, and trajectory analyses were frozen before accessing official test sets. The final set contains 40 primary evaluations (2 optimizers × 4 workloads × 5 seeds) and 24 prespecified secondary evaluations.

### Estimands

For optimizer $o$, let $L_o(x)$ denote interpolated validation loss against processed data or measured training seconds. For a paired common horizon $H$:

$$\mathrm{AUC}_o(H) = \frac{1}{H} \int_0^H L_o(x) \, dx, \quad \mathrm{gain} = 1 - \frac{\mathrm{AUC}_{Muon}}{\mathrm{AUC}_{AdamW}}. \tag{2}$$

A **stable crossover** is the first recorded point after which Muon remains at least 1% below AdamW through the common data horizon. The seed is the replication unit; 10,000 paired bootstrap resamples provide within-workload intervals.

## Empirical Validation / Results

### Data Efficiency (Equal-Data Comparison)

Muon lowers validation-loss AUC at equal data on **every workload**:

| Workload | AUC Gain (Muon over AdamW) |
|----------|---------------------------|
| Language-micro | 3.25% |
| Language-small | 4.46% |
| Vision-balanced | 16.98% |
| Vision-long-tail | 22.44% |

All 20 paired trajectories reach a stable 1%-margin crossover. Mean crossover occurs at 4.8% and 3.0% of language horizons; vision crossovers occur at the first 1% evaluation point.

### Time Efficiency (Equal-Time Comparison)

The wall-clock accounting **divides the workloads**:

| Workload | Time-Normalized AUC Gain | Equal-Time Endpoint Loss Gain |
|----------|--------------------------|------------------------------|
| Vision-balanced | +19.68% (Muon) | +39.71% (Muon) |
| Vision-long-tail | +22.31% (Muon) | +41.09% (Muon) |
| Language-micro | −6.96% (AdamW) | −3.56% (AdamW) |
| Language-small | −3.38% (AdamW) | −0.73% (AdamW) |

### Optimizer-Step Cost

**Table 1: Mean systems measurements over five seeds.** The throughput ratio is Muon/AdamW. Optimizer shares cover the full optimizer call, including auxiliary AdamW in the reference Muon implementation.

| Workload | Throughput ratio | AdamW opt. share | Muon opt. share |
|----------|-----------------|------------------|-----------------|
| Language-micro | 0.377 | 8.8% | 63.1% |
| Language-small | 0.468 | 9.1% | 56.1% |
| Vision-balanced | 0.339 | 7.4% | 69.6% |
| Vision-long-tail | 0.414 | 6.1% | 63.1% |

Peak allocated memory is 0.6–2.4% lower for Muon, but this metric alone does not identify the cause of the slowdown.

### Frozen Test Evaluation

Muon lowers test loss in **every paired seed**:

| Workload | Mean Test-Loss Reduction | Additional Metrics |
|----------|-------------------------|-------------------|
| Language-micro | 4.77% | Perplexity −23.40% |
| Language-small | 7.01% | Perplexity −32.04% |
| Vision-balanced | 11.52% | Accuracy +6.76 points |
| Vision-long-tail | 26.03% | Accuracy +7.39 points; tail/worst-group +2.08 points each |

Across the four equally weighted workloads, mean relative test-loss reduction is **12.33%** (stratified bootstrap 95% interval [11.84, 12.83]%).

### Tuning-Budget Regret

Expected relative validation-AUC regret with restricted access to frozen candidate pools:

| Candidates | AdamW | Muon | Variance-Muon |
|------------|-------|------|---------------|
| 1 | 7.52% | 13.66% | 29.46% |
| 5 | 0.42% | 1.69% | 7.70% |
| 10 | 0.09% | 0.67% | 1.73% |
| 23 | 0.00% | 0.00% | 0.00% |

### Variants and Approximations

- **Late switch**: Switching from AdamW to Muon after 25% of training reduces test loss by 3.97–23.40% relative to AdamW, but full-horizon Muon achieves lower loss on all workloads.
- **Variance-Muon** (exploratory, 3 seeds): improves language models and balanced vision but increases long-tailed test loss by 11.71%.
- **Fewer Newton–Schulz steps** (3 vs. 5): saves 21.87% training time with 0.74% AUC penalty.
- **Restricted matrix routing**: saves 28–45% with penalties of 1.13–2.76%.

## Theoretical and Practical Implications

### Break-Even Analysis

Let $q_o$ denote measured throughput for optimizer $o$. At time $T$, its trajectory has processed approximately $q_o T$ data units, so a time-budget comparison couples the loss curve to the implementation that produces it. Here $q_{Muon}/q_{AdamW}$ ranges from 0.339 to 0.468.

The key insight: **The larger vision gains overcome the throughput gap; the smaller language gains do not.** Faster orthogonalization would raise $q_{Muon}$ and move the break-even point even if the loss trajectory per token is unchanged.

### Tuning Sensitivity

Within the frozen candidate pools, restricting the tuning budget increases Muon's expected regret more than AdamW's. This suggests Muon may have **greater tuning sensitivity**—a practical consideration for users with limited compute for hyperparameter search.

### Scale and Implementation Caveats

The authors are explicit about scope limitations:
- Language models use widths 256/384 with 4,096 tokens per optimizer update on one A100
- The relative cost of orthogonalization depends on matrix dimensions, token batch, and kernel implementation
- Auxiliary AdamW covers most language-model parameters, limiting comparisons with architectures dominated by Muon-eligible matrices
- Large-scale studies report favorable Muon compute efficiency (Liu et al., 2025; Khona et al., 2026); in an 8B model with tensor parallelism, Khaled et al. (2025) report a much smaller throughput penalty

### Recommendations for Credible Optimizer Comparisons

The paper argues that credible comparisons should:
1. Pair initializations across optimizers
2. Equalize tuning opportunities
3. Report both data- and time-normalized curves
4. Attribute optimizer cost separately
5. Freeze test evaluation before access

## Conclusion

The paper demonstrates that **"who wins" depends critically on the budget being measured**:

- **Under a data budget**: Muon consistently wins across all four workloads, with gains ranging from 3.25% to 22.44% in validation AUC and up to 26.03% test-loss reduction.
- **Under a wall-clock budget**: The reference Muon implementation's high per-step cost (56.1–69.6% of training time) reverses the ranking on language models, where gains are modest, but not on vision, where gains are large enough to overcome the throughput penalty.
- **Under a tuning budget**: Muon exhibits higher expected regret (13.66% vs. 7.52% with one candidate), suggesting greater sensitivity to hyperparameter selection.

### Future Directions

The authors identify several open questions:
- Selection stability across multiple development seeds
- Additional architectures, imbalance constructions, and longer training horizons
- Optimized kernels and multi-accelerator implementations
- The role of adaptive step scaling in reducing tuning sensitivity (Demidovich et al., 2026)

The paper's central methodological contribution is its **paired, budget-aware benchmark framework**—pairing initializations, equalizing tuning opportunities, reporting data- and time-normalized curves, attributing optimizer cost, and freezing test evaluation—which provides a template for future optimizer comparisons beyond the specific AdamW-vs-Muon question.

---

_Markdown view of https://picx.dev/p/X9aH6g, served by PicX — AI-generated visual whiteboard summaries of research papers._