# Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf

> Schedule-Free AdamW v2 wins AlgoPerf v1 self-tuning, but step-efficient methods like Muon need systems optimizations to match wall-clock performance.

- **Source:** [arXiv](https://arxiv.org/abs/2610.13036)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/rmO4VE
- **Whiteboard:** https://picx.dev/p/rmO4VE/image

## Summary

# Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf

## Summary (Overview)

- The paper presents results from the updated **AlgoPerf v1 benchmark**, evaluating 13 training algorithms on wall-clock time-to-target across 9 deep learning workloads on fixed hardware (4× A100 GPUs, upgraded from 8× V100).
- **Schedule-Free AdamW v2 (PyTorch)** ranked first on the self-tuning leaderboard, followed by **AdEMAMix (AdamW equiv.) (PyTorch)** — a submission that effectively degenerates into a well-tuned AdamW baseline.
- The benchmark introduces a new **decoder-only language modeling workload** (FineWeb-Edu 10B) and recalibrated runtime budgets compared to AlgoPerf v0.5.
- **Step-efficiency vs. wall-clock analysis** reveals that matrix-preconditioned methods (Muon) and two-loop methods (DiLoCo) achieve strong per-step progress but require systems optimizations like parameter sharding to be competitive in runtime.
- **Hyperparameter saliency analysis** using 1D Wasserstein distances shows that learning rate and momentum are the most sensitive hyperparameters across domains, with top-performing learning rate intervals often disjoint across workloads.

---

## Introduction and Theoretical Foundation

### Motivation

Neural network training relies on fragile empirical recipes governed by brittle hyperparameter tuning procedures and massive computational budgets. The literature on new optimizers suffers from confounding factors:

- Weak baselines
- Optimizer-model co-design
- Limited testing across different tasks
- Obscured tuning recipes

### Benchmark Design

The **AlgoPerf: Training Algorithms benchmark** [Dahl et al., 2023] was designed to eliminate these confounding factors and measure practical speed-ups in neural network training achieved solely by improving the underlying training algorithms. Key features:

- **Two rulesets**: external-tuning (workload-agnostic hyperparameter search spaces) and self-tuning (no hyperparameter sampling)
- **Fixed hardware** and **time-to-result** scoring across multiple workloads
- **Fixed architectures** — submissions may tune batch size per workload but cannot modify model architecture

### Prior Results

The original AlgoPerf competition [Kasimbeg et al., 2025] ranked:
- **Distributed Shampoo** first on the external-tuning leaderboard
- **Schedule-Free AdamW** [Defazio et al., 2024] first on the self-tuning leaderboard

### Related Work

The paper situates itself against several recent benchmarks:
- **modded nanogpt challenge** [Jordan et al., 2024a]: allows simultaneous optimizer and architecture modifications (coupling effects)
- **Semenov et al. [2025]** and **Wen et al. [2026]**: benchmark optimizers for LLMs under fixed architectures, finding that optimizer rankings shift with training horizon (Muon excels on short runs; SOAP and Kron overtake on longer horizons)
- **Schaipp [2025]**: matrix-preconditioned methods achieve substantially lower final losses than AdamW for diffusion models
- **Zhang et al. [2026]**: most closely related — benchmarks 23 optimizers across vision and language, but lacks AlgoPerf's stringent cross-domain generalization test

---

## Methodology

### Benchmark Modifications (AlgoPerf v1 vs. v0.5)

1. **Added** a decoder-only NanoDO-style language model workload with FineWeb-Edu 10B dataset
2. **Upgraded** hardware from 8× V100 (16 GB) to 4× A100 (40 GB) NVIDIA GPUs
3. **Adjusted** runtime budgets based on v0.5 competition results and A100 hardware calibration
4. **Migrated** JAX workloads from pmap sharding to jit sharding
5. **Extended** submission API for batch normalization statistic updates
6. **Extended** submission API for evaluation parameter specification (useful for EMA methods)

### Workloads

The benchmark evaluates algorithms across **nine base workloads**:

| Task | Dataset | Model | Loss | Metric | Validation Target | Base Runtime Budget (s) |
|------|---------|-------|------|--------|-------------------|------------------------|
| Clickthrough rate prediction | Criteo 1TB | DLRMsmall | CE | CE | 0.123735 | 8915 |
| MRI reconstruction | fastMRI | U-Net | L1 | SSIM | 0.723653 | 2745 |
| Image classification | ImageNet | ResNet-50 | CE | ER | 0.22569 | 49,918 |
| Image classification | ImageNet | ViT | CE | ER | 0.22691 | 64,292 |
| Speech recognition | LibriSpeech | Conformer | CTC | WER | 0.085884 | 43,680 |
| Speech recognition | LibriSpeech | DeepSpeech | CTC | WER | 0.119936 | 36,949 |
| Molecular property prediction | OGBG | GNN | CE | mAP | 0.28098 | 11,303 |
| Language modeling | FineWeb-Edu 10B | Decoder-only Transformer | CE | PPL | 22.2995 | 31,967 |
| Translation | WMT | Transformer | CE | BLEU | 30.8491 | 16,114 |

*Self-tuning runtime budgets are 1.5× these base values.*

### Submission API

Training algorithm submissions are limited to **five submission functions**:

1. **`update_params`** — modifies network parameters during training (optimization algorithms)
2. **`init_optimizer_state`** — initializes internal state (e.g., learning rate schedules)
3. **`data_selection`** — controls batch construction (curriculum learning, data echoing)
4. **`get_batch_size`** — defines batch size per workload (only workload-dependent parameter permitted)
5. **`prepare_for_eval`** — prepares for evaluation (e.g., switching to EMA parameters)

This API isolates training speed-ups from pipeline optimizations or model changes.

### Self-Tuning Track

- Submissions provide a **single, fixed hyperparameter configuration** executed across all 9 workloads
- Runtime budget is **1.5×** the external-tuning track budget
- Submissions may adjust hyperparameters during training based on heuristics
- Rewards **out-of-the-box generalizability** — critical for practical adoption where hyperparameter sweeps are expensive

### Scoring

A submission's score is based on **performance profiles** — plots of the fraction of workloads where a submission trains successfully within a factor τ of the per-workload fastest submission:

$$\rho_s(\tau) = \frac{1}{W} \sum_{w=1}^{W} \mathbb{1}[\text{submission } s \text{ solves workload } w \text{ within } \tau \times t_{\min,w}]$$

The final scalar benchmark score is the **normalized area under the performance profile**, with 1.0 corresponding to a submission that trained fastest on all workloads.

---

## Empirical Validation / Results

### Leaderboard Results

| Rank | Submission | Score |
|------|-----------|-------|
| 1 | Schedule-Free AdamW v2 (PyTorch) | 0.5679 |
| 2 | AdEMAMix (AdamW-equiv.) (PyTorch) | 0.5191 |
| 3 | Schedule-Free AdamW (PyTorch) | 0.4669 |
| 4 | NAdamW (Baseline v0.5) (JAX) | 0.4506 |
| 5 | Schedule-Free AdamW v2 (JAX) | 0.4405 |
| 6 | Muon (PyTorch) | 0.4231 |
| 7 | Schedule-Free AdamW (JAX) | 0.4108 |
| 8 | NAdamW (JAX) | 0.3844 |
| 9 | Cautious NAdamW (JAX) | 0.3143 |
| 10 | Muon (JAX) | 0.2845 |
| 11 | Lion (PyTorch) | 0.2783 |
| 12 | AdEMAMix (PyTorch) | 0.2464 |
| 13 | Single-worker DiLoCo (JAX) | 0.1302 |

**Key findings:**
- Top-3 submissions were PyTorch implementations
- Schedule-Free AdamW v2 (PyTorch) achieved fast (< 1.25τ) time-to-target on 5 of 9 workloads, reaching the target on 6 workloads total
- PyTorch vs. JAX implementation differences led to different workload success sets

### Wall-Clock vs. Step Efficiency

Re-scoring submissions by replacing wall-clock time-to-target with median **number of optimizer steps** reveals:

- **Step-efficient but wall-clock expensive** algorithms (DiLoCo, Muon JAX) receive better scores under step-based scoring — they take better steps but each step costs more
- **Muon (PyTorch)** benefits from more optimized implementations (sharding, bfloat16) to achieve better wall-clock performance

### Relaxed Leaderboards

When validation targets are relaxed by percentage r (making targets easier):

- **Muon (PyTorch)** climbs from 6th place to **1st place at 12% relaxation**
- Schedule-Free AdamW v2 (PyTorch) performs consistently well across all relaxation levels
- AlgoPerf scores are more stable than leaderboard rankings with respect to target relaxation

### Hyperparameter Saliency Analysis

The paper defines a **normalized Importance Index** using 1D Wasserstein distance:

$$I(\theta) = \frac{W_1(P_{\text{top-}k}(\theta), P_{\text{all}}(\theta))}{\theta_{\max} - \theta_{\min}}$$

Key observations from Figure 4:
- **Learning rate (η)** consistently ranks among the top 2 most important hyperparameters across workloads and algorithms
- **Momentum (β₁ or β)** also shows particular sensitivity
- Top-5% learning rate intervals are **often disjoint across workloads** — e.g., for AdEMAMix, top configurations for fastMRI U-Net are nearly disjoint from three workloads and entirely disjoint from two others

### Tuning Overhead and Retention Analysis

Using **extrapolated time-to-target scoring** (power law fit $y(t) = y_\infty \pm A t^{-\gamma}$ capped at $2T_{\max,w}$):

| Tuning Metric | DiLoCo (D=4) | NAdamW (D=3) | Muon (D=5) | AdEMAMix (D=7) |
|---------------|--------------|--------------|------------|----------------|
| Golden config score (S̄(c*)) | 0.786 | 0.989 | 0.927 | 0.583 |
| Oracle score (S̄*) | 1.279 | 1.509 | 1.973 | 1.403 |
| **Retention Rate (R)** | **61.4%** | **65.5%** | **47.0%** | **41.5%** |
| Median Speedup (p₅₀) | 0.427 | 0.000 | 0.063 | 0.000 |
| Failure Rate (P(S̄ ≤ 0.20)) | 1.0% | 70.0% | 71.0% | 85.0% |

**Key findings:**
- **DiLoCo** is most robust under time-based evaluation (1.0% failure rate, highest median speedup)
- **Muon** achieves the highest per-workload oracle ceiling (1.973) but retains only 47% under a universal configuration
- **NAdamW** and **DiLoCo** retain the largest share of oracle performance under a single configuration
- **DiLoCo** achieves the highest expected score at minimal tuning budgets (k ≤ 5)

### Case Study: Schedule-Free AdamW Batch Normalization Fix

The original Schedule-Free AdamW suffered from **Batch Normalization covariate shift**: running statistics were accumulated at extrapolated weights $y_t$ but evaluation occurred at averaged weights $x_t$. Since $y_t \neq x_t$, this induced severe covariate shift.

**Fix in v2:**
1. Training forward-backward executes at $y_t$ using strictly in-batch statistics (running buffer updates disabled)
2. Running statistics calibrated independently to $x_t$ via an additional forward pass (gradient tracking disabled) every third training step

This transformed ImageNet ResNet-50 from a failure mode into a stable run.

### Case Study: Muon Distributed Sharding

Muon applies spectral orthogonalization to momentum matrices:

$$O_t = M_t(M_t^T M_t)^{-1/2}$$

approximated via quintic Newton-Schulz iterations:

$$X_{k+1} = aX_k + X_k\left(bX_k^T X_k + c(X_k^T X_k)^2\right)$$

**Implementation optimizations in Muon (PyTorch):**
1. **Greedy complexity partitioning** — matrices sorted by FLOP complexity and assigned to GPUs round-robin
2. **Asynchronous collective communication** — internal reduce-scatter for gradients, asynchronous all-gather for updates
3. **Reduced-precision compilation** — bfloat16 Newton-Schulz iterations fused via torch.compile

**Ablation results:**

| Framework | Implementation | HPs | Score | Δ |
|-----------|---------------|-----|-------|---|
| JAX | vanilla | JAX | 0.2845 | — |
| PyTorch | vanilla | JAX | 0.2969 | +0.012 |
| PyTorch | sharded | JAX | 0.3137 | +0.017 |
| PyTorch | sharded | Torch | 0.4231 | **+0.109** |
| PyTorch | vanilla | Torch | 0.3827 | −0.040 |

### Case Study: AdEMAMix Warmup Collapse

The second-place submission set a **500,000-step warmup** for the slow EMA mixing weight α(t). Since all workloads operate on 8k–113k step horizons, α(t) remained near zero throughout training, making the submission **effectively equivalent to AdamW**.

The canonical AdEMAMix submission (Rank 12) scaled warmup to workload step budgets (0.60 × step_hint), successfully activating the slow EMA on language modeling (0.27× budget vs. 0.38× for baseline) but performing poorly elsewhere — reflecting AdEMAMix's low cross-workload retention (R = 41.5%).

---

## Theoretical and Practical Implications

### For Optimizer Design

1. **Ease-of-tuning matters as much as raw performance**: The self-tuning track reveals that algorithms with more hyperparameters (AdEMAMix with D=7) suffer disproportionately in cross-workload generalization.

2. **Systems optimizations are crucial**: Muon's success depends heavily on sharding and reduced-precision computation — matrix-preconditioned methods require aggressive computational optimization to translate sample efficiency into wall-clock wins.

3. **Learning rate remains the universal bottleneck**: The saliency analysis shows that learning rate is consistently among the top-2 most sensitive hyperparameters across all workloads and algorithms, and top-performing intervals are often disjoint across domains.

4. **Batch normalization calibration is essential for schedule-free methods**: The covariate shift between evaluation and training weights must be addressed for EMA-style methods to work with batch-normalized architectures.

### For Benchmarking Practice

1. **Wall-clock vs. step-based evaluation captures different aspects**: Step-based scoring reveals algorithmic efficiency while wall-clock scoring reveals practical utility — both are needed for a complete picture.

2. **Relaxed targets provide robustness information**: Algorithms that nearly reach targets (like Muon) may be practically useful despite failing strict targets, and rankings can shift substantially under relaxation.

3. **Retention analysis quantifies cross-workload transfer**: The retention rate R provides a principled measure of how much performance is lost when moving from per-workload tuning to a universal configuration.

---

## Conclusion

The paper presents the updated **AlgoPerf v1 self-tuning leaderboard** with 13 training algorithm submissions across PyTorch and JAX. Key takeaways:

1. **Schedule-Free AdamW v2 (PyTorch)** retains its crown from AlgoPerf v0.5, demonstrating that schedule-free iterate averaging generalizes well across heterogeneous architectures.

2. **Step-efficient methods need systems optimization**: Matrix-preconditioned (Muon) and two-loop (DiLoCo) methods make strong per-step progress but require parameter sharding and other optimizations to be competitive in wall-clock time.

3. **Hyperparameter sensitivity is dominated by learning rate and momentum**, with top-performing intervals often disjoint across workloads — highlighting the need for more robust, self-tuning methods.

4. **Tuning overhead varies substantially across algorithms**: DiLoCo and NAdamW retain 61–66% of oracle performance under a universal configuration, while Muon (47%) and AdEMAMix (41.5%) suffer more from cross-workload tuning constraints.

**Future directions**: The results show considerable room for self-tuning optimizers to improve — hyperparameter tuning (or the lack thereof) remains responsible for a large share of an optimizer's practical performance. The authors emphasize the need for methods that are more robust against hyperparameter tuning and can generalize well to new tasks.

---

_Markdown view of https://picx.dev/p/rmO4VE, served by PicX — AI-generated visual whiteboard summaries of research papers._