Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf

Summary (Overview)

• The paper presents results from the updated AlgoPerf v1 benchmark, evaluating 13 training algorithms on wall-clock time-to-target across 9 deep learning workloads on fixed hardware (4× A100 GPUs, upgraded from 8× V100).
• Schedule-Free AdamW v2 (PyTorch) ranked first on the self-tuning leaderboard, followed by AdEMAMix (AdamW equiv.) (PyTorch) — a submission that effectively degenerates into a well-tuned AdamW baseline.
• The benchmark introduces a new decoder-only language modeling workload (FineWeb-Edu 10B) and recalibrated runtime budgets compared to AlgoPerf v0.5.
• Step-efficiency vs. wall-clock analysis reveals that matrix-preconditioned methods (Muon) and two-loop methods (DiLoCo) achieve strong per-step progress but require systems optimizations like parameter sharding to be competitive in runtime.
• Hyperparameter saliency analysis using 1D Wasserstein distances shows that learning rate and momentum are the most sensitive hyperparameters across domains, with top-performing learning rate intervals often disjoint across workloads.

---

Introduction and Theoretical Foundation

Motivation

Neural network training relies on fragile empirical recipes governed by brittle hyperparameter tuning procedures and massive computational budgets. The literature on new optimizers suffers from confounding factors:

• Weak baselines
• Optimizer-model co-design
• Limited testing across different tasks
• Obscured tuning recipes

Benchmark Design

The AlgoPerf: Training Algorithms benchmark [Dahl et al., 2023] was designed to eliminate these confounding factors and measure practical speed-ups in neural network training achieved solely by improving the underlying training algorithms. Key features:

• Two rulesets: external-tuning (workload-agnostic hyperparameter search spaces) and self-tuning (no hyperparameter sampling)
• Fixed hardware and time-to-result scoring across multiple workloads
• Fixed architectures — submissions may tune batch size per workload but cannot modify model architecture

Prior Results

The original AlgoPerf competition [Kasimbeg et al., 2025] ranked:

• Distributed Shampoo first on the external-tuning leaderboard
• Schedule-Free AdamW [Defazio et al., 2024] first on the self-tuning leaderboard

Related Work

The paper situates itself against several recent benchmarks:

• modded nanogpt challenge [Jordan et al., 2024a]: allows simultaneous optimizer and architecture modifications (coupling effects)
• Semenov et al. [2025] and Wen et al. [2026]: benchmark optimizers for LLMs under fixed architectures, finding that optimizer rankings shift with training horizon (Muon excels on short runs; SOAP and Kron overtake on longer horizons)
• Schaipp [2025]: matrix-preconditioned methods achieve substantially lower final losses than AdamW for diffusion models
• Zhang et al. [2026]: most closely related — benchmarks 23 optimizers across vision and language, but lacks AlgoPerf's stringent cross-domain generalization test

---

Methodology

Benchmark Modifications (AlgoPerf v1 vs. v0.5)

1. Added a decoder-only NanoDO-style language model workload with FineWeb-Edu 10B dataset
2. Upgraded hardware from 8× V100 (16 GB) to 4× A100 (40 GB) NVIDIA GPUs
3. Adjusted runtime budgets based on v0.5 competition results and A100 hardware calibration
4. Migrated JAX workloads from pmap sharding to jit sharding
5. Extended submission API for batch normalization statistic updates
6. Extended submission API for evaluation parameter specification (useful for EMA methods)

Workloads

The benchmark evaluates algorithms across nine base workloads:

Self-tuning runtime budgets are 1.5× these base values.

Submission API

Training algorithm submissions are limited to five submission functions:

1. update_params — modifies network parameters during training (optimization algorithms)
2. init_optimizer_state — initializes internal state (e.g., learning rate schedules)
3. data_selection — controls batch construction (curriculum learning, data echoing)
4. get_batch_size — defines batch size per workload (only workload-dependent parameter permitted)
5. prepare_for_eval — prepares for evaluation (e.g., switching to EMA parameters)

This API isolates training speed-ups from pipeline optimizations or model changes.

Self-Tuning Track

• Submissions provide a single, fixed hyperparameter configuration executed across all 9 workloads
• Runtime budget is 1.5× the external-tuning track budget
• Submissions may adjust hyperparameters during training based on heuristics
• Rewards out-of-the-box generalizability — critical for practical adoption where hyperparameter sweeps are expensive

Scoring

A submission's score is based on performance profiles — plots of the fraction of workloads where a submission trains successfully within a factor τ of the per-workload fastest submission:

$$\rho_s(\tau) = \frac{1}{W} \sum_{w=1}^{W} \mathbb{1}[\text{submission } s \text{ solves workload } w \text{ within } \tau \times t_{\min,w}]$$

The final scalar benchmark score is the normalized area under the performance profile, with 1.0 corresponding to a submission that trained fastest on all workloads.

---

Empirical Validation / Results

Leaderboard Results

Key findings:

• Top-3 submissions were PyTorch implementations
• Schedule-Free AdamW v2 (PyTorch) achieved fast (< 1.25τ) time-to-target on 5 of 9 workloads, reaching the target on 6 workloads total
• PyTorch vs. JAX implementation differences led to different workload success sets

Wall-Clock vs. Step Efficiency

Re-scoring submissions by replacing wall-clock time-to-target with median number of optimizer steps reveals:

• Step-efficient but wall-clock expensive algorithms (DiLoCo, Muon JAX) receive better scores under step-based scoring — they take better steps but each step costs more
• Muon (PyTorch) benefits from more optimized implementations (sharding, bfloat16) to achieve better wall-clock performance

Relaxed Leaderboards

When validation targets are relaxed by percentage r (making targets easier):

• Muon (PyTorch) climbs from 6th place to 1st place at 12% relaxation
• Schedule-Free AdamW v2 (PyTorch) performs consistently well across all relaxation levels
• AlgoPerf scores are more stable than leaderboard rankings with respect to target relaxation

Hyperparameter Saliency Analysis

The paper defines a normalized Importance Index using 1D Wasserstein distance:

$$I(\theta) = \frac{W_1(P_{\text{top-}k}(\theta), P_{\text{all}}(\theta))}{\theta_{\max} - \theta_{\min}}$$

Key observations from Figure 4:

• Learning rate (η) consistently ranks among the top 2 most important hyperparameters across workloads and algorithms
• Momentum (β₁ or β) also shows particular sensitivity
• Top-5% learning rate intervals are often disjoint across workloads — e.g., for AdEMAMix, top configurations for fastMRI U-Net are nearly disjoint from three workloads and entirely disjoint from two others

Tuning Overhead and Retention Analysis

Using extrapolated time-to-target scoring (power law fit $y(t) = y_\infty \pm A t^{-\gamma}$ capped at $2T_{\max,w}$):

Key findings:

• DiLoCo is most robust under time-based evaluation (1.0% failure rate, highest median speedup)
• Muon achieves the highest per-workload oracle ceiling (1.973) but retains only 47% under a universal configuration
• NAdamW and DiLoCo retain the largest share of oracle performance under a single configuration
• DiLoCo achieves the highest expected score at minimal tuning budgets (k ≤ 5)

Case Study: Schedule-Free AdamW Batch Normalization Fix

The original Schedule-Free AdamW suffered from Batch Normalization covariate shift: running statistics were accumulated at extrapolated weights $y_t$ but evaluation occurred at averaged weights $x_t$. Since $y_t \neq x_t$, this induced severe covariate shift.

Fix in v2:

1. Training forward-backward executes at $y_t$ using strictly in-batch statistics (running buffer updates disabled)
2. Running statistics calibrated independently to $x_t$ via an additional forward pass (gradient tracking disabled) every third training step

This transformed ImageNet ResNet-50 from a failure mode into a stable run.

Case Study: Muon Distributed Sharding

Muon applies spectral orthogonalization to momentum matrices:

$$O_t = M_t(M_t^T M_t)^{-1/2}$$

approximated via quintic Newton-Schulz iterations:

$$X_{k+1} = aX_k + X_k\left(bX_k^T X_k + c(X_k^T X_k)^2\right)$$

Implementation optimizations in Muon (PyTorch):

1. Greedy complexity partitioning — matrices sorted by FLOP complexity and assigned to GPUs round-robin
2. Asynchronous collective communication — internal reduce-scatter for gradients, asynchronous all-gather for updates
3. Reduced-precision compilation — bfloat16 Newton-Schulz iterations fused via torch.compile

Ablation results:

Case Study: AdEMAMix Warmup Collapse

The second-place submission set a 500,000-step warmup for the slow EMA mixing weight α(t). Since all workloads operate on 8k–113k step horizons, α(t) remained near zero throughout training, making the submission effectively equivalent to AdamW.

The canonical AdEMAMix submission (Rank 12) scaled warmup to workload step budgets (0.60 × step_hint), successfully activating the slow EMA on language modeling (0.27× budget vs. 0.38× for baseline) but performing poorly elsewhere — reflecting AdEMAMix's low cross-workload retention (R = 41.5%).

---

Theoretical and Practical Implications

For Optimizer Design

1. Ease-of-tuning matters as much as raw performance: The self-tuning track reveals that algorithms with more hyperparameters (AdEMAMix with D=7) suffer disproportionately in cross-workload generalization.

2. Systems optimizations are crucial: Muon's success depends heavily on sharding and reduced-precision computation — matrix-preconditioned methods require aggressive computational optimization to translate sample efficiency into wall-clock wins.

3. Learning rate remains the universal bottleneck: The saliency analysis shows that learning rate is consistently among the top-2 most sensitive hyperparameters across all workloads and algorithms, and top-performing intervals are often disjoint across domains.

4. Batch normalization calibration is essential for schedule-free methods: The covariate shift between evaluation and training weights must be addressed for EMA-style methods to work with batch-normalized architectures.

For Benchmarking Practice

1. Wall-clock vs. step-based evaluation captures different aspects: Step-based scoring reveals algorithmic efficiency while wall-clock scoring reveals practical utility — both are needed for a complete picture.

2. Relaxed targets provide robustness information: Algorithms that nearly reach targets (like Muon) may be practically useful despite failing strict targets, and rankings can shift substantially under relaxation.

3. Retention analysis quantifies cross-workload transfer: The retention rate R provides a principled measure of how much performance is lost when moving from per-workload tuning to a universal configuration.

---

Conclusion

The paper presents the updated AlgoPerf v1 self-tuning leaderboard with 13 training algorithm submissions across PyTorch and JAX. Key takeaways:

1. Schedule-Free AdamW v2 (PyTorch) retains its crown from AlgoPerf v0.5, demonstrating that schedule-free iterate averaging generalizes well across heterogeneous architectures.

2. Step-efficient methods need systems optimization: Matrix-preconditioned (Muon) and two-loop (DiLoCo) methods make strong per-step progress but require parameter sharding and other optimizations to be competitive in wall-clock time.

3. Hyperparameter sensitivity is dominated by learning rate and momentum, with top-performing intervals often disjoint across workloads — highlighting the need for more robust, self-tuning methods.

4. Tuning overhead varies substantially across algorithms: DiLoCo and NAdamW retain 61–66% of oracle performance under a universal configuration, while Muon (47%) and AdEMAMix (41.5%) suffer more from cross-workload tuning constraints.

Future directions: The results show considerable room for self-tuning optimizers to improve — hyperparameter tuning (or the lack thereof) remains responsible for a large share of an optimizer's practical performance. The authors emphasize the need for methods that are more robust against hyperparameter tuning and can generalize well to new tasks.