Optimizer and Weight Decay Calibration on the Overtraining Axis
Highlights of This Issue
The most prominent theme of this issue is turning "training duration" from a fixed background in scaling laws into a core axis for optimizer evaluation. Optimizer Memory Schedules for Outscaling the Overtraining Axis is the first systematic study of optimizer relative performance and optimal hyperparameter changes under overtraining factors OT 1×–256×: ADANA's equivalent OT exponent fits to 1.15–1.20, close to DANA's theoretical prediction 2−κ=1.15, while Muon/SOAP provide an approximately constant token multiplier advantage, and actionable rules such as weight decay coefficient ∝√OT and momentum cooling are given. This extends the static spectral analysis of Spectral Allocation in Issue 4 to the dynamic dimension of "training duration," directly addressing the scaling of hyperparameters with training budget.
Weight decay itself is also being recalibrated. Scale Weight Decay and Train Better, inspired by the Robbins–Monro condition, scales weight decay by the learning rate fraction η_t/η_max, changing the shrinkage term from O(η_t) to O(η_t²), preserving the asymptotic stationarity guarantees of unregularized SGD and Muon while avoiding the bias of constant weight decay, and reports up to 30% step savings on 72M–930M MoE. In a related vein, When Does Scale-Invariant Optimization Become Unstable? elevates the effective learning rate from trajectory-level validation to a precisely computable discrete-time law, compressing the interaction between learning rate schedules and weight decay into a single scalar B_t, and proves that under constant learning rate plus weight decay, the equilibrium point is an unstable spiral source.
Stability fixes for Muon-family optimizers continue to advance. Musec replaces Muon's spectral flattening with spectral clipping (clipping only singular values above a threshold) and feeds the clipped momentum back into the EMA recursion, providing the first non-convex non-smooth convergence guarantees for spectral-clipped Muon variants; On the Design of Qwen3.8-Next Architecture empirically demonstrates hyperparameter scaling laws under the joint action of Muon and gated residual architecture at 125B MoE scale—optimal batch and learning rate shift upward, batch warmup is no longer necessary, and zero loss spikes are maintained under stress tests at 4× the optimal learning rate.
The extrapolation reliability of hyperparameter scaling laws is also tested from both positive and negative sides. Does Step Law Transfer to Small-Scale Language Models? finds that in the sub-59M regime, the power-law form of Step Law holds but coefficients shift significantly (optimal learning rate overestimated by ~4×, batch exponent nearly doubled), serving as a direct calibration warning; Amortizing Scaling Law Construction Costs formalizes the data collection for scaling law construction as a Bayesian optimization problem, using compute slicing and surrogate fantasization to recover dense grid fits at 10–100× lower cost.
Community and Updates
- Scaling Laws, Carefully systematically reviews how procedural choices such as parameter counting, precision, and loss aggregation cause extrapolation divergence, a valuable meta-analysis for practitioners relying on scaling law extrapolation.
- Dude, Where's My Update? isolates the gap between "intended updates" and "realized updates" in bf16 as an independent axis for optimizer comparison, providing a diagnostic perspective beyond final loss for optimizer degradation under low precision.
- Muon vs AdamW points out that Muon's benefits are coupled with architecture choices and that plain Muon under constant learning rate is more prone to loss spikes, a practical warning against "plug-and-play" Muon; Why Muon Is Replacing AdamW in LLM Training traces Muon's industrial adoption from speedrun records to Kimi K2, GLM-5, and NVIDIA Megatron.
- ICML Poster - Hyperparameter Transfer with Mixture-of-Expert Layers uses DMFT to derive parameterized rules for hyperparameter transfer across scales for MoE layers, directly addressing the known limitation that muP does not guarantee learning rate transfer for sparse expert models.
Open Questions
- Optimizer Memory Schedules finds ADANA's equivalent OT exponent close to DANA's theoretical prediction 2−κ, but the scale is limited to 253M. Can this exponent advantage be maintained at larger scales and longer OT factors, or will it exhibit layer-dependent corrections similar to Muon's spectral scaling law in Issue 4?
- Does the η_t/η_max scaling in Scale Weight Decay and Train Better and the ELR collapse in Issue 4 point to the same coordinate—is scaling weight decay with learning rate equivalent to maintaining a constant shrinkage rate in ELR coordinates? Can the two be unified into a single norm control law?
- Are the EMA-feedback spectral clipping in Musec and the post-hoc clipping in the contemporaneous SPECTRA truly equivalent in convergence guarantees, or are there dynamical differences not covered by convergence analysis? How does this connect with the per-rank optimal step size spectrum in Issue 4's Spectral Allocation?
- Can the phase boundary 1−β2 ∝ (1−β1) in Beyond Quadratic Loss be verified at real frontier scales, or does it only hold for small models? Do momentum timescale mismatch and the norm compression mechanism of Issue 4's Weight-norm Criticality jointly determine the triggering of loss spikes?
Papers in this issue
Optimizer rankings and hyperparameters shift with training horizon, and ADANA's scheduled memory outscales AdamW, matching theory with a 1.15–1.20 exponent.
Editor's noteFirst to make training duration (overtraining factor OT 1×–256×) a core axis for optimizer evaluation: systematically compares AdamW/ADANA/Muon/SOAP across 51M–253M scales for relative performance and optimal hyperparameter changes, finding ADANA's equivalent OT exponent (1.15–1.20) close to DANA's theoretical prediction 2−κ, Muon/SOAP providing an approximately constant token multiplier advantage, and giving actionable rules such as weight decay coefficient ∝√OT and momentum cooling. Compared to Issue 4's Spectral Allocation static spectral analysis, this paper incorporates training duration into optimizer evaluation, providing direct empirical evidence for hyperparameter scaling with duration.
Scaling weight decay by the learning-rate fraction preserves convergence guarantees and speeds up MoE transformer training by up to 29.4%.
Editor's noteInspired by the Robbins–Monro condition, scales weight decay by the learning rate fraction η_t/η_max, changing the shrinkage term from O(η_t) to O(η_t²), preserving the asymptotic stationarity guarantees of unregularized SGD and Muon while avoiding the bias of constant weight decay. Provides convergence proofs, weight norm steady-state analysis, and reports up to 30% step savings on 72M–930M MoE. Compared to Defazio's similar rule for AdamW, the increment lies in Muon extension and multi-scale MoE systematic validation.
Musec replaces Muon's spectral flattening with spectral clipping, achieving the first convergence guarantees for Muon-type optimizers in nonconvex nonsmooth settings with optimal complexity while stabilizing training across learning rates.
Editor's noteReplaces Muon's spectral flattening with spectral clipping (clipping only singular values above a threshold), preserving momentum spectral skewness, and feeds the clipped momentum back into the EMA recursion—mechanistically distinct from the contemporaneous SPECTRA's 'post-hoc clipping, forward carrying unclipped states'. Provides the first non-convex non-smooth convergence guarantees for spectral-clipped Muon variants, validated on 491M–1.63B.
Qwen3.8-Flash-Next matches its 397B predecessor's quality using one-third activated parameters and one-ninth training FLOPs via GDN, QSA, GR, and n-gram embeddings.
Editor's noteEmpirically demonstrates hyperparameter scaling laws under the joint action of Muon and gated residual (GR) architecture at 125B MoE scale: optimal batch and learning rate shift upward, batch warmup is no longer necessary, and zero loss spikes are maintained under stress tests at 4× the optimal learning rate. Compared to Issue 4's OpenEuroLLM study on WSD annealing-induced hyperparameter shifts, this paper validates hyperparameter transfer under architecture-optimizer joint action at larger scale.
An exact discrete-time law governs scale-invariant optimization, revealing an intrinsically unstable balance point that explains recurrent training dynamics and why adaptive optimizers are more expansion-prone.
Editor's noteFirst to provide an exact step-by-step discrete-time law for the effective step size Φ_t in scale-invariant optimization, compressing the interaction between learning rate schedules and weight decay into a single scalar B_t, and proving that under constant learning rate plus weight decay, the equilibrium point is an unstable spiral source. Compared to Issue 4's trajectory-level validation of Effective Learning Rate, this paper provides an exact algebraic identity without linearization or continuous-time approximation, and uses a homogeneous optimizer framework (ν exponent) to unify the explanation of self-quenching intensity differences between SGD/Adam under normalization.
Step Law's power-law form transfers to sub-59M-parameter language models, but its coefficients do not, overestimating optimal learning rate by roughly 4x.
Editor's noteFirst work to empirically test Step Law in the sub-59M parameter regime: the power-law form holds but coefficients shift significantly—optimal learning rate overestimated by ~4×, batch exponent nearly doubled. Directly addresses the extrapolation reliability of hyperparameter scaling laws, with multi-scale points and reported extrapolation residuals.
Bayesian optimization with compute slicing and surrogate fantasization recovers scaling law fits at 10-100x less compute, with extrapolation error under 1.6%.
Editor's noteFormalizes data collection for scaling law construction as a Bayesian optimization problem, introducing compute slicing (progressively expanding compute windows during acquisition) and surrogate fantasization, recovering dense grid fits at 10–100× lower cost. Compared to Issue 4's Power-Law Entropy Search for active experimental design, this is the first to explicitly handle auxiliary hyperparameter spaces (LR, batch) and use compute slicing as a myopic constraint.
Adam's loss spikes in neural networks follow a near-linear beta boundary because confident cross-entropy losses form a superquadratic core-wall landscape, unlike the cubic scaling of quadratic losses.
Editor's noteFirst to systematically map Adam's stability phase diagram on the (β1, β2) plane, finding a near-linear boundary 1−β2 ∝ (1−β1) separating spiky and non-spiky dynamics, and linking the boundary coefficient to the effective loss exponent n via a one-dimensional super-quadratic loss model. Compared to Issue 4's Weight-norm Criticality and Issue 3's Grokking or Glitching? mechanism explanations, this paper offers a new perspective from momentum timescale mismatch and finite-scale non-quadratic loss geometry.
Uniform and Gaussian diffusion provably outperform masked diffusion by a sqrt(d) factor in query complexity, due to wider critical windows rather than commitment issues.
Editor's noteFirst to provide provable separations in query complexity for few-step generation among masked, uniform, and Gaussian diffusion language models: uniform/Gaussian diffusion can be done with O~(√d) queries, but for random empirical measure masked diffusion requires Ω~(d) while uniform/Gaussian only need O~(√d), with the separation arising from a narrower critical window rather than the 'commitment' intuition. Provides a theoretical foundation for the data efficiency debate between masked diffusion and autoregressive models.
