Scaling Laws & Training MethodsIssue 15Nov 14 – 21, 2026

Muon Update Dissected Band-by-Band and Neuron-by-Neuron

Highlights

This issue's most concentrated thread is the band-by-band and neuron-by-neuron dissection of the Muon optimizer's update itself. Equal Steps, Unequal Curvature (https://arxiv.org/abs/2611.24644) is the first to use held-out gradients and Hessian-vector products to perform a joint curvature decomposition of Muon's update across singular value bands and individual neurons: it proves that under a fixed norm, Muon's flat spectrum allocation is not second-order optimal, that the top spectral directions are over-stepped because their curvature decays faster than the signal, and it localizes the neuron imbalance to the fused QKV matrix (CV 0.43, far exceeding MLP's 0.18 and attention output's 0.02). The proposed QKV split intervention yields a −0.025 nats gain at 124M/355M, and outperforms per-head blocking and NorMuon's row normalization. Compared to Issue 9's BulkBoost two-band first-order analysis, this paper provides a complete cross-group curvature and fixed-norm criterion; compared to Issue 11's A Quadratic Lens quadratic theory, it offers controlled attribution driven by measured curvature.

Complementing this, Delayed Effects of Temporary Weight Decay in Muon Pretraining (https://arxiv.org/abs/2611.17876) is the first to systematically characterize the "delayed effect" of temporary optimization interventions: an intervention can be harmful at removal time yet beneficial at the final training budget, with the sign and ranking reversing with start time and evaluation level. The mechanism is that early weight decay leaves a persistent norm deficit, causing the relative step size to remain large after removal; fixed-norm replay and norm reset experiments validate this mechanism. The schedule selected at 162M transfers to 1.4B without retuning. Compared to Issue 13's Tune the Baseline First, which focuses on the tuning gap of scaled weight decay, and Issue 13's MuonGS, which focuses on trajectory-level guarantees of norm control, this paper is the first to treat the evaluation methodology itself (removal time vs. final budget) as the object of study.

On the low-precision training side, Delta-Matching (https://arxiv.org/abs/2609.37852) attributes the failure of FlashAttention's saved-output backward row-correction shortcut in FP8 attention training to stale deltas caused by separate rounding of forward/backward operands, and proves that it violates the zero row-sum invariant of the softmax gradient; it restores this invariant using the FP32 product of unquantized probabilities and FP8 backward operands, achieving native block-scaled FP8 training on all seven attention-core matmuls, matching BF16/FP32 reference loss at 569M/1.67B/5.29B and across GDN/GQA, GDN/MLA, KDA/GQA, and Mamba-2/GQA architectures. Compared to Issue 12's The Role of Quantization in Loss Spikes, which focuses on Adam-induced logit mean drift, this paper focuses on the specific mechanism of forward-backward numerical inconsistency in the attention core. On the resource side, OpenGNS (https://arxiv.org/abs/2611.24331) releases the first open dataset of gradient noise scales across vision, language, and diffusion domains, while LightMTP (https://arxiv.org/abs/2610.06031) changes the target of latent MTP from an external auxiliary model to the model's own intermediate-layer hidden states.

Community and Dynamics

  • A NeurIPS 2026 poster (https://news.neurips.cc/poster/155300) provides a new theoretical foundation for hyperparameter transfer in sparse MoE: using Dynamical Mean Field Theory to characterize MoE limit training dynamics, it proves that μP cannot reliably guarantee learning rate transfer and monotonic improvement with scale in Regime II/III where expert count/sparsity diverge jointly, and proposes Maximally Scale-Stable Parameterization (MSSP)—Regime II requires scaling up expert output initialization variance, Regime III requires shared expert initialization. This is a direct advance on the active line that "μP does not guarantee learning rate transfer for sparse expert models."
  • A preregistered replication study (https://rstation.ai/problems/general_pretraining/papers/3) retracts its earlier claim of a 20× speedup of Muon over AdamW at low precision after a second seed reversed the result, and shows a probe pricing law failing out-of-sample—providing a rare systematic challenge to optimizer/precision claims.
  • A first-hand stability case from 32B pretraining (https://forge.coreweave.com/wandb/marin-community/marin/reports/Marin-32B-Work-In-Progress--VmlldzoxMzM1Mzk1NQ) documents the specific fix of QK-norm warmstart: in the face of persistent loss spikes that grad-norm/update clipping and Muon could not eliminate, warmstarting a Llama-style architecture to a Qwen3-style architecture with QK-Norm made the spikes disappear and the model recovered its trajectory within about 10B tokens. Causal attribution still requires caution, but this is a rare reproducible stability fix at frontier scale.
  • A controlled negative result (https://proceedings.mlr.press/v334/parviz26a.html) isolates architecture and data confounds on low-rank matrix factorization and shows that Muon does not consistently beat AdamW, with the advantage being hyperparameter-sensitive—providing a clean small-scale control for large-scale Muon reports.

Open Questions

  1. Equal Steps, Unequal Curvature's curvature decomposition is based on a local second-order model from a single run, and the scale stops at 355M. Do top-spectral over-stepping and QKV neuron imbalance still hold at frontier scales where fused QKV is standard (e.g., MLA), and can the QKV split gain extrapolate?
  2. The norm deficit mechanism in Delayed Effects is not optimizer-agnostic at 405M (as the authors state). Is this delayed effect specific to Muon, or does AdamW's temporary weight decay also have similar delayed benefits that are masked by norm dynamics?
  3. Delta-Matching restores the zero row-sum invariant before dS quantization, but is the residual 5.3–6.3× row-sum residual after FP8 cast still negligible at longer contexts and larger vocabulary sizes?
  4. OpenGNS (https://arxiv.org/abs/2611.24331) reveals that GNS systematically underestimates critical batch on generative tasks and overestimates on classification tasks. Can a corrected GNS be constructed that accounts for pathologies due to heavy-tailed token distributions, making critical batch predictions consistent across tasks?

Papers in this issue

  1. Temporary weight decay in Muon-trained transformers can improve final loss despite hurting loss at removal, because persistent weight-norm deficits keep relative step sizes larger after the intervention ends.

    Editor's note

    First systematic characterization of the delayed effect of temporary optimization interventions: an intervention can be harmful at removal time yet beneficial at the final training budget, with sign and ranking reversing with start time and evaluation level. The mechanism is that early weight decay leaves a persistent norm deficit, causing the relative step size to remain large after removal; fixed-norm replay and norm reset experiments validate this mechanism; the schedule selected at 162M transfers to 1.4B without retuning. Compared to Issue 13's Tune the Baseline First, which focuses on the tuning gap of scaled weight decay, and Issue 13's MuonGS, which focuses on trajectory-level guarantees of norm control, this paper is the first to treat the evaluation methodology itself (removal time vs. final budget) as the object of study. Limitation: the norm deficit mechanism is not optimizer-agnostic at 405M.

  2. Muon's equal step lengths across singular directions over-step high-curvature top directions, and splitting QKV orthogonalization plus a spectral tilt improves loss by 0.046 nats.

    Editor's note

    First to use held-out gradients and Hessian-vector products to perform a joint curvature decomposition of Muon's update across singular value bands and individual neurons: proves that under a fixed norm, Muon's flat spectrum allocation is not second-order optimal, that top spectral directions are over-stepped because their curvature decays faster than the signal, and localizes the neuron imbalance to the fused QKV matrix. The QKV split intervention yields a −0.025 nats gain at 124M/355M and outperforms per-head blocking and NorMuon's row normalization. Compared to Issue 9's BulkBoost two-band first-order analysis, it provides a complete cross-group curvature and fixed-norm criterion; compared to Issue 11's A Quadratic Lens quadratic theory, it provides controlled attribution driven by measured curvature. Limitations: scale stops at 355M, token budget only 100.7M.

  3. Delta-Matching restores the zero-row-sum invariant in FP8 attention training, matching BF16/FP32 performance by fixing the stale-delta failure that degrades larger models.

    Editor's note

    First to attribute the failure of FlashAttention's saved-output backward row-correction shortcut in FP8 attention training to stale deltas caused by separate rounding of forward/backward operands, and proves that it violates the zero row-sum invariant of the softmax gradient; restores this invariant using the FP32 product of unquantized probabilities and FP8 backward operands, achieving native block-scaled FP8 training on all seven attention-core matmuls. Matches BF16/FP32 reference loss at 569M/1.67B/5.29B and across GDN/GQA, GDN/MLA, KDA/GQA, and Mamba-2/GQA architectures. Compared to Issue 12's The Role of Quantization in Loss Spikes, which focuses on Adam-induced logit mean drift, this paper focuses on the specific mechanism of forward-backward numerical inconsistency in the attention core. Limitations: scale stops at 5.29B, no extrapolation residual reported.

  4. OpenGNS, the first open dataset of gradient noise scale measurements, reveals that GNS underestimates critical batch size for generative models but overestimates it for classification.

    Editor's note

    First open dataset of gradient noise scales (GNS) across vision, language, and diffusion domains: dense batch and learning rate scans on 11M–1B models under muP, totaling 433 training trajectories. Key finding is that GNS is consistent across model scales at equivalent loss (supporting small proxy models for batch selection), and reveals task-dependent GNS-CBS bias—underestimating critical batch on generative tasks and overestimating on classification tasks. Provides a public resource for hyperparameter transfer and batch selection reliability. Limitation: measurements are based on raw gradients, not recording AdamW-preconditioned states.

  5. LightMTP predicts future hidden states with a lightweight head, matching next-token performance while boosting planning, reasoning, and code generation at under 3B parameters.

    Editor's note

    First to change the target of latent MTP from an external auxiliary model to the model's own intermediate-layer hidden states (bootstrap stop-gradient target), proposing single-target (S) and concatenated multi-target (C) variants, achieving planning/inference gains comparable to token-level MTP at 1.4–1.5B scale with ≤1% extra parameters and almost no NTP performance loss, and introducing an anticipation cost metric to quantify the trade-off between future token prediction and NTP. Compared to Issue 13's Soft-MTP gradient decomposition and KL recovery techniques, this paper focuses on the lightweightness of the target source (bootstrap vs. external). Limitation: scale stops at 1.5B, no extrapolation residual reported.

  6. CAST adapts spectral gradient clipping thresholds using optimizer-state histories and output spectral-norm feedback, beating nine baselines in MoE pretraining loss and stability.

    Editor's note

    First to make both the input coordinates and thresholds of spectral clipping history-driven: using row/column means of AdamW's second moment from the previous step to define standardized coordinates, and using the EMA of the spectral norm of the clipped output to adaptively set per-matrix thresholds, achieving the lowest loss with minimal relative gradient spikes on 19B MoE pretraining. Compared to Issue 7's Musec spectral clipping variant, the increment is the introduction of a dual-history mechanism of historical scaling and output feedback, with larger experimental scale (19B MoE). Limitation: only 5000 steps, single seed, no extrapolation residual reported.

  7. Freezing the pretrained context tower during AR-to-diffusion conversion preserves knowledge and yields 11.6x higher HumanEval scores than in-place conversion at matched 1B-token budgets.

    Editor's note

    First to systematically compare in-place and frozen-tower AR-to-diffusion conversion configurations under the same parent model, corpus, supervised token budget, trainable parameter set, and evaluation protocol, finding that in-place conversion exhibits output-length-dependent degradation, while frozen-tower preserves most generation capability at 1B token budget (HumanEval pass@10 71.60 vs 6.19), and proving that freezing removes one gradient term. Compared to Issue 10's Revisiting the 16x Pretraining Compute Gap, which focuses on data efficiency, this paper is the first to provide a matched comparison of conversion architecture choice on generation retention. Limitation: scale stops at 30B MoE and 4B dense.

  8. The learning rule, not the task, determines neural circuit size in grokked networks, enabling deliberate trade-offs between robustness, compressibility, and transfer speed.

    Editor's note

    First to unify three paths to unused neurons (coupled weight decay, Adam's epsilon, and decay of trainable per-neuron gains) into a single per-neuron condition—the size of credit relative to the non-gradient term of the update—and proves that credit equalization interventions can increase circuit concentration from 60% to 99% effective neurons without changing the learned algorithm. Finds that only per-neuron normalization (Nero, DFC) can distribute circuits, while 8+ neuron group normalization maintains concentration, a mechanistic insight for optimizer design. Compared to Issue 12's The Role of Quantization in Loss Spikes, which focuses on numerical precision, this paper is the first to link credit allocation granularity to circuit scale. Limitation: small scale (modular arithmetic, MLP, up to 30M LM).