Issue 3: Mechanistic Localization of Low-Precision Numerical Channels and Muon Spectral Scaling Laws
Highlights of This Issue
This week's low-precision training leads show a notable convergence: two independent works recharacterize "loss spikes / training collapse" from optimization dynamics to numerical artifacts, each localizing a single intervenable failure site. One QK Channel, Many Sources proves that bf16 transformer collapse follows a "source-channel separation": different failure sources such as streaming-softmax accumulators and weight gradient quantization all converge to the same QK spectral runaway channel, with the entry determined by temporal sign consistency of per-head errors (rather than aggregate magnitude); parameter-free QK-Guard triggers blocking at the QK site, curbing all tested runaways, matching always-on QK-norm within 60k steps, and reproducing on a second GPU architecture. Complementarily, Grokking or Glitching? proves that Slingshot spikes are not intrinsic optimization dynamics but numerical artifacts of finite-precision cross-entropy computation: correct-class gradients are rounded to zero, breaking the gradient zero-sum constraint, forming a positive feedback between classifier mean and feature mean (Numerical Feature Inflation), driving exponential norm growth. Together, they point to a judgment: stability issues under low precision should first be localized at the "numerical channel" level before considering optimizer or parameterization fixes.
The Muon line this week moves from "engineering variant" to "scaling law axis." Spectral Scaling Laws of Muon provides the first systematic study of the singular value spectrum of Muon's momentum buffer: on 77M–2.8B, per-layer quantiles stabilize after burn-in, with stable values following layer-dependent power laws in model scale—mid-shallow layers about M^-0.25, final-layer projections up to M^-0.96, the latter falling into the Newton–Schulz failure zone at 300B scale, yielding a layer-aware NS configuration recipe. This directly connects to the "optimizer-induced spectral scaling differences" question raised in the previous issue, turning orthogonalization cost into a predictable scaling law quantity. MONA injects curvature-aware acceleration terms into Muon's momentum buffer before orthogonalization, validated across scales on 1B–68B MoE. On the hyperparameter transfer side, Let's Scale Step by Step extends muP-for-MoE from width/expert-count transfer to the token dimension: using EMA weights to extract optimal learning rates at multiple token budget points from a single short-budget run, fitting a log-log linear law extrapolated to 10T tokens (R²=0.95), validated on 155B/17B.
On the pretraining objective side, LoopMTP extends multi-token prediction to the latent space of recurrent transformers, using soft cosine alignment to guide each recurrent hidden state, avoiding vocabulary-sized projection head overhead. The data efficiency debate between masked diffusion and autoregression gains a large-scale compute-matched data point: Parallelism and Generation Order in Masked Diffusion Language Models systematically evaluates 8 MDLMs (up to 100B) on 58 benchmarks, quantifying dependency loss from parallel factorization and providing a theoretical path for generate-then-edit.
Community and Updates
- The Safety Mechanism Caused the Divergence is a mechanism-level incident postmortem: the root cause of FP8 MoE training loss divergence was not the FP8 format itself, but a fixed activation-row clipping cap—when legitimate activation norms grow far beyond the cap, clipping shifts from "circuit breaker" to "continuous transformation," with three control groups pulling final loss from 11.68 back to 3.51. An important warning for anyone running low-precision MoE at scale.
- Conformalized Scaling Laws applies conformal prediction to scaling law extrapolation, proving that OLS confidence intervals systematically undercover in extrapolated compute domains (in Pythia-scale simulations, 95% intervals achieve only 61% joint coverage); CSL uses log-scale residuals as nonconformity scores to provide distribution-free finite-sample coverage guarantees.
- Set Diffusion proposes continuous interpolation of token ordering between autoregressive and diffusion, rather than fixed block lengths, controlling parallelism and ordering bias via position-offset reveal-time scheduling, outperforming pure masked diffusion baselines on compute/speed trade-offs.
- mssp-moe provides a codebase for MSSP parameterization, treating expert count and expert width as scaling axes, serving as a reproducible landing point for the muP-for-MoE line.
Open Questions
- The QK spectral runaway in One QK Channel and the Numerical Feature Inflation in Grokking or Glitching? each localize a single failure site—do these two "numerical channels" unify under a more general spectral geometric coordinate? Can the same singular value/eigenvalue diagnostics cover both attention collapse and classifier drift?
- The layer-dependent power laws in Spectral Scaling Laws of Muon are fitted on 77M–2.8B, with final-layer exponents up to -0.96; if this exponent holds at larger scales, will layer-aware NS configurations couple with the token-dimension learning rate scaling law in Let's Scale Step by Step, i.e., do different NS configurations change the extrapolation of optimal learning rates?
- The latent-space soft alignment in LoopMTP avoids vocabulary-sized projection heads, but is this merely a cheap approximation of standard MTP? A compute-matched comparison with standard MTP is needed to judge whether latent-space objectives bring essential gains.
Papers in this issue
Low-precision transformer collapse converges on a single query-key channel regardless of error source, and per-head QK normalization at that locus contains the runaway.
Editor's noteReframes low-precision training collapse from "source-by-source fixes" to "single-channel blocking": proves that different bf16 failure sources such as streaming-softmax accumulators and weight gradient quantization all converge to the same QK spectral runaway channel, with the entry determined by temporal sign consistency of per-head errors rather than aggregate magnitude. QK-Guard uses parameter-free QK normalization for triggered blocking at the QK site, matching always-on QK-norm within 60k steps, and reproduces across architectures and a second GPU. A mechanism-level rather than patch-level contribution to low-precision training stability.
A two-step framework combining µP-based width transfer and a token-scaling law predicts optimal learning rates for MoE pretraining, enabling a stable 155B-parameter, 10T-token training run.
Editor's noteOn the muP-for-MoE line (contrast with MSSP and Complete-muE in Issue 1, which only handle width/expert-count transfer), this work is the first to include the token dimension in the same framework: first adapt μP to MoE (including MLA and Muon) to validate width transfer, then use EMA weights to extract optimal learning rates at multiple token budget points from a single short-budget run, fit a log-log linear law extrapolated to 10T tokens (R²=0.95), and validate on 155B/17B. The increment lies in the EMA trick and the token-dimension learning rate scaling law, but the extrapolation span from 255B–500B to 10T is large; readers should assess the risk themselves.
Slingshot loss spikes in neural network training are a floating-point arithmetic artifact, caused by softmax collapse triggering exponential weight and feature drift via Numerical Feature Inflation.
Editor's noteProvides a mechanistic proof of Slingshot loss spikes: they are not intrinsic optimization dynamics but numerical artifacts of finite-precision cross-entropy computation. When the difference between the correct-class logit and the rest exceeds the absorption error threshold, the correct-class gradient is rounded to zero, breaking the gradient zero-sum constraint, forming a positive feedback between classifier mean and feature mean (Numerical Feature Inflation), driving exponential norm growth and spikes. Recharacterizes loss spikes from an "optimization problem" to a "numerical problem" and offers a testable intervention (restoring the zero-sum constraint).
Muon momentum singular value quantiles stabilize during training and follow layer-dependent power laws in model size, making uniform Newton-Schulz configurations suboptimal at scale.
Editor's noteThe first systematic scaling study of the singular value spectrum of Muon's momentum buffer: on 77M–2.8B, per-layer quantiles stabilize after burn-in, with stable values following layer-dependent power laws in model scale (mid-shallow layers about M^-0.25, final-layer projections up to M^-0.96). Provides a layer-aware Newton–Schulz configuration recipe—the final layer falls into the NS failure zone at 300B scale, requiring more iterations or better coefficients. Turns Muon's orthogonalization cost from a "fixed detail" into a predictable scaling law axis, directly connecting to the previous issue's discussion of optimizer spectral scaling differences.
MONA adds curvature-aware acceleration to Muon's orthogonalization, achieving consistent gains over Muon and AdamW across 1B-68B parameter MoE models with proven convergence guarantees.
Editor's noteInjects ALTO's curvature-aware acceleration term (EMA of gradient differences) into Muon's momentum buffer before orthogonalization, achieving sharp minima escape while preserving spectral norm regularization, with non-convex convergence analysis and Hessian eigenvalue selectivity proofs. Validated across scales on 1B–68B MoE, achieving SOTA after SFT on 68B. Compared to Dion3 (orthogonalization efficiency) and SOAP/Muon stability fixes in Issue 1, the increment lies in combining curvature-aware acceleration with geometric structure; however, the acceleration term is essentially a port of ALTO, and learning rate transfer or extrapolation residuals are not reported.
LOOPMTP combines looped transformers with soft multi-token prediction supervision, achieving up to 8.1% relative gains over parameter-matched baselines and outperforming LoopFormer in 27 of 28 comparisons while enabling stable training and small domain-specific experts.
Editor's noteExtends multi-token prediction to the latent space of recurrent transformers: using soft cosine alignment to guide the t-th recurrent hidden state toward the embedding of the token t steps ahead, with lightweight gated aggregation across recurrent steps, avoiding vocabulary-sized projection head overhead. Stable training on 3–9 recurrent steps with up to 8.1% improvement over non-recurrent baselines. As a latent-space variant of MTP successors, it is worth reading, but compute-matched comparisons are only against non-recurrent baselines and LoopFormer, not aligned with standard MTP; readers should assess independently.
