Effective Learning Rate Becomes a Unified Coordinate, Muon Spectral Allocation Moves Toward Measurability

Highlights of This Issue

The most notable progress this issue is elevating the "effective learning rate" from a heuristic to a verifiable macroscopic coordinate. Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining proposes ELR collapse: when the ratio of learning rate to parameter norm η/‖W‖_F is matched pointwise, even if the LR and norm trajectories differ significantly, the loss trajectories collapse to within a median error of 2.5×10⁻³ (all 26 comparisons below 5×10⁻³). More importantly, it proves that the impact of weight decay and Hyperball on loss is mainly realized through the ELR trajectory they induce, and enables the fitted scaling law (FSL) to transfer across norm control methods—when parameterized with ELR, the error is reduced by 12.9 times. This directly connects the explicit norm control of Hyperball from Issue 2 with the nonlinearity of learning rate, unifying "LR scheduling" and "norm control" as two implementation means of the same dynamical coordinate.

The Muon line moves from "spectral scaling law" to "per-rank measurability." Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon uses out-of-sample spectral probing to measure the loss-optimal step size in each singular direction of the momentum buffer, obtaining a stable anisotropic spectrum where "the head is at Edge-of-Stability, the body can tolerate larger steps," which uniformly explains why Muon outperforms Adam, which outperforms SGD, and proposes SAMuon/SAMuon-lite to amplify the body and anchor the head, saving 13.3–24% tokens relative to Muon at 124M–1B. Compared to the static spectral scaling law of Spectral Scaling Laws of Muon from Issue 3, this is the first per-rank out-of-sample measurement and static spectral prior, turning Muon improvements from "empirical recipes" into "measurement-driven design."

On the low-precision training side, M+Adam addresses the complementary failure modes where additive updates are rounded to zero for large weights and multiplicative updates fail near zero, combining Adam-style additive and Madam-style multiplicative updates into a unified optimizer, validated at 60M–1B across BF16/FP8/FP4 precisions and scales, with the largest gains under FP4; Weight-norm Criticality provides a mechanism for loss spikes complementary to the numerical artifact explanation from Issue 3: weight decay continuously compresses the norm of scale-invariant components, causing Hessian eigenvalues to amplify by α⁻² and cross the stability boundary, offering a layer-localizable criterion. Scaling law estimation itself is also being "proactivized": Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search applies the acquisition function directly to the uncertainty of power-law coefficients, an active design route relative to the passive experimental design theory from Issue 1.

Community and Updates

• SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation provides a controlled compute/scale-matched data point for the data efficiency debate between masked diffusion and autoregression: at 2B scale, AR is significantly more compute-efficient than masked diffusion, and AR is a better backbone for blockwise diffusion mixtures; it also provides a recipe to convert 1.7B–30B AR models to block diffusion models at minimal cost, and reports 5× theoretical and 2.3× wall-clock parallel decoding speedups.
• OmniOpt offers a controlled benchmark of 24+ optimizers at 60M–1B across multiple scales on C4 (and transfers to FineWeb-Edu and four architectures), decomposing optimizer updates into a five-stage meta-pipeline and four geometric axes, classified by both mechanism family and effect target—a reproducible landing point for those wanting to compare frontier optimizers horizontally.
• When Activation Checkpointing Calls a Stateful Quantizer Twice identifies a previously undocumented engineering-level bug: activation checkpointing's double call to a stateful delayed-scaling quantizer advances the quantization state twice, producing intermittent gradient spikes that masquerade as training instability. This has direct diagnostic value for practitioners debugging loss spikes on FP8/FP4.

Open Questions

1. The ablation in ELR collapse shows that QK-Norm and learnable RMSNorm gains actually improve collapse accuracy, while fixed gains (more scale-invariant) break collapse. Static scale symmetry cannot explain this counterintuitive phenomenon—the microscopic dynamical origin of ELR collapse remains open; does it share the same spectral geometric root as the QK spectral runaway channel from Issue 3?
2. SAMuon uses a static, head-anchored spectral prior, while SOAP learns the spectrum online. Can the static prior transfer across model scales and architectures, or does each new configuration require re-measuring the spectral profile? How does this connect with the layer-dependent power law of the Muon spectral scaling law from Issue 3?
3. OpenEuroLLM finds that WSD annealing shifts the optimal learning rate up by up to 4×. If ELR is the unified coordinate, does annealing shift the optimal LR or the optimal ELR? Can both be normalized under the ELR coordinate into a schedule-independent quantity?
4. Does the additive-multiplicative combination in M+Adam interact with the QK channel blocking and numerical feature inflation mechanisms from Issue 3—do the scale-aware updates of the multiplicative branch under low precision change the trigger conditions for QK spectral runaway?