Issue 1: Extrapolability of Scaling Laws, MoE Parameterization, and Large-Batch Optimizer Stability
Highlights of This Issue
This week's focus is on two main threads: the extrapolability and experimental design of scaling laws themselves, and the parameterization and hyperparameter transfer of sparse expert models (MoE), as well as optimizer stability under large batch sizes.
On the scaling law side, Skaling unifies Chinchilla's additive form with Kaplan's coupled form using a single coupling exponent k, recovering the interaction between N and D, and reducing MAPE by 1.5–3x in both interpolation and extrapolation regimes. This echoes the community's ongoing scrutiny of Chinchilla—a quick check of Chinchilla and related Meta FAIR coverage both point to the weak link of the "additive independence assumption." Meanwhile, research on tokens-per-parameter coverage theoretically demonstrates that fixed TPP grids lead to ill-conditioned scaling law parameter estimation, providing actionable thresholds for experimental design; whether small-scale experiments suffice offers a slightly counterintuitive conclusion: as long as hyperparameters are thoroughly tuned, 4M-parameter models can reveal scaling laws, with hyperparameter sensitivity decreasing with scale.
The MoE side is equally vibrant. MSSP points out that muP does not reliably transfer learning rates for MoE and proposes the "Maximal Scale Stability" principle with scale-specific corrections (this work also appears as an Oral at the ICML HiLD Workshop); Complete-muE provides a two-stage combined transfer framework, advocating "tune dense once, transfer to all." Additionally, compute-optimal is not cluster-optimal reminds us that sparse optimality often only emerges internally under hardware-deliverable FLOPs constraints.
Regarding optimizers and training stability, SOAP/Muon large-scale pre-training research diagnoses the "slingshot" instability of SOAP under large batches and provides a fix; Dion3 compresses Muon's iteration time by 6x through Gram orthogonalization and row subsampling. These piecemeal advances need to be viewed in a broader comparative context—a large-scale benchmark of 24 optimizers shows that AdamW's margin still stands, reminding us that many "renamed" optimizers exist, and only those validated at multiple scales with publicly available training curves count.
Community and Updates
- Scaling Laws That Extrapolate 300x Past the Fit claims to extrapolate scaling laws 300x beyond the fitted regime, a radical test of extrapolation reliability, worth reading alongside Skaling's boundary residual diagnostics this week.
- How to Set the Learning Rate for Large-Scale Pre-training? proposes a scale-aware learning rate schedule to bridge the gap between theoretical scaling laws and engineering practice.
Open Questions
- In Skaling, there is a clear trade-off between coupling exponent k and irreducible loss E (when k<1, E is suppressed or even approaches zero), and data cannot determine both simultaneously. Does this imply that any coupled form faces an inherent "k-E unidentifiability" issue at unsaturated scales?
- MSSP and Complete-muE provide two different MoE parameterization recipes, each claiming verification at multiple scales. Do their predictions agree in Regime II (fine-grained bottleneck)? Can a controlled comparison be made on the same experimental setup?
- SOAP's slingshot instability under large batch is attributed to stale preconditioners, but the relationship between batch size itself and scaling laws (whether batch should grow with model size) remains deferred to future work by most optimizer studies. When will batch scaling laws become a predictable target?
Papers in this issue
Skaling law couples model size and data via one interaction exponent, cutting loss-prediction error 1.5–3x over Chinchilla while requiring roughly 10x less tuning compute.
Editor's noteUnifies Chinchilla's additive form and Kaplan's coupled form with a single coupling exponent k, recovering the interaction between N and D, and reducing MAPE by 1.5–3x in interpolation and extrapolation regimes. Mixed derivative analysis and boundary residual diagnostics directly address the failure of the additive independence assumption. L-shaped sparse sampling reduces fitting computation by ~10x. Recommended reading for anyone fitting scaling laws.
Maximal Update Parametrization fails for Mixture-of-Experts models, so the authors propose Maximal Scale Stability, new scaling rules ensuring stable feature learning across all experts.
Editor's notePoints out that although muP formally satisfies maximal-update conditions, it fails to transfer learning rate reliably on MoE or improve monotonically with scale, and proposes the 'Maximal Scale Stability' (MSSP) principle with an architectural fix in three scale regimes. Each regime has independent DMFT analysis support, marking a key advance in the muP-for-MoE line.
viewaling-law fits from collinear token-per-parameter designs are fundamentally ill-conditioned, so non-collinear training grids are essential for robust extrapolation.
Editor's noteTheoretically proves that fixed tokens-per-parameter grids cause ill-conditioned scaling law parameter estimation, with condition numbers growing inversely with exponent differences, and provides closed-form thresholds for TPP diversity to ensure identifiability. Validated on ~1900 trained models with four scaling law forms, offering actionable prior diagnostics for experimental design.
Scaling laws do exist at 4M parameters, but only emerge with rigorous hyperparameter tuning, which shrinks the loss surface's effective dimensionality as models scale.
Editor's noteSystematically proves that as long as hyperparameters are thoroughly tuned, small 4M-parameter models can exhibit scaling laws, and observes that the effective dimension of the hyperparameter loss landscape decreases with scale (mainly driven by parameter count). Provides a counterexample to the prevailing 'small-scale experiments are unreliable' conclusion and demonstrates a diagnostic methodology comparing pre-norm vs post-norm.
SOAP and Muon optimizers stably scale to 100M-token batches on 72B-parameter models where AdamW fails, with per-step QR updates and KL-Shampoo covariance estimation fixing SOAP's instability.
Editor's noteThe first study to push SOAP/Muon to 72B MoE, 3T token scale, diagnosing SOAP's 'slingshot' instability (caused by stale preconditioners) under large batch sizes, and fixing it with per-step QR + KL covariance estimation. At batch sizes up to 100M tokens, Muon and KL-SOAP remain stable while AdamW degrades; the update-RMS matching protocol is also noteworthy.
Compute-optimal MoE configurations are not cluster-optimal: optimizing for hardware-deliverable FLOPs, not model FLOPs, yields interior sparsity optima and flips loss rankings.
Editor's noteIntroduces the MOSAIC framework, coupling scaling laws with calibration-based performance models to jointly optimize MoE architecture, token budgets, and opt partitioning. Key finding: model-FLOPs-optimal sparsity is boundary-seeking, with internal optimum only appearing under hardware-deliverable FLOPs constraints—highlighting the distinction between 'compute-optimal' and 'cluster-optimal.'
Byte-level modeling's compute penalty is objective-dependent: autoregressive models approach BPE parity at scale, while masked diffusion models suffer a persistent, prohibitive performance gap due to context fragility.
Editor's noteThe first compute-matched scaling study isolating the interaction between byte-level tokenization and masked diffusion objectives, proving that the cost of byte-level modeling is objective-dependent: far worse for MDMs than AR. Provides a mechanistic explanation using 'context fragility' and extrapolates a closed-form budget for performance gaps for each metric. Direct guidance for designing tokenizer-free diffusion models.
Complete-muE enables tuning hyperparameters once on a dense model and transferring them to any sparse MoE variant, achieving up to 5.5x faster convergence without retuning.
Editor's noteUses a two-stage composite transfer (dense FFN↔Dense MoE using active-width muP, Dense MoE↔sparse MoE using expert-side SDE cancellation) to cover active experts, total capacity, granularity, and mixed blocks. Advocates 'tune dense once, transfer to all,' validating across LM and diffusion tasks, providing a complementary route to muP-for-MoE.
Dion3 speeds up Muon optimization up to 6x by orthogonalizing only a fraction of momentum rows, which unexpectedly improves training quality across model scales.
Editor's noteConfronts Muon's cubic-time Newton-Schulz orthogonalization bottleneck, proposing Gram Newton-Schulz iterated on symmetric Gram matrices with row/column subsampling updates and megabatching to compress the optimizer step time by 6x without loss degradation. Validated at 1B–14B scales with open-source code—a practical engineering advance for Muon-family optimizers.
