Issue 2: Optimizers Become a First-Class Axis of Scaling Laws, Batch Scaling Law Gets a Closed-Form Solution
Highlights of This Issue
This week's leads elevate the "optimizer" from a training detail to a first-class axis of scaling laws. Hyperball replaces the implicit norm control of weight decay with explicit constraints on the Frobenius norm of weight matrices and their updates, turning the directional step size into an explicit design quantity. Across multiple scales from 130M to 1.2B, it reduces the drift in learning rate transfer from a baseline of 2–4x to about 1.4x, and boosts Muon's token-equivalent speedup to 20–30%. Complementarily, Optimizer-Induced Spectral Scaling Laws demonstrate that the same architecture achieves different spectral capacity scaling under AdamW versus Muon—AdamW's hard-rank exponent on rare tokens is only about 0.44, while Muon's is near linear (about 1.02), and this difference is not visible in validation loss. Together, these point to a judgment: treating the optimizer as a "fixed detail" of scaling laws systematically underestimates or overestimates extrapolation reliability.
On the functional form side of scaling laws, Practical Scaling Laws add an independent training duration axis T and an explicit overfitting term c·N^γ/D^δ to Chinchilla, wrapped with a saturating baseline, achieving SOTA extrapolation residuals across four domains and five public LLM grids. Under data constraints, its cost-aware allocation shifts toward smaller corpora and more epochs. Unified Neural Scaling Laws generalize BNSL to a multivariate form, explicitly modeling non-monotonic scaling of hyperparameters such as learning rate and initialization standard deviation. Last issue's question "when is batch scaling law predictable" received a direct response this week: Towards Joint Scaling Laws with Optimal Batch Size Schedules derive a closed-form solution for the optimal batch size schedule under any learning rate schedule from convex optimization theory, and prove it is decoupled from the peak learning rate and independent of model size/architecture/optimizer. In cross-modality, Abra provides the first systematic scaling laws for text-to-image diffusion models, covering 10^19–10^22 FLOPs, finding an optimal ratio of about 200 image tokens per parameter—10 times the LLM Chinchilla ratio—and showing that diffusion models are more robust to overtraining.
The muP line also has a counterintuitive increment: Quantifying Hyperparameter Transfer finds through all 16 ablation groups that muP's advantage over SP under AdamW comes almost entirely from the embedding layer learning rate—SP only needs to scale the embedding learning rate to Θ(1) with width to match muP, and provides three quantifiable transfer metrics. This offers a lighter alternative path for the "tune dense once" practice and highlights "low embedding learning rate causing training instability" as a neglected source of instability.
Community and Updates
- Megatron-LM's ChainedOptimizer bug documents a previously undocumented failure mode: global grad-norm clipping pushes per-matrix gradients of Muon's orthogonalization group below the Newton–Schulz eps, causing silent training stagnation—an important warning for anyone running Muon-family optimizers at scale.
- NITP proposes using shallow representations as implicit token targets, improving downstream performance on dense and MoE models from 0.5B to 9B with about 2% FLOP overhead, a new path for implicit token prediction as a pretraining objective.
- Scaling is subtler than it seems casts doubt on the assumption of universal scaling laws for RL reasoning, creating tension with several functional form works this week.
- The ProbAI Scaling Laws 2026 workshop and the LLM μP production transfer checklist point to the theoretical frontier and engineering failure modes of the muP successor line, respectively.
Open Questions
- Hyperball explicitly constrains weight norms, muP adjusts parameterization, and MSSP corrects scale ranges—these three routes all improve learning rate transfer, but do they mechanistically converge to the same "effective step size" quantity? Can the effective learning rate (step size in normalized weight space) proposed in On the Nonlinearity of Learning Rate Scaling serve as a unified coordinate for cross-evaluation?
- The closed-form optimal batch size schedule is independent of model size and optimizer, but this conclusion is based on convex assumptions; in MoE and large-scale non-convex scenarios, does batch scheduling remain decoupled from learning rate? Last issue's question about batch scaling laws is thus partially answered, but sparse and non-convex cases remain uncovered.
- Does the optimizer-induced spectral scaling difference (AdamW and Muon differ by more than 2x in hard-rank) imply that the "compute-optimal" ratio itself is optimizer-dependent? If optimizers change representation geometry, does fitting scaling laws with loss as the sole objective systematically underestimate the impact of such differences on extrapolation?
Papers in this issue
Hyperball replaces weight decay with explicit norm constraints on weight matrices, turning it into a direct angular learning rate controller that delivers 20–30% speedups and more stable hyperparameter transfer across model scales.
Editor's noteReplaces the implicit norm control of weight decay with explicit Frobenius norm constraints, turning the directional step size into an explicit design quantity. Across multiple scales from 130M to 1.2B, it reduces learning rate transfer drift from 2–4x to about 1.4x and boosts Muon's token-equivalent speedup to 20–30%. Instead of adjusting parameterization, it constrains the optimization trajectory, complementing muP/MSSP; worth a close read.
Dynamic batch size schedules derived from a closed-form convex optimization formula improve compute efficiency by 6–15% over static batch sizes, independent of model size or optimizer.
Editor's noteFor the first time, derives a closed-form solution for the optimal batch size schedule under any learning rate schedule from convex optimization theory, and proves it is decoupled from the peak learning rate and independent of model size/architecture/optimizer, forming a joint scaling law. It improves efficiency by 6–15% under fixed compute while maintaining universal dynamics, directly addressing the open problem of batch scaling laws.
This paper introduces a closed-form scaling law that extends Chinchilla to data-constrained, multi-epoch regimes, achieving state-of-the-art extrapolation across vision, scientific, and language domains.
Editor's noteAdds an independent training duration axis T and an explicit overfitting term c·N^γ/D^δ to Chinchilla, wrapped with a saturating baseline, achieving SOTA extrapolation residuals across four domains and five public LLM grids. Compared to last issue's Skaling coupled form, it adds the T axis and overfitting term, providing a more complete functional form for data-limited/single-epoch scenarios.
Text-to-image diffusion transformers reach compute-optimal training at roughly 200 tokens per parameter, ten times higher than the Chinchilla rule for LLMs, and overtraining is surprisingly forgiving.
Editor's noteThe first systematic scaling law study for text-to-image diffusion models, covering 10^19–10^22 FLOPs, finding an optimal ratio of about 200 image tokens per parameter (about 10 times the LLM sampling ratio) and showing that diffusion models are more robust to overtraining. It extends predictability to generation quality, optimal CFG, and training curve shape, serving as an important cross-modal scaling law reference.
A unified functional form for neural scaling laws accurately extrapolates performance across multiple interacting dimensions, capturing non-monotonic effects like overfitting that simpler laws miss.
Editor's noteGeneralizes BNSL to a multivariate unified form, explicitly modeling overfitting and non-monotonic scaling of hyperparameters (learning rate, initialization standard deviation) via additive symmetry, simultaneously extrapolating parameters/data/steps/inference steps/hyperparameters across vision and language tasks, significantly reducing extrapolation RMSLE. Compared to last issue's Skaling bivariate form, it is the first to systematically handle hyperparameter non-monotonicity.
Setting the embedding layer learning rate to Θ(1) instead of Θ(1/n) lets standard parameterization match μP's hyperparameter transfer quality in Transformers, making the full μP prescription unnecessary.
Editor's noteProvides three quantitative transfer metrics and, through systematic ablations, finds that muP's advantage over SP under AdamW comes almost entirely from the embedding layer learning rate—SP only needs to scale the embedding learning rate to match muP. It offers a diagnostic explanation and minimal fix for muP's advantage, an increment validated across multiple scales.
The optimal learning rate for LLM training exhibits upward curvature at scale, breaking log-linear scaling, but effective learning rate with data-axis extrapolation transfers accurately with only ~2% extra compute.
Editor's noteIdentifies and mechanistically explains the convex curvature in LLM learning rate scaling laws, showing that using the effective learning rate (step size in normalized weight space) and extrapolating along the data axis eliminates most nonlinearity, and attributes the curvature to transient dynamics of weight norms under AdamW. It provides mechanistic rather than merely fitted-exponent explanations for the reliability of learning rate axis extrapolation.
