Scaling Laws & Training MethodsIssue 13Oct 31 – Nov 7, 2026

Muon's win/loss begins to flip with data scale and tuning protocol

Highlights of this issue

The most prominent clue this issue is that Muon's advantage over AdamW is beginning to be proven 'conditional', and the conditions themselves are shifting with scale and protocol. Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap (https://arxiv.org/abs/2611.03633) for the first time extends optimizer comparison from pretraining loss to the learning-forgetting trade-off after fine-tuning: under small data budgets Muon dominates, under large data budgets AdamW overtakes, and the crossover point advances as model size increases. It also proposes margin-to-spread ratio as a new metric for predicting fine-tuning robustness. Tune the Baseline First (https://arxiv.org/abs/2611.02529) offers a warning from the tuning protocol side: under a protocol that jointly tunes the LR·WD product, about 90% of Muon's untuned advantage with scaled weight decay is a tuning gap, but the residual advantage grows with scale (survival rate 10%→21%). Together, the two papers rewrite the conclusion 'Muon is better' from an unconditional proposition to a conditional one—'depends on data budget, model scale, and tuning fairness'—continuing the paired-budget protocol established in Issue 11's When Does Muon Actually Win, but advancing the axes from data/time/tuning budgets to fine-tuning adaptability and tuning gap. MuonGS (https://arxiv.org/abs/2610.37647) complements from the mechanism side: using cumulative effective learning rate C_T to uniformly analyze Muon-family norm control, proving that vanilla Muon's cumulative can only grow logarithmically, while MuonW and fixed-norm guarantees grow linearly, and proposing MuonGS, which uses Gram-Schmidt projection to remove the radial component of updates, achieving Ω(√T) cumulative guarantee without fixed norms.

The second line concerns the pretraining objective itself. Multi-Token Prediction Done Right (Soft-MTP, https://arxiv.org/abs/2610.38331) for the first time precisely decomposes the standard MTP cross-entropy gradient into a KL self-distillation term and a residual term, proving that the residual term has low SNR and worsens with scale, and is anti-aligned with the NTP learning direction. It proposes the KL gradient–recovery trick—using straight-through gradient routing to recover the KL gradient exactly from existing NTP/MTP cross-entropy gradients at zero cost, implementing Soft-MTP with one line of code. Compared to Issue 11's ECHO, which reframed MTP as a causal autoregressive main objective, this paper reveals the hidden self-distillation signal in standard MTP at the gradient decomposition level. Train the Sampler You Run (https://arxiv.org/abs/2611.02666) instead treats the few-step sampler of masked diffusion itself as the fine-tuning target: using the importance-weighted IWAE bound to directly optimize the sampler's mixture likelihood, achieving 16-step performance surpassing the original model's 32-step on 8B, with text8 exact likelihood decomposition proving the gains mainly come from schedule weighting.

On the scaling law theory side, A Scaling Theory of Multi-Epoch Pretraining (https://arxiv.org/abs/2611.03590) for the first time incorporates multi-epoch training and ensembling jointly into scaling law theory: deriving the DMFT for multi-epoch SGD under fixed data repetition, giving optimal stopping time and compute allocation laws (ensemble size grows at least as fast as training time with compute), and validating on FineWeb that ensembling can delay optimal stopping and outperform wider/deeper single models under matched parameter-token budgets—directly responding to data-constrained scaling laws and Issue 12's empirical work treating ensembling as an independent axis.

Community and dynamics

  • The official code for Issue 10's decomposition of the 16× pretraining compute gap between diffusion and autoregressive models has been released (https://github.com/amazon-science/dlm-scaling), breaking the gap into exact likelihood readout and architecture (block/encoder-decoder) components, providing a basis for reproducible verification of compute-matched and data-matched claims.
  • A practitioner-oriented loss spike diagnostic (https://juhoplayground.github.io/posts/llm-pretraining-loss-spike-batch-size-muon/) attributes spikes to a feedback loop between weight decay and the Adam denominator via scale-invariant norm shrinkage, giving a WD threshold (0.001–0.01) and a stability recipe combining Muon+RMS+QK-Clip, and emphasizing that 'Muon has no spikes' is inaccurate—large-scale stability comes from the combination of weight decay, RMS matching, and QK-Clip rather than the optimizer itself. Worth referencing as a source for stability hypotheses, but lacks multi-scale training curves and does not meet the standard for close reading.

Open questions

  1. The crossover point for the Muon-AdamW fine-tuning gap reversal advances with model scale (from 320 TPR at 30M to 10 TPR at 600M). Does this trend continue at larger scales, and can the MSR metric stably predict fine-tuning robustness on real large models, or does it only hold on linear associative memories and four fine-tuning datasets?
  2. Soft-MTP proves that the residual term's SNR worsens with scale. Does this 'worsening with scale' prediction hold quantitatively above 1.2B, and does the one-line code change maintain zero-cost gains at true frontier scales (including MoE and Muon combinations)?
  3. Tune the Baseline First reveals that about 90% of Muon-SW's gains are a tuning gap. Does this 'tuning gap' critique equally apply to other reported optimizer/hyperparameter gains—is there a standard paired joint tuning protocol that could allow optimizer benchmarks to exclude tuning unfairness before reporting?
  4. Mix-FT requires a CE pretraining starting point; starting from undertrained weights loses to continuing CE. Can the mixture likelihood objective be trained from scratch, or does its benefit inherently depend on the two-stage structure of 'first learn the conditional distribution, then align the sampler'?

Papers in this issue

  1. Multi-Token Prediction Done Right (Soft-MTP)

    A one-line change removes the noisy residual from standard multi-token prediction gradients, recovering pure self-distillation signal and improving perplexity, decoding speed, and downstream accuracy across model scales.

    Editor's note

    For the first time precisely decomposes the standard MTP cross-entropy gradient into a KL self-distillation term and a residual term, proving that the residual term has low SNR and worsens with scale, and is anti-aligned with the NTP learning direction; accordingly proposes the KL gradient–recovery trick, using straight-through gradient routing to recover the KL gradient exactly from existing NTP/MTP cross-entropy gradients at zero cost, implementing Soft-MTP with one line of code. Compared to Issue 11's ECHO, which reframed MTP as a causal autoregressive main objective, this paper is the first to reveal the hidden self-distillation signal in standard MTP at the gradient decomposition level and eliminate noise residuals, and demonstrates efficiency and effectiveness advantages over explicit KL distillation. Validated at 167M–1.2B multiple scales, with public code and combination with the Muon optimizer. Limitation: scale stops at 1.2B; extrapolation to larger scales requires self-assessment.

  2. A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling

    Ensembling suppresses late-time overfitting in data-limited pretraining, making ensemble size and training epochs jointly optimal and beating larger single models at matched compute.

    Editor's note

    For the first time incorporates multi-epoch training and ensembling jointly into scaling law theory: deriving the DMFT for multi-epoch SGD under fixed data repetition, giving optimal stopping time and compute allocation laws (ensemble size grows at least as fast as training time with compute), and validating on FineWeb that ensembling can delay optimal stopping and outperform wider/deeper single models under matched parameter-token budgets. Compared to Issue 12's Improving Data-Constrained Scaling, which treats ensemble size as an independent axis in empirical scaling laws, this paper provides theoretical mechanisms (variance reduction with E^-1, cross-correlation saturation) and stopping/allocation laws, and clearly distinguishes compute-matched from data-matched. Limitation: theory is based on random feature models; quantitative transfer to real Transformers requires self-assessment.

  3. Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap

    Muon's pretraining loss advantage over AdamW reverses after fine-tuning at larger data budgets, with the crossover shrinking as model size grows.

    Editor's note

    For the first time systematically reveals that pretraining data scale can reverse the fine-tuning performance gap between Muon and AdamW: under small data budgets Muon dominates, under large data budgets AdamW overtakes, and the crossover point advances as model size increases. Proposes margin-to-spread ratio (MSR) as a new metric for predicting fine-tuning robustness, and proves in a linear associative memory model that GD's MSR grows with data budget and surpasses Muon's. Compared to Issue 11's When Does Muon Actually Win paired benchmark, this paper is the first to extend optimizer comparison from pretraining loss to the learning-forgetting trade-off after fine-tuning, and provides a theoretical explanation. Validated at 30M–600M multiple scales and multiple data budgets. Scale stops at 600M, fine-tuning covers only four datasets; extrapolation to larger scales and broader tasks requires caution.

  4. Train the Sampler You Run: Few-Step Masked Diffusion Language Models via Mixture-Likelihood Fine-Tuning

    Mix-FT fine-tunes diffusion language models on their own few-step sampler's mixture likelihood, halving inference steps while matching or beating original quality at under 0.002% of pretraining compute.

    Editor's note

    For the first time treats the few-step sampler of masked diffusion language models itself (the mixture distribution of random reveal schedules) as the fine-tuning target: using the importance-weighted IWAE bound to directly optimize the sampler's mixture likelihood, rather than cross-entropy fitting the true conditional distribution. Compared to Issue 8's PUMBA trajectory-aware training, this is the first scheme that is teacher-free, architecture-unchanged, and only changes the loss function to align training and inference objectives, validated at 110M/1.1B/8B multiple scales, with 16-step performance on 8B surpassing the original model's 32-step. text8 exact likelihood decomposition proves the gains mainly come from schedule weighting. Limitation: requires a CE pretraining starting point; starting from undertrained weights loses to continuing CE.

  5. Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model Size

    Jointly tuning learning rate and weight decay eliminates about 90% of Muon-SW's reported advantage, leaving a small residual that grows with model scale.

    Editor's note

    For the first time quantifies the true gains of Muon with scaled weight decay (Muon-SW) relative to constant weight decay under a protocol that jointly tunes the LR·WD product: finds that about 90% of the untuned advantage is a tuning gap, but the residual advantage grows with scale (survival rate 10%→21%), and gives a scaling law for the optimal LR·WD product moving with scale. Compared to Issue 7's Scale Weight Decay, which proposed Muon-SW and reported gains, this paper reveals through a paired tuning protocol that most of its gains come from an untuned baseline, and is the first to measure the scale trend of the residual advantage. Although the scale is small (15M active), the method (joint tuning, source scaling, multiple scale points) and conclusions (tuning gap, optimal product scaling) have methodological value.

  6. MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

    MuonGS, a Gram-Schmidt projection variant of Muon, achieves up to 15% faster LLM pretraining by guaranteeing square-root accumulation of effective learning rate without fixing weight norms.

    Editor's note

    For the first time elevates the analysis of Muon's weight norm control from a single-step perspective to the cumulative effective learning rate (C_T) over the entire training trajectory, proving that vanilla Muon's cumulative can only grow logarithmically, potentially limiting exploration, while MuonW and fixed-norm control guarantee linear growth, and proposes MuonGS (Gram-Schmidt projection to remove the radial component of updates) achieving Ω(√T) cumulative guarantee without fixed norms. Compared to Issue 12's Improving Data-Constrained Scaling, which only treats hyperparameters as scaling axes, this paper is the first to propose a trajectory-level exploration theory for norm control. Validated on 160M–400M dense and 135M active parameter MoE, reporting learning rate sweeps and long-horizon gains. Scale does not reach the frontier, but the mechanistic insight is clear.