Scaling Laws & Training MethodsIssue 12Oct 24 – 31, 2026

Same Scaling Law Fits, Radically Different Extrapolations

Highlights

This issue brings two threads together into one question: to what extent can we hold accountable the "laws" we believe we have established. The first thread targets the scaling law fitting process itself. Ambiguous Fits: The Rashomon Effect (https://arxiv.org/abs/2610.34514) elevates the fitting stage from a routine procedure to a verifiable optimization problem for the first time: using a deterministic branch-and-bound algorithm to provide provable lower bounds on global near-optimality, and systematically mapping the Rashomon effect of the near-optimal fitting family—i.e., parameter regions where fitting loss is nearly identical but extrapolation predictions differ dramatically (on real multi-scale data, cross-extrapolation efficiency predictions can differ by tens to hundreds of times)—thereby providing strict upper and lower bounds on the prediction envelope. This corroborates several threads of community self-examination: A Steeper Slope Is Not Yet a New Law (https://research.pranaymahendrakar.com/p/a-steeper-slope-is-not-yet-a-new-law) systematically decomposes five protocol selection artifacts behind "reported steepening exponents," Zotpaper's report on "equally good scaling law fits can predict very different results" (https://zot.news/article/equally-good-scaling-law-fits-can-predict-very-different-25i3z), and the retrospective on scaling law history (https://caiosaldanha.com/en/essays/scaling-laws-and-the-data-wall/) all push the field from "trusting a single optimal curve" toward "reporting a family of fits."

The second thread concerns mechanistic accountability for training stability. The Role of Quantization in Loss Spikes (https://arxiv.org/abs/2610.38277) advances the cause of loss spikes to the hidden mechanism of "how Adam-induced logit mean drift amplifies finite-precision quantization errors in the LM head," unifying existing mitigations such as z-loss, μ-centering, and LM head SGD-M under the common mechanism of "suppressing quantization errors," and proposing a new category of time-bomb spikes. Does LLM Pre-Training Occur at the Edge of Stability? (https://arxiv.org/abs/2610.36778) systematically characterizes two failure modes of the quadratic approximation on LLM pretraining up to 787M—Edge of Convexity caused by negative curvature and generalized Edge of Stability under stochastic adaptive optimizers—and provides a scalable detection algorithm, revealing that EoS depends on critical thresholds of batch size and learning rate schedules. The earlier spike attribution debate (Issue 3 identifying low-precision cross-entropy numerical artifacts) is here advanced to a new scale. On the community side, Adaptive Preconditioners Trigger Loss Spikes in Adam (https://proceedings.mlr.press/v306/bai26o.html) also refutes the sharpness explanation and identifies the decoupling of Adam's second-moment estimator from instantaneous gradients as the root cause, complementing the mechanism in this issue's Quantization paper.

On optimizer theory, this issue continues the inquiry into Muon. When and How Does Muon Precondition? (https://arxiv.org/abs/2610.31728) for the first time treats the structure of stochastic gradient noise (task frequency, label noise, sparse support) as the decisive factor in Muon's preconditioning gains, proving that under isotropic noise Muon provides no directional preconditioning, and establishing a finite-batch balance threshold and a square-root law for task frequency; Last-Iterate Stationarity of Muon (https://arxiv.org/abs/2610.34158) provides the first finite-step Newton–Schulz last-iterate convergence guarantee, and clearly distinguishes two obstacles: two-cycles caused by lack of input stabilization and stochastic noise. The accountability approach to Muon mirrors that for scaling laws: not trusting default assumptions, advancing from "why it works" to "when it fails."

Community and Updates

  • Diffusion Beats Autoregressive in Data-Constrained Settings (https://proceedings.neurips.cc/paper_files/paper/2025/file/0f705a932553c08ebf0d1bc520b7cbc6-Paper-Conference.pdf) advances the comparison between masked diffusion and autoregressive models under a compute-matched protocol into the data-constrained regime with repeated epochs: the previously reported need for up to 16× compute for diffusion is shown to be partly an artifact of single-epoch evaluation. The paper derives a critical compute crossover power law C_crit(U) ∝ U^2.174 where diffusion overtakes autoregressive models, and explicitly decouples compute from unique data, continuing this line's compute-matched requirement for the diffusion vs. autoregressive data efficiency debate.
  • NVIDIA Megatron Core's bitwise deterministic pretraining at scale (https://developer.nvidia.com/blog/scale-bitwise-deterministic-pretraining-with-nvidia-megatron-core/) implements bitwise deterministic reproducibility at frontier scale, serving as a tooling engineering contribution for reproducing and localizing loss spikes, worth attention from stability researchers.
  • Criticism of the Chinchilla scaling law continues to ferment: discussions arguing that it fails calibration in the repeated-token data-constrained regime and that the repetition threshold itself is an extrapolation from sub-10B fits (https://proofoftech.org/blog/chinchilla-wrong-in-the-regime-you-train/) provide testable hypotheses for cross-regime transfer of data-constrained scaling laws.

Open Questions

  1. Ambiguous Fits' certified prediction envelope is computationally expensive for the five-parameter law and only practical for the three-parameter law. As the degrees of freedom in functional forms increase, is there a fundamental trade-off between "verifiable global near-optimality" and "solution cost," or could a certificate algorithm emerge that satisfies both while covering hyperparameter axes (learning rate, batch)?
  2. This issue's Quantization paper unifies z-loss, μ-centering, and LM head SGD-M under the same mechanism of "suppressing LM head quantization errors." Is this the same spike at a different scale as the Slingshot mechanism from Issue 3's Grokking or Glitching (cross-entropy numerical artifacts, zero-sum constraint violation), or are they two independent mechanisms—can they be distinguished on naturally occurring spikes in real large models?
  3. The Edge of Stability paper reveals that EoS depends on critical thresholds of batch size and learning rate schedules. Does the scalable detection remain effective above 787M, and does Edge of Convexity occur frequently in real LLM pretraining, or mainly in synthetic objectives?
  4. When and How Does Muon Precondition predicts that under label noise, task-frequency dependence softens from linear to square-root. Real pretraining data has far more heterogeneous task/domain distributions than associative memory models; at what scale does this square-root law break down under real data distributions?

Papers in this issue

  1. Ambiguous Fits: The Rashomon Effect and Challenges in Scaling Law Estimation

    Scaling law fitting suffers a Rashomon effect where near-optimal fits predict wildly divergent compute needs, and certified branch-and-bound reveals standard heuristics can miss the optimum by up to 180%.

    Editor's note

    For the first time treats the scaling law fitting stage itself as the object of study: using a deterministic branch-and-bound algorithm to provide provable lower bounds on global near-optimality, and systematically mapping the Rashomon effect of the near-optimal fitting family—parameter regions where fitting loss is nearly identical but extrapolation predictions differ dramatically (on real multi-scale data, cross-extrapolation efficiency gaps can reach 74.6× or even 146×)—providing strict upper and lower bounds on the prediction envelope. Compared to Issue 8's ScAn-Bench fitting methodology evaluation and Issue 10's SLDBench v2 active experimental design, this is the first work providing verifiable fitting loss lower-bound certificates and quantifying extrapolation uncertainty, a first step from "reporting a single optimal curve" to "reporting a family with guaranteed envelopes." Limitation: strict envelopes for the five-parameter law are computationally expensive; only the three-parameter law is practical.

  2. The Role of Quantization in Loss Spikes

    Finite precision error in the LM head, amplified by Adam-induced logit drift, is the hidden driver of time-bomb loss spikes, preventable by using SGD with momentum on that head.

    Editor's note

    For the first time advances the cause of loss spikes to the hidden mechanism of "Adam-induced logit mean drift amplifying finite-precision quantization errors in the LM head": mean drift is harmless under exact arithmetic but amplifies quantization errors; lower precision accelerates time-bomb spikes while higher precision stabilizes training, and unifies z-loss, μ-centering, and LM head SGD-M as suppressing quantization errors. Compared to Issue 3's Grokking or Glitching attributing spikes to the Slingshot mechanism of cross-entropy numerical artifacts, this paper focuses on the interaction between Adam preconditioning and quantization errors, reporting stability and quantization error curves across 28M–1.4B scales. Limitation: scale stops at 1.4B, spikes are artificially induced by high learning rates; extrapolation to natural large-scale spikes requires self-assessment.

  3. Does LLM Pre-Training Typically Occur at the Edge of Stability?

    LLM pretraining always exhibits edge of convexity (negative Hessian curvature), while edge of stability occurs only above a batch-size threshold that correlates strongly with the critical batch size.

    Editor's note

    For the first time systematically characterizes two failure modes of the quadratic approximation at LLM pretraining scales up to 787M: divergence due to negative curvature, independent of learning rate (Edge of Convexity), and generalized Edge of Stability under stochastic adaptive optimizers, providing a scalable detection algorithm with theoretical guarantees that reduces to Cohen's full-batch EoS while additionally identifying EoC. Compared to Issue 9's Muon Sublates the Edge of Stability focusing on Muon's stability phase diagram, this paper targets AdamW and quadratic approximation failure modes, and reveals that EoS depends on critical thresholds of batch size and learning rate schedules, providing direct guidance for scheduling and batch selection. Limitation: scale stops at 787M; scalability to larger scales requires self-assessment.

  4. When and How Does Muon Precondition? The Role of Structured Gradient Noise

    Muon's preconditioning benefits require structured gradient noise, not isotropic noise, and it softens task-frequency dependence from linear to square-root scaling.

    Editor's note

    For the first time answers when and how Muon preconditioning works under the geometry of stochastic gradient noise: under isotropic Gaussian noise with low signal-to-noise ratio, Muon's expected update is equivalent to learning-rate-rescaled SGD (no directional preconditioning), and establishes a two-task finite-batch balance threshold B_bal and a square-root law decoupling task frequency from gradient reliability under label noise. Compared to Issue 11's A Statistical Viewpoint statistical unification framework and Issue 9's Muon Sublates the Edge of Stability stability picture, this paper is the first to treat the structured noise of stochastic mini-batches as the decisive factor in Muon's preconditioning gains. Limitation: scale stops at 0.12B single run, theory based on associative memory models; transfer to natural LLM data distributions requires self-assessment.

  5. Last-Iterate Stationarity of Muon with Finitely Many Newton-Schulz Iterations

    Finite-step Muon achieves last-iterate gradient convergence on general nonconvex objectives via input stabilization with fixed steps, or via decreasing learning rates paired with increasing momentum under stochastic noise.

    Editor's note

    For the first time provides last-iterate (rather than averaged/best-iterate) gradient convergence guarantees for finite-step Newton–Schulz Muon, and proves that without input stabilization, two-cycles appear on smooth convex quadratic objectives, while positive stabilization combined with descent-tracking Lyapunov analysis restores fixed-step convergence. Compared to Issue 11's A Quadratic Lens exact one-step quadratic analysis, this paper provides sector inequalities for finite NS maps and full-sequence convergence, clearly distinguishing stabilization from stochastic noise as two classes of obstacles. Purely theoretical, no experimental validation; scale extrapolation and practicality require self-judgment. Its density-one rate concept offers a new perspective for evaluating practical Muon.

  6. Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling

    Data-constrained scaling laws must treat ensembling as a third compute-allocation axis and scale hyperparameters via power laws, yielding 14% lower perplexity than prior recipes.

    Editor's note

    For the first time in data-constrained scaling laws explicitly models hyperparameters (weight decay/effective learning rate) and ensemble size as independent axes, jointly predicting optimal (N,R,M) configurations, and fixes the pathology of existing scaling laws where loss is non-monotonic in compute under fixed hyperparameters. Compared to Issue 2's Practical Scaling Laws which only added the T axis and overfitting terms, this paper adds power-law scaling of hyperparameters with scale and variance-reduction forms for ensembles, with 30M–398M multi-scale, up to 12× extrapolation, and public code enhancing credibility. Note: loss reporting focuses on final values rather than training curve stability; Muon and special architecture components (value lookup, conditional decay) may introduce domain limitations; transferability requires assessment in your own setting.