The "Pathology" of Scaling Law Extrapolation Dissected: Pre-asymptotic Expansions and Negative Results under Data Constraints

Highlights of This Issue

The most prominent thread this issue is the further dissection of the reliability of scaling law extrapolation itself. Beyond the Log-Linear Frontier (https://arxiv.org/abs/2611.13534) introduces, for the first time, Mellin–Barnes residue expansions and KKT dual water-filling methods into the pre-asymptotic analysis of finite-budget scaling laws: under spectral geometry assumptions, it provides a third-order capacity expansion, proves that the local log-log slope systematically deviates from the asymptotic exponent (overshoot or undershoot) at finite scales, and gives logarithmic corrections when poles coalesce, uses Aitken's delta-squared process to eliminate the ill-conditioned multi-power-law fitting initialization of loss floors, and derives a variance identity for the drift of the compute-optimal allocation exponent, validated on Pythia, Chinchilla, and RegMix. This directly responds to the "fitting family uncertainty envelope" problem raised by Ambiguous Fits in Issue 12—but advances the focus from "how uncertain is the fit" to "why is it uncertain, and how can it be characterized analytically from spectral geometry." In parallel, a practical negative result (https://researchaudio.io/p/more-compute-made-the-model-worse-here-is-why) reports a specific inflection point where loss increases with compute after approximately 1e19 FLOPs under a data-constrained scaling recipe, providing an independent anchor for the same pathology that Improving Data-Constrained Scaling in Issue 12 aimed to fix.

Optimizer theory is equally dense. When and How SignSGD Outperforms SGD (https://arxiv.org/abs/2611.16560) extends, for the first time, the functional scaling law of SGD to sign-based optimizers: it rigorously derives the signal learning and noise forgetting laws for SignSGD in power-law kernel regression, revealing that gradient noise induces an effective preconditioner through the smoothed sign operator (transforming the effective Hessian from H to H^{1/2}), thereby achieving a faster data scaling exponent in the hard-task regime and a larger critical batch size in the easy-task regime, and provides an explicit phase diagram for when SignSGD outperforms SGD. This shares the same origin as the noise geometry analysis in When and How Does Muon Precondition from Issue 12, but generalizes the "noise-induced preconditioning" mechanism to sign operators and provides a complete batch-size dependence. FOAM (https://arxiv.org/abs/2611.16905) offers a systematic compression of SOAP-class optimizers from the engineering side: based on the Kronecker structure of the Hessian and the near-diagonality of the output side, it justifies retaining the input-side factor and discarding the output-side factor, reducing the SOAP state from 2mn+2m²+2n² to mn+2n²+m+n, while maintaining a loss gap within 0.005 nats compared to SOAP/Muon on models from 124M to 1.56B. Complete Scion (https://arxiv.org/abs/2611.15291) generalizes, for the first time, the 1/L residual scaling depth-transfer framework of CompleteP to constrained spectral optimizers (Frank-Wolfe and projected non-Euclidean normalized steepest descent), proving that Scion satisfies the two desiderata of stable initialization and stable feature updates, and provides a width-direction transfer proof that was previously only available for Unconstrained Scion.

On the masked diffusion side, Masked diffusion as random sequential adsorption (https://arxiv.org/abs/2611.07845) precisely maps, for the first time, the parallel decoding error to a random sequential adsorption (RSA) process: for Markovian data, it gives closed-form solutions for the path error and output perplexity at any number of steps, proves that the optimal schedule is a geodesic of the thermodynamic metric, and reveals the influence of correlation length on error concentration and schedule convergence—providing the first exactly solvable statistical physics model for the data-efficiency debate between masked diffusion and autoregressive models. On the pretraining objective side, an ICLR 2026 work (https://proceedings.iclr.cc/paper_files/paper/2026/file/693d7541bd31cca2dcb780abfd8d204c-Paper-Conference.pdf) proposes Future Summary Prediction (FSP) as a successor to MTP: using a reverse language model or bag-of-words summaries as future supervision targets, it validates on 3B/8B models that the advantage of future-aware objectives over NTP in the data-constrained (20-epoch iso-compute) setting grows as unique tokens decrease.

Community and Dynamics

• A frontier-scale engineering report (https://pytorch.org/blog/training-a-1t-parameter-moe-on-eu-public-infrastructure-with-pytorch-and-torchtitan/) publicly discloses, for the first time, the application of Muon (Newton–Schulz orthogonalization) to a 952B MoE: using FlexShard to shard by entire experts, running the Newton–Schulz for expert matrices on each rank without communication, and moving the optimizer offload to Grace processors before moving Newton–Schulz back to GPUs, reducing the optimizer step from 41.7s to 2.2s and improving throughput by 35%. This is a direct engineering response to the open question of whether Muon's benefits survive at frontier scale in sparse expert models, but the report focuses on throughput and configuration, without providing training curves or scaling law comparisons.
• A pre-registered coupled scaling law study (https://www.alphaxiv.org/abs/2611.autoregressive-masked-diffusion-scaling-laws) directly compares autoregressive and masked diffusion language models (100M–2.9B) on shared data, explicitly modeling compute-optimal allocation and the economics of "converting AR checkpoints to MDLM": the pre-registered coupled power law hits two runs at 2.9B (errors 0 and +1.0 millinats), while a broken power law with a fixed irreducible loss of 1 nat is rejected; conversion is only cheaper when the AR phase is very short after accounting for parent model costs. This is the first head-to-head with pre-registration and conversion analysis in the masked diffusion vs. AR data-efficiency debate.
• An ICLR 2026 work on RL as a pretraining objective (https://proceedings.iclr.cc/paper_files/paper/2026/file/44a45e27879b8fbab6e123ad8b93afc2-Paper-Conference.pdf) formalizes RL pretraining with an information-gain reward, provides a monotonic improvement guarantee, and compares compute-matched with RPT/CPT, offering a theoretical framework for RL-as-pretraining; a discussion note (https://en.papernotes.org/ICLR2026/reinforcement_learning/rlp_reinforcement_as_a_pretraining_objective/) adds a cross-entropy reduction identity and an EMA-baseline dynamics perspective, but both lack multi-scale extrapolation residuals, so it is advisable to verify the compute-matched claims before adopting them.

Open Questions

1. The third-order expansion and pole-coalescence logarithmic corrections in Beyond the Log-Linear Frontier rely on spectral assumptions and quadratic risk reduction. Can these pre-asymptotic predictions (local slope overshoot/undershoot, allocation exponent drift) hold quantitatively on real Transformers, and can the Aitken delta-squared initialization stably recover multi-power-law fits on narrow pilot grids?
2. SignSGD's noise-induced preconditioning relies on noise-curvature alignment and diagonal feature assumptions. Does this mechanism also apply to Muon-family optimizers that depend on noise geometry, and does its critical batch size prediction (SignSGD uses D^{1/(2α)} steps to maintain the data scaling rate) hold at real LLM scales?
3. FOAM's input-side priority criterion is built on K-FAC assumptions (with a self-reported violation degree of 15–20%, mainly in the most critical projection layers). At frontier scales where output-side curvature is more significant (very large vocabularies, multi-task heads), does discarding the output-side factor still cost only about 0.002 nats?
4. The Muon report on the 952B MoE uses throughput as the metric and does not provide training curves. Do Muon's loss benefits and stability relative to AdamW still hold at that scale, and can FlexShard's whole-expert sharding technique be generalized to architectures with a higher proportion of dense matrices?
5. The RSA exact model is limited to Markovian data (bigram) and does not apply to the long-range power-law correlations of real language. Can the "influence of correlation length on error concentration and schedule convergence" be extended to power-law correlated data, thereby providing testable predictions for scheduling design in real MDLMs?