The 16x Gap Between Diffusion and Autoregressive Models Reassessed
Highlights of This Issue
The most prominent theme this issue is the recalibration of the widely cited 16x pretraining compute gap between diffusion language models and autoregressive models. Revisiting the 16x Pretraining Compute Gap (https://arxiv.org/abs/2610.17590) systematically decomposes the sources of this gap using about 1,700 models under a compute-matched IsoFLOP protocol: replacing ELBO scoring with exact likelihood (DUEL) reduces the multiplier from 17.8 to 12.8, and block diffusion with encoder-decoder architectures further reduces it to 7.8 (b=32) and even 3.0 (b=4), along with an empirical rule that optimal decoder depth depends only on block size. This is corroborated by concurrent work deriving critical compute crossover power laws for diffusion and autoregressive models under data-limited settings (https://dl.acm.org/doi/10.52202/085713-0354), advancing the data efficiency debate between masked diffusion and autoregressive models from "which is better" to "under which protocol and budget is which better."
Progress has also been made on numerical issues in low-precision training and the reliability of scaling law extrapolation. Mixed Precision Scaling Laws for Quantized Pretraining (https://arxiv.org/abs/2610.17637) is the first to explicitly model persistent optimization errors in forward/backward passes within quantized pretraining scaling laws, proving that these errors do not vanish with model or data scale, thus predicting that compute-optimal precision increases with budget—directly contrasting with existing precision scaling laws that predict budget-independent optimal precision. The axis of optimizer evaluation continues to expand: How Inner Optimizers Shape Replica Scaling (https://arxiv.org/abs/2610.19064) extends the batch size axis established in Issue 9 to the replica count R, showing that tuning β2 with R allows IsoLoCo to outperform the more expensive MuLoCo.
On the hyperparameter transfer and experimental design side, ShellElastic (https://arxiv.org/abs/2610.15124) is the first to extend μP's hyperparameter transfer to elastic architectures (nested sub-networks), proving that all sub-networks can simultaneously maintain maximal updates under shared parameters; SLDBench v2 (https://arxiv.org/abs/2610.16297) extends scaling law discovery from passive fitting to an active experimental design problem, using real training records to reveal mechanisms where cheap acquisition leads to unobserved extrapolation conditions.
Community and Updates
- The companion blog for Issue 9's The Best Optimizer Depends on Batch Size (https://dangxingyu.github.io/batch-size-blog/) reiterates from a practitioner's perspective that "there is no context-free AdamW replacement": optimizer rankings vary with scale, data-parameter ratio, batch size, schedule, parameter partitioning, and tuning budget, providing a testable comparison framework for optimizer benchmarks.
- The project page for Revisiting the 16x Pretraining Compute Gap (https://www.guanghanwang.com/dlm-scaling/) characterizes the 16x gap as an artifact of outdated scaling law protocols (ELBO scoring, decoder-only full sequence), arguing that under modern diffusion language model practices the gap narrows to 3–8x, contributing to the ongoing debate on the reliability of scaling law protocols.
Open Questions
- Revisiting the 16x Pretraining Compute Gap narrows the 16x gap to 3–8x, but does the empirical rule that optimal decoder depth depends only on block size hold at larger scales and longer sequences? Can the two correction sources—evaluation protocol (ELBO vs. exact likelihood) and architectural choices—be further decoupled?
- Mixed Precision Scaling Laws predicts that compute-optimal precision increases with budget, complementing Issue 9's work on hardware failure rates as a scaling law axis. Can the persistent optimization error and hardware failure rate—both terms that do not vanish with scale—be unified into a joint precision-robustness scaling law?
- How Inner Optimizers Shape Replica Scaling transfers the scaling rule for β2 to replica count R, but Issue 9 showed that batch size can reverse optimizer rankings. Do the replica count R and batch size axes interact and further reverse rankings of whitening-type optimizers?
- Compute-Optimal Scaling Laws for Spiking LMs shows that parameter counting conventions shift the exponent from 0.75 to 0.56, crossing the Kaplan-Chinchilla gap. Is this convention sensitivity universal across non-standard architectures like MoE and spiking, or is it significant only for activation-binarized models?
Papers in this issue
The widely cited 16x compute gap between diffusion and autoregressive language models is overstated, shrinking to just 3x with exact likelihood scoring, block diffusion, and encoder-decoder architectures.
Editor's noteFirst systematic decomposition of the sources of the 16x pretraining compute gap between diffusion language models and autoregressive models under a compute-matched IsoFLOP protocol: replacing ELBO with exact likelihood (DUEL) reduces the multiplier from 17.8 to 12.8, and block diffusion with encoder-decoder architectures further reduces it to 7.8 (b=32) and even 3.0 (b=4), along with an empirical rule that optimal decoder depth depends only on block size. The scale of about 1,700 models and held-out budget validation directly address the compute-matched requirement in the data efficiency debate between masked diffusion and autoregressive models; worth a close read.
A new scaling law for quantized pretraining reveals compute-optimal precision increases with budget, unlike prior laws, due to persistent quantization error.
Editor's noteFirst to explicitly model persistent optimization errors (ϵ_f, ϵ_b) in forward/backward passes within quantized pretraining scaling laws, proving that these errors do not vanish with model or data scale, thus predicting that compute-optimal precision increases with budget—contrasting with existing precision scaling laws that predict budget-independent optimal precision. Validated across 280M–910M multiple scales and publicly releasing the Qwen3 experiment grid, directly addressing numerical issues in low-precision training and scaling law extrapolation reliability, complementing Issue 9's work on hardware failure rates as a scaling law axis.
ShellElastic's block lower-triangular architecture enables one forward pass to train all nested sub-networks simultaneously, matching separately trained models that use 5x more compute.
Editor's noteFirst to extend μP's hyperparameter transfer to elastic architectures (nested sub-networks): proposes μ-Elastic parameterization, proves that all sub-networks can simultaneously maintain maximal updates (μ-Internal), and achieves two-level transfer from non-elastic proxies to elastic families and from small to large families (μ-Transfer). Compared to Issue 1's Complete-muE and Issue 3's Let's Scale Step by Step, which only handle width/expert count transfer for MoE, this paper is the first to systematically address hyperparameter scaling for elastic architectures with theoretical guarantees.
Whitening-based inner optimizers like SPlus achieve near-optimal replica scaling by decorrelating pseudo-gradients, degrading only 2.1% versus data-parallel training at 96 replicas.
Editor's noteExtends the optimizer ranking reversal axis established in Issue 9's The Best Optimizer Depends on Batch Size from batch size to replica count R: decomposes the performance degradation of DiLoCo replica scaling into mean mismatch and stochastic error, proves that whitening-type inner optimizers (SOAP/SPlus) improve scaling by controlling the maximum directional variance of pseudo-gradient covariance, and transfers the token half-life scaling rule for β2 to R, showing that tuning β2 with R allows IsoLoCo to outperform the more expensive MuLoCo.
Active scaling-law discovery separates frontier AI agents far more than passive evaluation, with prediction error rising 1.5-4.7x because agents favor cheap experiments that leave extrapolation-critical conditions unobserved.
Editor's noteFirst to extend scaling law discovery from passive fitting to an active experimental design problem: an agent purchases experiments from a pool with a price list, iteratively revises the law, and decides when to stop. Compared to Issue 7's Amortizing Scaling Law Construction Costs, which optimizes acquisition efficiency, this paper constructs 30 difficult scenarios from real training records, systematically revealing new phenomena such as system ranking reversals and unobserved extrapolation conditions due to cheap acquisition in active settings.
Training spikes in LLM pretraining are reproduced by quadratic models with heavy-tailed inputs, where stochastic gradient noise interacting with power-law eigenvalue distributions breaks concentration of measure.
Editor's noteFirst to shift the cause of training spikes from Hessian maximum eigenvalue/third-order terms to the interaction between stochastic gradient noise and heavy-tailed data distributions: even convex quadratic models can produce spikes under heavy-tailed inputs, and provides a layer-wise random feature model simulation algorithm to predict spike behavior from a given checkpoint. Compared to Issue 3's Grokking or Glitching?, which attributes spikes to low-precision numerical artifacts, this paper offers a purely optimization dynamics mechanism; the two are complementary.
Compute-optimal scaling for spiking language models matches dense Transformers (exponent 0.56 vs 0.50), with the apparent deviation explained by parameter-counting conventions.
Editor's noteFirst to conduct a Chinchilla-style IsoFLOP scaling law study for spiking language models, finding that the compute-optimal allocation exponent α≈0.56 is close to the 0.5 of dense Transformers, and decomposing the loss gap into a compute-independent backbone cost and a compute-dependent spiking cost. The analysis of how parameter counting conventions affect the exponent (0.75→0.56) has methodological value for any researcher comparing scaling laws.
Neural Dimensional Analysis unifies Transformer scale-control mechanisms via dimensional typing, yielding a Gauge-Canonical Transformer with superior depth transfer and quantization robustness.
Editor's noteFirst to systematize physical dimensional analysis into a unified scale consistency framework for neural network computation graphs (NDA), proposing local type rules for neural units and designing a canonical transformer (GCT) that implements orthogonal addition and dimensionless coordinate computation in residual streams. Elevates scale control from optimizers or single mechanisms to a full-graph type system covering activations, objectives, gradients, and optimizers, with learning rate robustness and depth transfer validated across 51M–1B multiple scales.
Schedule-Free AdamW v2 wins AlgoPerf v1 self-tuning, but step-efficient methods like Muon need systems optimizations to match wall-clock performance.
Editor's noteSystematically evaluates 13 optimizers on 9 heterogeneous workloads under AlgoPerf v1's self-tuning rules in terms of wall-clock performance, and introduces hyperparameter significance analysis (1D Wasserstein distance) and retention rate metrics to quantify the tuning cost of a single configuration across workloads. Compared to Issue 7, which uses training time as the evaluation axis, this paper focuses on cross-domain generalization and tuning robustness, revealing that Muon excels in step efficiency but requires systematic optimization to be competitive, providing a quantifiable framework for optimizer benchmarks and hyperparameter transfer.
