Summary (Overview)

  • OpenGNS Dataset: The paper introduces the first open dataset of Gradient Noise Scale (GNS) measurements, containing over 400 training runs across three domains: image classification (ResNets on CIFAR-10), autoregressive language modeling (GPTs on FineWeb), and image synthesis (DiTs on ImageNet), covering models from 11M to 1B parameters under Maximal Update Parameterization (muP).

  • GNS as "Temperature of Training": The study confirms that GNS behaves consistently across runs that achieve similar validation losses, validating McCandlish et al.'s (2018) theoretical framework for modern Transformer and diffusion architectures.

  • Task-Dependent GNS-CBS Bias: The GNS tends to overestimate the Critical Batch Size (CBS) for classification tasks (ResNet) and underestimate it for generative tasks (GPT, DiT), with consistent direction across common CBS definitions.

  • Cross-Size Consistency under muP: GNS measurements remain roughly consistent across model sizes at equivalent training loss levels, suggesting that measurements from smaller proxy models could inform batch-size selection for larger models.

  • Practical Resource Allocation: GNS-guided batch-size scheduling could enable more efficient GPU utilization by using smaller batches early in training and scaling up as training progresses.


Introduction and Theoretical Foundation

Background and Motivation

The Gradient Noise Scale (GNS), introduced by McCandlish et al. (2018), provides a principled way to understand the trade-offs in stochastic gradient descent. The intuition is straightforward: when gradient noise is high, larger batch sizes help by averaging over more samples; when noise is low, additional samples only increase computational cost without improving gradient quality. This connects GNS to the Critical Batch Size (CBS)—the batch size that balances runtime and computational effort to reach a given loss level.

The practical importance of this metric has grown with the rise of foundation models. Pre-training large language models and diffusion models consumes substantial computational resources, often distributed across thousands of GPUs. Since the global batch size directly determines how many GPUs can be used in data-parallel training, batch-size selection critically affects training efficiency, resource utilization, and final model quality.

Despite its appearance in technical reports of frontier models (e.g., GPT-3), no comprehensive analysis has examined how GNS behaves across modern architectures, training regimes, and hyperparameter configurations. No open dataset of GNS measurements has been available to the community for systematic study.

Theoretical Foundation: The Gradient Noise Scale

The GNS is derived from a second-order Taylor expansion of the loss LL of a neural network. Let θ\theta denote the NN parameters, GG the true gradient, and HH the Hessian. Perturbing the parameters by a step ϵ\epsilon along direction VV:

L(θ−ϵV)≈L(θ)−ϵGTV+12ϵ2VTHV.(1)L(\theta - \epsilon V) \approx L(\theta) - \epsilon G^{T} V + \frac{1}{2} \epsilon^{2} V^{T} H V. \tag{1}

With V=GV = G, this corresponds to gradient descent with learning rate ϵ\epsilon. Since SGD only provides an unbiased gradient estimate GestG_{est} with E[Gest(θ)]=G(θ)\mathbb{E}[G_{est}(\theta)] = G(\theta), and the covariance matrix scales as:

cov(Gest(θ))=1BΣ(θ),(3)cov(G_{est}(\theta)) = \frac{1}{B} \Sigma(\theta), \tag{3}

where BB is the batch size and Σ\Sigma is the per-example covariance matrix, the expected loss becomes:

E[L(θ−ϵGest)]≈L(θ)−ϵ∣G∣2+12ϵ2(GTHG+tr(HΣ)B).(4)\mathbb{E}[L(\theta - \epsilon G_{est})] \approx L(\theta) - \epsilon |G|^{2} + \frac{1}{2} \epsilon^{2} \left(G^{T} H G + \frac{tr(H\Sigma)}{B}\right). \tag{4}

The optimal batch size (under optimal step size) is attained when the batch size equals the GNS:

Bnoise=tr(HΣ)GTHG.(5)B_{noise} = \frac{tr(H\Sigma)}{G^{T} H G}. \tag{5}

Under the assumption of perfect conditioning (Hessian proportional to identity), this simplifies to:

Bsimple=tr(Σ)GTG,(6)B_{simple} = \frac{tr(\Sigma)}{G^{T} G}, \tag{6}

which is easier to measure since computing the Hessian is expensive. The paper uses this BsimpleB_{simple} approximation throughout and refers to it as GNS.

Measuring the GNS in Data-Parallel Training

Given gradients at small and large batch sizes, the expected norms can be expressed as:

E[∣Gsmall∣2]=∣G∣2+1Bsmalltr(Σ),(7)\mathbb{E}[|G_{small}|^{2}] = |G|^{2} + \frac{1}{B_{small}} tr(\Sigma), \tag{7} E[∣Gbig∣2]=∣G∣2+1Bbigtr(Σ).(8)\mathbb{E}[|G_{big}|^{2}] = |G|^{2} + \frac{1}{B_{big}} tr(\Sigma). \tag{8}

Solving for ∣G∣2|G|^2 and tr(Σ)tr(\Sigma):

∣G∣2≈1Bbig−Bsmall(Bbig∣GBbig∣2−Bsmall∣GBsmall∣2)(9)|G|^{2} \approx \frac{1}{B_{big} - B_{small}} \left(B_{big} |G_{B_{big}}|^{2} - B_{small} |G_{B_{small}}|^{2}\right) \tag{9} tr(Σ)≈11/Bsmall−1/Bbig(∣GBsmall∣2−∣GBbig∣2)(10)tr(\Sigma) \approx \frac{1}{1/B_{small} - 1/B_{big}} \left(|G_{B_{small}}|^{2} - |G_{B_{big}}|^{2}\right) \tag{10}

In data-parallel training, the global gradient is computed as:

Gglobal(θ)=1N∑i=1NGi(θ),(11)G_{global}(\theta) = \frac{1}{N} \sum_{i=1}^{N} G_{i}(\theta), \tag{11}

where NN is the number of devices. By setting Gbig=GglobalG_{big} = G_{global}, Gsmall=GiG_{small} = G_{i}, and Bbig=N⋅BsmallB_{big} = N \cdot B_{small}, the necessary measurements are available with no additional overhead.

Definition of the Critical Batch Size

The CBS has multiple definitions in the literature. McCandlish et al. (2018) define it as the batch size where scaling efficiency drops below 50%. Zhang et al. (2024) tighten this to a 20% overhead allowance. The general ordering is:

Optimal Batch Size < Critical Batch Size (CBS) < Maximum Stable Batch Size


Methodology

Experimental Setup

Three model architectures were trained across three domains:

HyperparameterResNet18GPTDiT
DatasetCIFAR-10FineWebImageNet
Training data10M images20B tokens51M images
Validation data10,000 images100M tokens40,504 images
Widths[1, 2, 4][256, 512, 1,024, 2,048, 2,560][144, 288, 576]
Depth181228
Parameters11M, 45M, 179M22M, 64M, 203M, 707M, 1B11M, 43M, 169M
Batch size25−2142^{5} - 2^{14} images215−2212^{15} - 2^{21} tokens26−2102^{6} - 2^{10} images
Learning rate10−5−10−110^{-5} - 10^{-1}2−11−2−62^{-11} - 2^{-6}2−13−2−92^{-13} - 2^{-9}
OptimizerAdamWAdamWAdamW
β1\beta_10.90.90.9
β2\beta_20.990.950.99
Weight decay0.00.010.01
LR scheduleCosine (epoch based)Cosine (step based)Constant
Final LR10%10%-
LR warmupNone5% of training durationNone
α for GNS0.9950.950.995
Bbig/BsmallB_{big}/B_{small}488

Total configurations: 180 (ResNet) + 178 (GPT) + 75 (DiT) = 433 training runs

Key Implementation Details

  • Hardware: All models trained on the Leonardo Booster HPC system with NVIDIA A100 GPUs connected via InfiniBand
  • GNS measurement: Custom hook added to PyTorch-DDP, based on implementations from Crowson et al. (2024) and Gray et al. (2024)
  • GPU allocation: GPT and DiT models use 2 nodes (8 GPUs); ResNet models use 1 node (4 GPUs)
  • Total compute: ≈ 200,000 GPU hours for development and production runs

Dataset Contents

Each training run records:

  • Training and validation loss
  • Simplified noise scale BsimpleB_{simple} (stored as gns)
  • Squared gradient norm ∣G∣2|G|^2 (stored as gns_norm)
  • Trace of per-example gradient covariance tr(Σ)tr(\Sigma) (stored as gns_var)

Releasing the raw components separately allows users to study them independently and reconstruct the GNS themselves. The dataset is hosted on HuggingFace with three Parquet files: cv.parquet, nlp.parquet, and diffusion.parquet.


Empirical Validation / Results

GNS over Hyperparameters

The GNS measurement is highly sensitive to hyperparameter choices:

  • For the same model architecture and task, the measured GNS varies by nearly an order of magnitude when the learning rate is scaled by factors of 8 and 16 from the baseline
  • When the learning rate is too small relative to the batch size, the optimizer explores only a narrow region, leading to artificially inflated GNS measurements
  • When learning rate is appropriately scaled with batch size, configurations reaching similar validation losses settle into the same temperature regime with comparable GNS trajectories

Recovering Scaling Rules

The experiments recover the Adam-specific scaling behavior where the optimal learning rate scales as:

ϵ∝B\epsilon \propto \sqrt{B}

Specifically, scaling the batch size by a factor of four while doubling the learning rate produces nearly identical GNS trajectories for GPT and DiT models.

Additionally, consistent with predictions from Li et al. (2024), the learning-rate peak in batch-size–learning-rate scaling for well-optimized GPT runs occurs at batch sizes (2192^{19} and 2202^{20}) close to the final GNS values, between 0.5⋅1060.5 \cdot 10^6 and 10610^6 tokens.

GNS and CBS over Loss and Model Size

Key quantitative findings (Figure 3):

TaskGNS/CBS MagnitudeGNS vs. CBS Relationship
GPT (FineWeb)Up to 10610^6 tokensUnderestimates CBS
ResNet (CIFAR-10)Tens of thousands of imagesOverestimates CBS
DiT (ImageNet)Hundreds of imagesUnderestimates CBS

Cross-model-size consistency: GNS values remain roughly consistent across model sizes at the same training loss level. Large models attain higher GNS values primarily because they reach lower training losses than smaller models. This consistency under muP suggests that measurements from smaller proxy models could inform both learning-rate and batch-size selection for larger models.

Direction of GNS-CBS deviation: The systematic discrepancy persists under alternative overhead thresholds of 10%, 30%, and 50%, though the magnitude varies. The gap for the classification task largely closes under the original 50% definition from McCandlish et al. (2018).


Theoretical and Practical Implications

Theoretical Implications

  1. Validation of the "Temperature of Training" Concept: The GNS behaves as a consistent "temperature" across runs that reach similar validation losses, extending McCandlish et al.'s (2018) findings to modern Transformer and diffusion architectures.

  2. Ill-Conditioning as an Explanation for GNS-CBS Bias: The simplified assumption of a well-conditioned Hessian used to derive BsimpleB_{simple} in Eq. (5) may hold to different degrees across task types. For NLP tasks, the heavy-tailed class imbalance of the token distribution is known to induce exactly this kind of ill-conditioning (Kunstner et al., 2024).

  3. muP Enables Cross-Size Transfer: The combination of muP and GNS consistency across model sizes at equivalent loss suggests that scaling laws for batch-size selection may be transferable across model widths.

Practical Implications

  1. Reduced HPO Cost: Cross-size consistency of GNS under muP suggests that measurements from smaller proxy models could inform batch-size selection for larger ones, reducing the computational cost of hyperparameter optimization at scale.

  2. GNS-Guided Batch-Size Scheduling: Since smaller global batch sizes correspond to fewer GPUs required for data-parallel training, GNS-guided batch-size scheduling could enable more efficient resource utilization—using fewer GPUs early in training when small batches suffice, and scaling up as training progresses.

  3. Task-Specific Batch-Size Recommendations: The results confirm that GPTs benefit from batch sizes in the millions of tokens, while DiTs operate efficiently at much smaller batch sizes on the order of hundreds of images (corresponding to roughly tens of thousands of patch tokens).

  4. Stability Boundary Detection: The dataset includes runs that diverge (9 out of 75 DiT runs and 12 out of 210 NLP runs), as the onset of instability itself marks a stability boundary of the batch-size–learning-rate grid.


Conclusion

This study contributes OpenGNS, the first open dataset of GNS measurements spanning three domains: image classification, autoregressive language modeling, and diffusion-based image synthesis. The empirical study enabled by this dataset:

  1. Confirms that GNS behaves as a "temperature of training," remaining consistent across runs that reach similar validation losses, and extends this observation to modern Transformer and diffusion architectures under muP.

  2. Demonstrates that the GNS-CBS relationship is task-dependent: GNS acts more as a lower bound for the CBS in generative tasks and more as an upper bound in classification tasks.

  3. Suggests that cross-size consistency of GNS under muP enables proxy-model-based batch-size selection for larger models, relevant for the AutoML community.

  4. Points to practical avenues for resource allocation through GNS-guided batch-size scheduling.

Future directions: The authors hope the release of OpenGNS will encourage deeper theoretical understanding of the GNS metric and further investigation of efficient training strategies. The increase of GNS as training progresses suggests potential for dynamic batch-size scheduling that scales GPU allocation with training progress.

Limitations (referenced in App. A): The paper acknowledges certain limitations in theoretical assumptions and experimental scale, though these are not detailed in the main text.


Broader Impact

The paper's goal is to advance machine learning efficiency. Although the evaluations require significant computational resources, the insights provided enable more efficient training practices in the future, ultimately reducing resource waste and energy consumption in large-scale AI training.

Related papers