# OpenGNS: An Open Dataset of the Gradient Noise Scale across Vision, Language, and Diffusion Models

> OpenGNS, the first open dataset of gradient noise scale measurements, reveals that GNS underestimates critical batch size for generative models but overestimates it for classification.

- **Source:** [arXiv](https://arxiv.org/abs/2611.24331)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/2918b7
- **Whiteboard:** https://picx.dev/p/2918b7/image

## Summary

## Summary (Overview)

- **OpenGNS Dataset**: The paper introduces the first open dataset of Gradient Noise Scale (GNS) measurements, containing over 400 training runs across three domains: image classification (ResNets on CIFAR-10), autoregressive language modeling (GPTs on FineWeb), and image synthesis (DiTs on ImageNet), covering models from 11M to 1B parameters under Maximal Update Parameterization (muP).

- **GNS as "Temperature of Training"**: The study confirms that GNS behaves consistently across runs that achieve similar validation losses, validating McCandlish et al.'s (2018) theoretical framework for modern Transformer and diffusion architectures.

- **Task-Dependent GNS-CBS Bias**: The GNS tends to **overestimate** the Critical Batch Size (CBS) for classification tasks (ResNet) and **underestimate** it for generative tasks (GPT, DiT), with consistent direction across common CBS definitions.

- **Cross-Size Consistency under muP**: GNS measurements remain roughly consistent across model sizes at equivalent training loss levels, suggesting that measurements from smaller proxy models could inform batch-size selection for larger models.

- **Practical Resource Allocation**: GNS-guided batch-size scheduling could enable more efficient GPU utilization by using smaller batches early in training and scaling up as training progresses.

---

## Introduction and Theoretical Foundation

### Background and Motivation

The Gradient Noise Scale (GNS), introduced by McCandlish et al. (2018), provides a principled way to understand the trade-offs in stochastic gradient descent. The intuition is straightforward: when gradient noise is high, larger batch sizes help by averaging over more samples; when noise is low, additional samples only increase computational cost without improving gradient quality. This connects GNS to the Critical Batch Size (CBS)—the batch size that balances runtime and computational effort to reach a given loss level.

The practical importance of this metric has grown with the rise of foundation models. Pre-training large language models and diffusion models consumes substantial computational resources, often distributed across thousands of GPUs. Since the global batch size directly determines how many GPUs can be used in data-parallel training, batch-size selection critically affects training efficiency, resource utilization, and final model quality.

Despite its appearance in technical reports of frontier models (e.g., GPT-3), no comprehensive analysis has examined how GNS behaves across modern architectures, training regimes, and hyperparameter configurations. No open dataset of GNS measurements has been available to the community for systematic study.

### Theoretical Foundation: The Gradient Noise Scale

The GNS is derived from a second-order Taylor expansion of the loss $L$ of a neural network. Let $\theta$ denote the NN parameters, $G$ the true gradient, and $H$ the Hessian. Perturbing the parameters by a step $\epsilon$ along direction $V$:

$$
L(\theta - \epsilon V) \approx L(\theta) - \epsilon G^{T} V + \frac{1}{2} \epsilon^{2} V^{T} H V. \tag{1}
$$

With $V = G$, this corresponds to gradient descent with learning rate $\epsilon$. Since SGD only provides an unbiased gradient estimate $G_{est}$ with $\mathbb{E}[G_{est}(\theta)] = G(\theta)$, and the covariance matrix scales as:

$$
cov(G_{est}(\theta)) = \frac{1}{B} \Sigma(\theta), \tag{3}
$$

where $B$ is the batch size and $\Sigma$ is the per-example covariance matrix, the expected loss becomes:

$$
\mathbb{E}[L(\theta - \epsilon G_{est})] \approx L(\theta) - \epsilon |G|^{2} + \frac{1}{2} \epsilon^{2} \left(G^{T} H G + \frac{tr(H\Sigma)}{B}\right). \tag{4}
$$

The **optimal batch size** (under optimal step size) is attained when the batch size equals the GNS:

$$
B_{noise} = \frac{tr(H\Sigma)}{G^{T} H G}. \tag{5}
$$

Under the assumption of perfect conditioning (Hessian proportional to identity), this simplifies to:

$$
B_{simple} = \frac{tr(\Sigma)}{G^{T} G}, \tag{6}
$$

which is easier to measure since computing the Hessian is expensive. The paper uses this $B_{simple}$ approximation throughout and refers to it as GNS.

### Measuring the GNS in Data-Parallel Training

Given gradients at small and large batch sizes, the expected norms can be expressed as:

$$
\mathbb{E}[|G_{small}|^{2}] = |G|^{2} + \frac{1}{B_{small}} tr(\Sigma), \tag{7}
$$

$$
\mathbb{E}[|G_{big}|^{2}] = |G|^{2} + \frac{1}{B_{big}} tr(\Sigma). \tag{8}
$$

Solving for $|G|^2$ and $tr(\Sigma)$:

$$
|G|^{2} \approx \frac{1}{B_{big} - B_{small}} \left(B_{big} |G_{B_{big}}|^{2} - B_{small} |G_{B_{small}}|^{2}\right) \tag{9}
$$

$$
tr(\Sigma) \approx \frac{1}{1/B_{small} - 1/B_{big}} \left(|G_{B_{small}}|^{2} - |G_{B_{big}}|^{2}\right) \tag{10}
$$

In data-parallel training, the global gradient is computed as:

$$
G_{global}(\theta) = \frac{1}{N} \sum_{i=1}^{N} G_{i}(\theta), \tag{11}
$$

where $N$ is the number of devices. By setting $G_{big} = G_{global}$, $G_{small} = G_{i}$, and $B_{big} = N \cdot B_{small}$, the necessary measurements are available with **no additional overhead**.

### Definition of the Critical Batch Size

The CBS has multiple definitions in the literature. McCandlish et al. (2018) define it as the batch size where scaling efficiency drops below 50%. Zhang et al. (2024) tighten this to a 20% overhead allowance. The general ordering is:

> **Optimal Batch Size < Critical Batch Size (CBS) < Maximum Stable Batch Size**

---

## Methodology

### Experimental Setup

Three model architectures were trained across three domains:

| Hyperparameter | ResNet18 | GPT | DiT |
|---|---|---|---|
| **Dataset** | CIFAR-10 | FineWeb | ImageNet |
| **Training data** | 10M images | 20B tokens | 51M images |
| **Validation data** | 10,000 images | 100M tokens | 40,504 images |
| **Widths** | [1, 2, 4] | [256, 512, 1,024, 2,048, 2,560] | [144, 288, 576] |
| **Depth** | 18 | 12 | 28 |
| **Parameters** | 11M, 45M, 179M | 22M, 64M, 203M, 707M, 1B | 11M, 43M, 169M |
| **Batch size** | $2^{5} - 2^{14}$ images | $2^{15} - 2^{21}$ tokens | $2^{6} - 2^{10}$ images |
| **Learning rate** | $10^{-5} - 10^{-1}$ | $2^{-11} - 2^{-6}$ | $2^{-13} - 2^{-9}$ |
| **Optimizer** | AdamW | AdamW | AdamW |
| **$\beta_1$** | 0.9 | 0.9 | 0.9 |
| **$\beta_2$** | 0.99 | 0.95 | 0.99 |
| **Weight decay** | 0.0 | 0.01 | 0.01 |
| **LR schedule** | Cosine (epoch based) | Cosine (step based) | Constant |
| **Final LR** | 10% | 10% | - |
| **LR warmup** | None | 5% of training duration | None |
| **α for GNS** | 0.995 | 0.95 | 0.995 |
| **$B_{big}/B_{small}$** | 4 | 8 | 8 |

**Total configurations**: 180 (ResNet) + 178 (GPT) + 75 (DiT) = **433 training runs**

### Key Implementation Details

- **Hardware**: All models trained on the Leonardo Booster HPC system with NVIDIA A100 GPUs connected via InfiniBand
- **GNS measurement**: Custom hook added to PyTorch-DDP, based on implementations from Crowson et al. (2024) and Gray et al. (2024)
- **GPU allocation**: GPT and DiT models use 2 nodes (8 GPUs); ResNet models use 1 node (4 GPUs)
- **Total compute**: ≈ 200,000 GPU hours for development and production runs

### Dataset Contents

Each training run records:
- Training and validation loss
- Simplified noise scale $B_{simple}$ (stored as `gns`)
- Squared gradient norm $|G|^2$ (stored as `gns_norm`)
- Trace of per-example gradient covariance $tr(\Sigma)$ (stored as `gns_var`)

Releasing the raw components separately allows users to study them independently and reconstruct the GNS themselves. The dataset is hosted on HuggingFace with three Parquet files: `cv.parquet`, `nlp.parquet`, and `diffusion.parquet`.

---

## Empirical Validation / Results

### GNS over Hyperparameters

The GNS measurement is **highly sensitive to hyperparameter choices**:

- For the same model architecture and task, the measured GNS varies by **nearly an order of magnitude** when the learning rate is scaled by factors of 8 and 16 from the baseline
- When the learning rate is too small relative to the batch size, the optimizer explores only a narrow region, leading to **artificially inflated GNS measurements**
- When learning rate is appropriately scaled with batch size, configurations reaching similar validation losses settle into the **same temperature regime** with comparable GNS trajectories

### Recovering Scaling Rules

The experiments recover the **Adam-specific scaling behavior** where the optimal learning rate scales as:

$$ \epsilon \propto \sqrt{B} $$

Specifically, scaling the batch size by a factor of four while doubling the learning rate produces nearly identical GNS trajectories for GPT and DiT models.

Additionally, consistent with predictions from Li et al. (2024), the learning-rate peak in batch-size–learning-rate scaling for well-optimized GPT runs occurs at batch sizes ($2^{19}$ and $2^{20}$) close to the final GNS values, between $0.5 \cdot 10^6$ and $10^6$ tokens.

### GNS and CBS over Loss and Model Size

**Key quantitative findings** (Figure 3):

| Task | GNS/CBS Magnitude | GNS vs. CBS Relationship |
|---|---|---|
| GPT (FineWeb) | Up to $10^6$ tokens | **Underestimates** CBS |
| ResNet (CIFAR-10) | Tens of thousands of images | **Overestimates** CBS |
| DiT (ImageNet) | Hundreds of images | **Underestimates** CBS |

**Cross-model-size consistency**: GNS values remain roughly consistent across model sizes at the same training loss level. Large models attain higher GNS values primarily because they reach lower training losses than smaller models. This consistency under muP suggests that measurements from smaller proxy models could inform both learning-rate and batch-size selection for larger models.

**Direction of GNS-CBS deviation**: The systematic discrepancy persists under alternative overhead thresholds of 10%, 30%, and 50%, though the magnitude varies. The gap for the classification task largely closes under the original 50% definition from McCandlish et al. (2018).

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Validation of the "Temperature of Training" Concept**: The GNS behaves as a consistent "temperature" across runs that reach similar validation losses, extending McCandlish et al.'s (2018) findings to modern Transformer and diffusion architectures.

2. **Ill-Conditioning as an Explanation for GNS-CBS Bias**: The simplified assumption of a well-conditioned Hessian used to derive $B_{simple}$ in Eq. (5) may hold to different degrees across task types. For NLP tasks, the heavy-tailed class imbalance of the token distribution is known to induce exactly this kind of ill-conditioning (Kunstner et al., 2024).

3. **muP Enables Cross-Size Transfer**: The combination of muP and GNS consistency across model sizes at equivalent loss suggests that scaling laws for batch-size selection may be transferable across model widths.

### Practical Implications

1. **Reduced HPO Cost**: Cross-size consistency of GNS under muP suggests that measurements from smaller proxy models could inform batch-size selection for larger ones, reducing the computational cost of hyperparameter optimization at scale.

2. **GNS-Guided Batch-Size Scheduling**: Since smaller global batch sizes correspond to fewer GPUs required for data-parallel training, GNS-guided batch-size scheduling could enable more efficient resource utilization—using fewer GPUs early in training when small batches suffice, and scaling up as training progresses.

3. **Task-Specific Batch-Size Recommendations**: The results confirm that GPTs benefit from batch sizes in the millions of tokens, while DiTs operate efficiently at much smaller batch sizes on the order of hundreds of images (corresponding to roughly tens of thousands of patch tokens).

4. **Stability Boundary Detection**: The dataset includes runs that diverge (9 out of 75 DiT runs and 12 out of 210 NLP runs), as the onset of instability itself marks a stability boundary of the batch-size–learning-rate grid.

---

## Conclusion

This study contributes **OpenGNS**, the first open dataset of GNS measurements spanning three domains: image classification, autoregressive language modeling, and diffusion-based image synthesis. The empirical study enabled by this dataset:

1. **Confirms** that GNS behaves as a "temperature of training," remaining consistent across runs that reach similar validation losses, and extends this observation to modern Transformer and diffusion architectures under muP.

2. **Demonstrates** that the GNS-CBS relationship is task-dependent: GNS acts more as a **lower bound** for the CBS in generative tasks and more as an **upper bound** in classification tasks.

3. **Suggests** that cross-size consistency of GNS under muP enables proxy-model-based batch-size selection for larger models, relevant for the AutoML community.

4. **Points to** practical avenues for resource allocation through GNS-guided batch-size scheduling.

**Future directions**: The authors hope the release of OpenGNS will encourage deeper theoretical understanding of the GNS metric and further investigation of efficient training strategies. The increase of GNS as training progresses suggests potential for dynamic batch-size scheduling that scales GPU allocation with training progress.

**Limitations** (referenced in App. A): The paper acknowledges certain limitations in theoretical assumptions and experimental scale, though these are not detailed in the main text.

---

## Broader Impact

The paper's goal is to advance machine learning efficiency. Although the evaluations require significant computational resources, the insights provided enable more efficient training practices in the future, ultimately reducing resource waste and energy consumption in large-scale AI training.

---

_Markdown view of https://picx.dev/p/2918b7, served by PicX — AI-generated visual whiteboard summaries of research papers._
