Visual Summary | Grokking or Glitching? How Low-Precision Drives Loss Spikes

Summary (Overview)

This paper provides a theoretical explanation for the "Slingshot Mechanism" in neural network training, revealing it as an artifact of floating-point arithmetic rather than intrinsic optimization dynamics.
The authors identify a novel mechanism called Numerical Feature Inflation (NFI), a positive feedback loop between the global classifier weight mean and global feature mean, triggered by Softmax Collapse (SC) under finite precision.
They prove that SC breaks the zero-sum constraint on gradients, causing exponential growth of both weight and feature means, which eventually leads to loss spikes.
The paper validates the mechanism across multiple architectures (MLP, CNN, ViT, Transformer) and datasets, and proposes practical interventions (zero-sum projection, BatchNorm, mixed precision) that suppress NFI-induced instability.
The work bridges a critical gap between gradient-flow theory and real-world finite-precision training, with implications for large-scale model training stability.

Introduction and Theoretical Foundation

Background

The Slingshot Mechanism refers to periodic loss spikes observed during long-term unregularized training with cross-entropy (CE) loss. Previous work attributed these spikes to intrinsic optimization dynamics, such as Edge of Stability (EOS) or Adam's adaptive learning rate behavior. However, the authors demonstrate that the root cause is finite-precision arithmetic in the CE loss computation.

Key Theoretical Concepts

Absorption Error (Definition 3.1): In IEEE 754 floating-point arithmetic, when adding two numbers $a$ and $b$ with $|a| \geq |b|$ , if $\frac{|b|}{|a|} < 2^{-(p-1)}$ where $p$ is the mantissa precision, the smaller value $b$ is absorbed. For float32 ( $p = 24$ ), this threshold is $2^{-23} \approx 1.19 \times 10^{-7}$ .

Softmax Collapse (Definition 3.2): In PyTorch's CE loss implementation using the Log-Sum-Exp trick:

Z = \log\left(\sum_k \exp(z_k)\right) = z_m + \log\left(\sum_k \exp(z_k - z_m)\right)

When the margin $z_m - \max_{k \neq m} z_k > (p-1)\ln 2 \approx 16$ for float32, the second term vanishes due to absorption error, giving $Z = z_m$ . The gradient for the correct class becomes exactly zero:

g_r = \hat{y}_r - y_r = e^{z_r - Z} - 1 = 0

Neural Collapse (Definition 3.3): The terminal-phase geometry where class means form a Simplex Equiangular Tight Frame (ETF), and classifier weights align with centered class means.

Methodology

Theoretical Framework

The authors formalize NFI through three key theoretical results:

Theorem 3.4 (Zero-Sum Breaking): Under SC, the expected update to the global classifier weight mean $W_G = \frac{1}{K}\sum_{k=1}^K W_k$ on a class-balanced batch is:

\mathbb{E}_{\mathcal{B}}[\Delta \boldsymbol{W}_G] = -\frac{\eta \epsilon}{K} \boldsymbol{\mu}_G

where $\epsilon = \mathbb{E}[\sum_{k \neq r} \hat{y}_k]$ is the residual probability mass on incorrect classes. The proof shows that while ideal gradients satisfy $\sum_k \nabla_{W_k}\mathcal{L} = 0$ , SC breaks this constraint because the correct-class gradient vanishes.

Proposition 3.6: Under NC conditions and SC, the gradient of loss with respect to features contains a component parallel to $W_G$ :

\text{Proj}_{W_G}(\nabla_h \mathcal{L}) = \epsilon W_G

Theorem 3.7 (NFI): The mutual reinforcement between weight drift and feature drift creates exponential growth:

\lim_{t \to \infty} \|W_G^{(t)}\| \propto \left(1 + \frac{\eta \epsilon}{\sqrt{K}}\right)^t

\lim_{t \to \infty} \|\mu_G^{(t)}\| \propto \left(1 + \frac{\eta \epsilon}{\sqrt{K}}\right)^t

with $\lim_{t \to \infty} \cos(W_G^{(t)}, \mu_G^{(t)}) \to -1$ (anti-parallel alignment).

Experimental Setup

Modular Arithmetic: 2-layer decoder-only Transformer and 6-layer MLP on modular division with prime $p=97$
Image Classification: CIFAR-10 with 6-layer MLP, VGG11, ResNet18, and 12-layer ViT
Language Modeling: nanoGPT (110M parameters) on FineWeb dataset
All experiments use CE loss, zero weight decay, and Adam optimizer

Empirical Validation / Results

Mechanistic Evidence

Gradient Re-emergence and Loss Spikes: The authors validate their theoretical predictions by computing the expected Adam update magnitude at spike onset. With $\eta = 10^{-3}$ , $\beta_1 = 0.9$ , $\beta_2 = 0.95$ , and pre-spike gradient $\approx 3 \times 10^{-9}$ :

First moment: $m_t = 0.9 \times 3 \times 10^{-9} + 0.1 \times 1.19 \times 10^{-7} = 1.46 \times 10^{-8}$
Second moment: $\sqrt{v_t} = \sqrt{9 \times 10^{-18} \times 0.95 + 1.19^2 \times 10^{-14} \times 0.05} = 2.7 \times 10^{-8}$
Adam update: $\eta \frac{m_t}{\sqrt{v_t} + \varepsilon} = 4 \times 10^{-4}$

This matches empirical observations: update magnitudes at spike epochs are ~50× larger than normal, with bimodal distribution around $\pm 4 \times 10^{-4}$ .

Architectural Dependence: All tested models exhibit Slingshot except ResNet18. This is consistent with Theorem 3.7: NFI requires $\epsilon$ to decay slowly relative to parameter dynamics. ResNet18's fast learning causes $\epsilon(t) \propto 1/t$ , leading to only polynomial (not exponential) growth.

Key Results Table

Architecture	Slingshot?	$\epsilon$ Behavior	NFI Growth
MLP	Yes	Slow decay/stagnation	Exponential
VGG11	Yes	Slow decay	Exponential
ViT	Yes	Slow decay	Exponential
ResNet18	No	Fast decay ( $\propto 1/t$ )	Polynomial

Mitigation Strategies

Mixed Precision: Casting logits to float64 during loss computation eliminates Slingshot (threshold becomes $2^{-52} \approx 2.22 \times 10^{-16}$ )
Zero-Sum Projection: Enforcing $\sum_k g_k = 0$ on gradients eliminates spikes
Adam's $\varepsilon$ : Increasing from $10^{-8}$ to $10^{-5}$ prevents spikes by lowering the maximum effective learning rate
BatchNorm: Effective when applied immediately before the classifier (removes $\mu_G$ drift)
LayerNorm: Ineffective—accelerates spikes by constraining feature norm
Label Smoothing: Eliminates precision-induced spikes but introduces EOS-type instabilities (finite Hessian eigenvalues)

Real-World Applications

Mini-batch Training: With batch size 256, ~50% of samples collapse after $10^6$ steps. While no visible Slingshot occurs, NFI still drives late-stage parameter growth. Removing $W_G$ or applying BatchNorm slows this growth.

Large Language Models: In GPT training, ~4000 of $1.3 \times 10^5$ tokens per step have exactly zero loss. High-precision computation (float64) increases logit growth (498 vs 183 after $10^5$ steps) due to Zipfian token distribution creating a large inherent $W_G$ that reinforces feature growth in the same direction—contrasting with NFI's anti-parallel interaction.

Theoretical and Practical Implications

Theoretical Significance

Reveals a fundamental gap between gradient-flow analysis and real finite-precision training
Challenges the assumption that CE loss with Adam is immune to loss spikes (contradicting Ma et al. [34])
Extends the understanding of Neural Collapse to include numerical precision effects
Provides the first causal link between Softmax Collapse and Slingshot Mechanism

Practical Implications

Training Stability: Finite-precision loss computation should be treated as a first-order factor in long-term training stability analysis
Architecture Design: Placement of normalization layers matters critically—BatchNorm before classifier prevents NFI, while LayerNorm can exacerbate it
Hyperparameter Selection: Adam's $\varepsilon$ serves as a practical control for effective learning rate amplification
Mixed Precision Training: Loss computation precision should be decoupled from parameter storage precision

Limitations

Analysis assumes the Unconstrained Feature Model—may oversimplify shallow network dynamics
Focuses on penultimate layer features and last-layer classifier interactions
May not fully capture dynamics in shallow networks where class means are insufficient descriptors

Conclusion

The paper successfully demystifies the Slingshot Mechanism as a floating-point arithmetic artifact rather than an intrinsic optimization phenomenon. The identified Numerical Feature Inflation mechanism provides a unified explanation for abnormal parameter growth, logit divergence, and loss spikes across diverse architectures and tasks. The work bridges theory and practice by offering actionable interventions (zero-sum projection, precision control, BatchNorm placement) that stabilize training. Future work should extend this analysis to shallow networks and explore the interplay between frequency-induced and precision-induced feature-embedding alignment in large language models.