Summary of "When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective"

Summary (Overview)

• Establishes a Functional Scaling Law (FSL) for SignSGD in power-law kernel regression with diagonal features, characterizing learning dynamics through two processes: signal learning (decay of initial error) and noise forgetting (persistence of injected noise).

• Identifies noise-induced preconditioning as the key mechanism: gradient noise smooths the sign operator, inducing an effective preconditioned SGD that transforms the effective Hessian from $\mathbf{H}$ to $\mathbf{H}^{1/2}$ when noise is curvature-aligned.

• Derives regime-dependent advantages over SGD: SignSGD achieves improved data efficiency in the hard-task regime ($\gamma_{\mathrm{sgn}} > \gamma_{\mathrm{sgd}}$) and improved step efficiency in the easy-task regime through a larger critical batch size.

• Validates predictions empirically in both power-law kernel regression and LLM pre-training (0.1B parameter model), showing the SignSGD FSL better captures batch-size dependence than the original SGD FSL.

• Provides rigorous theoretical guarantees including optimal data-scaling rates, critical batch size characterizations, and explicit formulas for signal-learning and noise-forgetting functions.

Introduction and Theoretical Foundation

Background and Motivation

Sign-based optimizers (SignSGD, Lion, Muon) play an increasingly important role in large-scale training, yet existing theoretical analyses focus primarily on deterministic full-batch dynamics or convergence guarantees. The interaction between the sign operator and gradient noise—critical for understanding scaling behavior—remains poorly understood.

Problem Setup

The paper studies power-law kernel (PLK) regression with:

• Labels generated by $y = \langle \phi(\mathbf{x}), \boldsymbol{\theta}^{*} \rangle + \epsilon$ with label noise $\epsilon \sim \mathcal{N}(0, \sigma^2)$
• Gaussian features $\phi(\mathbf{x}) \sim \mathcal{N}(0, \mathbf{H})$ with diagonal covariance
• Assumption 2.3 (Power-law structures): $\lambda_j \approx j^{-q}$, $\lambda_j |\theta_j^*|^2 \approx j^{-(1+sq)}$

Key Assumptions

• Assumption 2.1: Constant label noise ($\sigma \sim 1$)
• Assumption 2.2: Diagonal features ($\mathbf{H} = \text{diag}(\lambda_1, \cdots, \lambda_d)$)
• Assumption 2.5 (Curvature-aligned noise): $\Sigma \sim \mathbf{H}$, meaning noise variance along each direction scales with curvature

Theoretical Foundation

The paper extends the functional scaling law framework of Li et al. (2025) to SignSGD. The one-pass SignSGD update is:

$$\mathbf{g}_k = \frac{1}{B} \sum_{i=1}^{B} \nabla_{\boldsymbol{\theta}} \ell(\boldsymbol{\theta}_k; \mathbf{x}_{k,i}, y_{k,i}), \qquad \boldsymbol{\theta}_{k+1} = \boldsymbol{\theta}_k - \eta_k \operatorname{sgn}(\mathbf{g}_k)$$

Methodology

SDE Modeling

The paper develops an intrinsic-time SDE for SignSGD:

$$\mathrm{d}\boldsymbol{\vartheta}_t = -\mathbf{m}_B(\nabla\mathcal{R}(\boldsymbol{\vartheta}_t)) \mathrm{d}t + \sqrt{n_B(t)} \mathbf{Q}_B^{1/2}(\boldsymbol{\vartheta}_t) \mathrm{d}\mathbf{W}_t$$

where:

• Intrinsic time: $t = \Gamma_B(z) := \int_0^z \frac{\eta(u)}{\sigma_B}\mathrm{d}u$ (with $\sigma_B = 1/\sqrt{B}$)
• Noise intensity: $n_B(t) := \sigma_B \eta(\Gamma_B^{-1}(t))$
• Smoothed sign: $\mathbf{m}_B(\mathbf{x}) := \sigma_B \mathbb{E}_{\boldsymbol{\xi} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Sigma})}[\mathrm{sgn}(\mathbf{x} + \sigma_B \boldsymbol{\xi})]$

Key Structural Insight: Two-Phase Dynamics

The smoothed sign exhibits two limiting responses (Equation 8):

$$m_B(x) \approx \begin{cases} c x/a, & |x|/(\sigma_B a) \ll 1 \quad \text{(linear/local phase)} \\ \sigma_B \mathrm{sgn}(x), & |x|/(\sigma_B a) \gg 1 \quad \text{(saturated/burn-in phase)} \end{cases}$$

This leads to the effective preconditioned dynamics (Equation 10):

$$\mathrm{d}\boldsymbol{\delta}_t \approx -\mathbf{PH}\boldsymbol{\delta}_t \mathrm{d}t + \sqrt{n_B(t)} \mathrm{d}\mathbf{W}_t, \qquad \mathbf{P} = c\mathbf{D}^{-1/2}$$

Under noise-curvature alignment and diagonal features, $\mathbf{PH} = \text{diag}(\rho_j)$ with $\rho_j \approx \sqrt{\lambda_j}$, transforming the Hessian spectrum from $j^{-q}$ to $j^{-q/2}$.

Empirical Validation / Results

Main Theoretical Results

Theorem 3.1 (FSL for SignSGD): For $t \gtrsim B^{\max\{1/2, 1/(2s+2/q)\}}$,

$$\mathbb{E}[\mathcal{E}(\boldsymbol{\vartheta}_t)] \asymp S(t) + \int_0^t K(t-\tau) n_B(\tau) \mathrm{d}\tau$$

with $S(t) \asymp t^{-2s}$ and $K(t) \asymp t^{-(2-2/q)}$.

Comparison with SGD:

Theorem 4.1 (Data scaling): Optimal data-scaling rates partition the $(s,q)$ space:

$$(\gamma_{\mathrm{sgd}}, \gamma_{\mathrm{sgn}}) = \begin{cases} (\frac{sq}{sq+1}, \frac{sq}{sq+1}), & s \geqslant 1 - \frac{1}{q} \\ (s, \frac{sq}{sq+1}), & \frac{1}{2} - \frac{1}{q} \leqslant s < 1 - \frac{1}{q} \\ (s, 2s), & 0 < s < \frac{1}{2} - \frac{1}{q} \end{cases}$$

Theorem 4.4 (Critical batch size):

$$B_{\text{crit}}^{\text{sgd}}(D) \simeq \max\left\{1, D^{1 - \frac{1}{\alpha}}\right\}, \quad B_{\text{crit}}^{\text{sgn}}(D) \simeq \max\left\{1, D^{1 - \frac{1}{2\alpha}}\right\}$$

where $\alpha = s + \frac{1}{q}$.

Experimental Results

PLK Regression:

• FSL accurately captures risk dynamics under cosine, WSD, and cyclic LR schedules (Figure 1, left)
• Signal terms collapse onto a common curve when plotted against intrinsic time (Figure 2)
• SignSGD achieves faster data-scaling rate in hard-task regime ($\gamma_{\mathrm{sgn}} > \gamma_{\mathrm{sgd}}$)
• SignSGD has the largest CBS in easy-task regime, followed by Adam and SGD

LLM Pre-training (0.1B parameters, 10B tokens):

• FSL accurately predicts held-out WSD schedule loss after fitting to cosine and 8-1-1 schedules
• Batch-size extrapolation: fitting to $B \in \{64, 128\}$ accurately predicts $B = 256$ loss curves
• SignSGD FSL better captures batch-size dependence than original SGD FSL

Structural Conditions Validation (Figure 4):

• SignSGD advantage requires both diagonal features and curvature-aligned noise
• Rotating features (breaking diagonality) or using isotropic noise (breaking alignment) reduces the advantage
• Advantage scales with alignment parameter $\gamma$ in $\Sigma \propto \mathbf{H}^{\gamma}$

Theoretical and Practical Implications

Mechanism: Noise-Induced Preconditioning

The paper reveals a fundamental mechanism: gradient noise is not merely a nuisance but actively shapes the optimization landscape. The sign operator, when smoothed by noise, induces a preconditioner that:

• Normalizes gradients by per-coordinate noise scales: $P = cD^{-1/2}$
• Under curvature-aligned noise, transforms the effective Hessian from $\mathbf{H}$ to $\mathbf{H}^{1/2}$
• Accelerates signal learning in low-curvature directions

Scaling Law Implications

• Data efficiency: In hard tasks ($s < 1 - 1/q$), SignSGD achieves faster data-scaling rates, meaning it requires fewer samples for the same risk
• Step efficiency: In easy tasks, SignSGD supports larger batch sizes (CBS grows as $D^{1/(2\alpha)}$ vs $D^{1/\alpha}$ for SGD), requiring $D^{1/(2\alpha)}$ fewer sequential updates

Practical Guidance

• Sign-based optimizers (SignSGD, Lion, Muon) are particularly advantageous when:
  • Gradient noise is curvature-aligned (common in deep learning)
  • Features are approximately diagonalized
  • Large batch sizes are needed (SignSGD maintains efficiency at larger $B$)
• The square-root scaling rule $\eta_{\text{peak}} \propto \sqrt{B}$ preserves small-batch risk for SignSGD (vs. linear rule for SGD)

Connection to Minimax Rates

The optimal rate for SignSGD when $s \geqslant \frac{1}{2} - \frac{1}{q}$ matches the classical minimax rate $D^{-sq/(sq+1)}$ (Caponnetto & De Vito, 2007), showing SignSGD is statistically optimal in this regime.

Conclusion

Key Takeaways

1. SignSGD's advantage is task-dependent: Data efficiency improves in hard tasks, step efficiency improves in easy tasks, with no advantage in super-hard tasks.

2. The mechanism is noise-induced preconditioning: The sign operator, smoothed by gradient noise, induces an effective preconditioned SGD with square-root spectral transformation.

3. Structural conditions matter: Diagonal features and noise-curvature alignment are essential for the advantage; without them, SignSGD loses its edge.

4. The FSL is practically useful: It accurately captures loss dynamics across LR schedules and batch sizes in both kernel regression and LLM pre-training.

Future Directions

• Extensions to RMSprop and Adam: The same FSL form and batch-size dependence are predicted, with supporting numerical evidence, but rigorous analysis requires accounting for momentum and evolving second-moment estimates.

• From coordinate-wise sign to matrix sign: Extending to the matrix sign operator (Muon) would clarify how matrix structure and gradient noise jointly determine effective preconditioning, signal learning, and batch-size dependence.

The work establishes a principled framework for understanding when and how sign-based optimizers outperform SGD, with implications for choosing optimizers in large-scale training based on task difficulty and noise structure.