# When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective

> SignSGD outperforms SGD via noise-induced preconditioning that transforms the Hessian spectrum, yielding faster data scaling in hard tasks and larger critical batch sizes in easy tasks.

- **Source:** [arXiv](https://arxiv.org/abs/2611.16560)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/946Woc
- **Whiteboard:** https://picx.dev/p/946Woc/image

## Summary

# Summary of "When and How SignSGD Outperforms SGD: A Functional Scaling Law Perspective"

## Summary (Overview)

- **Establishes a Functional Scaling Law (FSL) for SignSGD** in power-law kernel regression with diagonal features, characterizing learning dynamics through two processes: signal learning (decay of initial error) and noise forgetting (persistence of injected noise).

- **Identifies noise-induced preconditioning as the key mechanism**: gradient noise smooths the sign operator, inducing an effective preconditioned SGD that transforms the effective Hessian from $\mathbf{H}$ to $\mathbf{H}^{1/2}$ when noise is curvature-aligned.

- **Derives regime-dependent advantages over SGD**: SignSGD achieves improved data efficiency in the hard-task regime ($\gamma_{\mathrm{sgn}} > \gamma_{\mathrm{sgd}}$) and improved step efficiency in the easy-task regime through a larger critical batch size.

- **Validates predictions empirically** in both power-law kernel regression and LLM pre-training (0.1B parameter model), showing the SignSGD FSL better captures batch-size dependence than the original SGD FSL.

- **Provides rigorous theoretical guarantees** including optimal data-scaling rates, critical batch size characterizations, and explicit formulas for signal-learning and noise-forgetting functions.

## Introduction and Theoretical Foundation

### Background and Motivation
Sign-based optimizers (SignSGD, Lion, Muon) play an increasingly important role in large-scale training, yet existing theoretical analyses focus primarily on deterministic full-batch dynamics or convergence guarantees. The interaction between the sign operator and gradient noise—critical for understanding scaling behavior—remains poorly understood.

### Problem Setup
The paper studies **power-law kernel (PLK) regression** with:
- Labels generated by $y = \langle \phi(\mathbf{x}), \boldsymbol{\theta}^{*} \rangle + \epsilon$ with label noise $\epsilon \sim \mathcal{N}(0, \sigma^2)$
- Gaussian features $\phi(\mathbf{x}) \sim \mathcal{N}(0, \mathbf{H})$ with diagonal covariance
- **Assumption 2.3 (Power-law structures)**: $\lambda_j \approx j^{-q}$, $\lambda_j |\theta_j^*|^2 \approx j^{-(1+sq)}$

### Key Assumptions
- **Assumption 2.1**: Constant label noise ($\sigma \sim 1$)
- **Assumption 2.2**: Diagonal features ($\mathbf{H} = \text{diag}(\lambda_1, \cdots, \lambda_d)$)
- **Assumption 2.5 (Curvature-aligned noise)**: $\Sigma \sim \mathbf{H}$, meaning noise variance along each direction scales with curvature

### Theoretical Foundation
The paper extends the **functional scaling law framework** of Li et al. (2025) to SignSGD. The one-pass SignSGD update is:

$$\mathbf{g}_k = \frac{1}{B} \sum_{i=1}^{B} \nabla_{\boldsymbol{\theta}} \ell(\boldsymbol{\theta}_k; \mathbf{x}_{k,i}, y_{k,i}), \qquad \boldsymbol{\theta}_{k+1} = \boldsymbol{\theta}_k - \eta_k \operatorname{sgn}(\mathbf{g}_k)$$

## Methodology

### SDE Modeling
The paper develops an **intrinsic-time SDE** for SignSGD:

$$\mathrm{d}\boldsymbol{\vartheta}_t = -\mathbf{m}_B(\nabla\mathcal{R}(\boldsymbol{\vartheta}_t)) \mathrm{d}t + \sqrt{n_B(t)} \mathbf{Q}_B^{1/2}(\boldsymbol{\vartheta}_t) \mathrm{d}\mathbf{W}_t$$

where:
- **Intrinsic time**: $t = \Gamma_B(z) := \int_0^z \frac{\eta(u)}{\sigma_B}\mathrm{d}u$ (with $\sigma_B = 1/\sqrt{B}$)
- **Noise intensity**: $n_B(t) := \sigma_B \eta(\Gamma_B^{-1}(t))$
- **Smoothed sign**: $\mathbf{m}_B(\mathbf{x}) := \sigma_B \mathbb{E}_{\boldsymbol{\xi} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Sigma})}[\mathrm{sgn}(\mathbf{x} + \sigma_B \boldsymbol{\xi})]$

### Key Structural Insight: Two-Phase Dynamics
The smoothed sign exhibits two limiting responses (Equation 8):

$$m_B(x) \approx \begin{cases} c x/a, & |x|/(\sigma_B a) \ll 1 \quad \text{(linear/local phase)} \\ \sigma_B \mathrm{sgn}(x), & |x|/(\sigma_B a) \gg 1 \quad \text{(saturated/burn-in phase)} \end{cases}$$

This leads to the effective preconditioned dynamics (Equation 10):

$$\mathrm{d}\boldsymbol{\delta}_t \approx -\mathbf{PH}\boldsymbol{\delta}_t \mathrm{d}t + \sqrt{n_B(t)} \mathrm{d}\mathbf{W}_t, \qquad \mathbf{P} = c\mathbf{D}^{-1/2}$$

Under noise-curvature alignment and diagonal features, $\mathbf{PH} = \text{diag}(\rho_j)$ with $\rho_j \approx \sqrt{\lambda_j}$, transforming the Hessian spectrum from $j^{-q}$ to $j^{-q/2}$.

## Empirical Validation / Results

### Main Theoretical Results

**Theorem 3.1 (FSL for SignSGD)**: For $t \gtrsim B^{\max\{1/2, 1/(2s+2/q)\}}$,

$$\mathbb{E}[\mathcal{E}(\boldsymbol{\vartheta}_t)] \asymp S(t) + \int_0^t K(t-\tau) n_B(\tau) \mathrm{d}\tau$$

with $S(t) \asymp t^{-2s}$ and $K(t) \asymp t^{-(2-2/q)}$.

**Comparison with SGD**:

| Optimizer | Intrinsic time | Noise intensity | $S(t)$ | $K(t)$ |
|-----------|---------------|-----------------|--------|---------|
| SGD | $\int_{0}^{z} \eta(\tau) \, \mathrm{d}\tau$ | $1/B$ | $t^{-s}$ | $t^{-(2-1/q)}$ |
| SignSGD | $\sqrt{B} \int_{0}^{z} \eta(\tau) \, \mathrm{d}\tau$ | $1/\sqrt{B}$ | $t^{-2s}$ | $t^{-(2-2/q)}$ |

**Theorem 4.1 (Data scaling)**: Optimal data-scaling rates partition the $(s,q)$ space:

$$(\gamma_{\mathrm{sgd}}, \gamma_{\mathrm{sgn}}) = \begin{cases} (\frac{sq}{sq+1}, \frac{sq}{sq+1}), & s \geqslant 1 - \frac{1}{q} \\ (s, \frac{sq}{sq+1}), & \frac{1}{2} - \frac{1}{q} \leqslant s < 1 - \frac{1}{q} \\ (s, 2s), & 0 < s < \frac{1}{2} - \frac{1}{q} \end{cases}$$

**Theorem 4.4 (Critical batch size)**:

$$B_{\text{crit}}^{\text{sgd}}(D) \simeq \max\left\{1, D^{1 - \frac{1}{\alpha}}\right\}, \quad B_{\text{crit}}^{\text{sgn}}(D) \simeq \max\left\{1, D^{1 - \frac{1}{2\alpha}}\right\}$$

where $\alpha = s + \frac{1}{q}$.

### Experimental Results

**PLK Regression**:
- FSL accurately captures risk dynamics under cosine, WSD, and cyclic LR schedules (Figure 1, left)
- Signal terms collapse onto a common curve when plotted against intrinsic time (Figure 2)
- SignSGD achieves faster data-scaling rate in hard-task regime ($\gamma_{\mathrm{sgn}} > \gamma_{\mathrm{sgd}}$)
- SignSGD has the largest CBS in easy-task regime, followed by Adam and SGD

**LLM Pre-training** (0.1B parameters, 10B tokens):
- FSL accurately predicts held-out WSD schedule loss after fitting to cosine and 8-1-1 schedules
- Batch-size extrapolation: fitting to $B \in \{64, 128\}$ accurately predicts $B = 256$ loss curves
- SignSGD FSL better captures batch-size dependence than original SGD FSL

**Structural Conditions Validation** (Figure 4):
- SignSGD advantage requires both diagonal features and curvature-aligned noise
- Rotating features (breaking diagonality) or using isotropic noise (breaking alignment) reduces the advantage
- Advantage scales with alignment parameter $\gamma$ in $\Sigma \propto \mathbf{H}^{\gamma}$

## Theoretical and Practical Implications

### Mechanism: Noise-Induced Preconditioning
The paper reveals a fundamental mechanism: **gradient noise is not merely a nuisance but actively shapes the optimization landscape**. The sign operator, when smoothed by noise, induces a preconditioner that:
- Normalizes gradients by per-coordinate noise scales: $P = cD^{-1/2}$
- Under curvature-aligned noise, transforms the effective Hessian from $\mathbf{H}$ to $\mathbf{H}^{1/2}$
- Accelerates signal learning in low-curvature directions

### Scaling Law Implications
- **Data efficiency**: In hard tasks ($s < 1 - 1/q$), SignSGD achieves faster data-scaling rates, meaning it requires fewer samples for the same risk
- **Step efficiency**: In easy tasks, SignSGD supports larger batch sizes (CBS grows as $D^{1/(2\alpha)}$ vs $D^{1/\alpha}$ for SGD), requiring $D^{1/(2\alpha)}$ fewer sequential updates

### Practical Guidance
- Sign-based optimizers (SignSGD, Lion, Muon) are particularly advantageous when:
  - Gradient noise is curvature-aligned (common in deep learning)
  - Features are approximately diagonalized
  - Large batch sizes are needed (SignSGD maintains efficiency at larger $B$)
- The square-root scaling rule $\eta_{\text{peak}} \propto \sqrt{B}$ preserves small-batch risk for SignSGD (vs. linear rule for SGD)

### Connection to Minimax Rates
The optimal rate for SignSGD when $s \geqslant \frac{1}{2} - \frac{1}{q}$ matches the classical minimax rate $D^{-sq/(sq+1)}$ (Caponnetto & De Vito, 2007), showing SignSGD is statistically optimal in this regime.

## Conclusion

### Key Takeaways
1. **SignSGD's advantage is task-dependent**: Data efficiency improves in hard tasks, step efficiency improves in easy tasks, with no advantage in super-hard tasks.

2. **The mechanism is noise-induced preconditioning**: The sign operator, smoothed by gradient noise, induces an effective preconditioned SGD with square-root spectral transformation.

3. **Structural conditions matter**: Diagonal features and noise-curvature alignment are essential for the advantage; without them, SignSGD loses its edge.

4. **The FSL is practically useful**: It accurately captures loss dynamics across LR schedules and batch sizes in both kernel regression and LLM pre-training.

### Future Directions
- **Extensions to RMSprop and Adam**: The same FSL form and batch-size dependence are predicted, with supporting numerical evidence, but rigorous analysis requires accounting for momentum and evolving second-moment estimates.

- **From coordinate-wise sign to matrix sign**: Extending to the matrix sign operator (Muon) would clarify how matrix structure and gradient noise jointly determine effective preconditioning, signal learning, and batch-size dependence.

The work establishes a principled framework for understanding when and how sign-based optimizers outperform SGD, with implications for choosing optimizers in large-scale training based on task difficulty and noise structure.

---

_Markdown view of https://picx.dev/p/946Woc, served by PicX — AI-generated visual whiteboard summaries of research papers._