Summary of: "When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay"

Summary (Overview)

• Exact discrete-time law: The paper derives an exact recurrence for the effective directional stepsize $\Phi_t$ in scale-invariant optimization blocks: $\Phi_{t+1} = \frac{B_t \Phi_t}{1 + \Phi_t^2 \|\bar{g}_t\|^2}$, where $B_t$ is a single scalar capturing all schedule and weight-decay forcing, and the denominator represents geometric self-quenching from norm growth.

• Sharp contraction/expansion boundary: When $B_t \leq 1$, the effective stepsize contracts unconditionally; when $B_t > 1$, contraction requires sufficient self-quenching ($\Phi_t \|\bar{g}_t\| \geq \sqrt{B_t - 1}$). This boundary is shown to be the same surface governing the fixed-point structure of a fully solved normalized-linear model.

• Intrinsic instability: In an exactly solvable isotropic normalized-linear model, the dynamics reduce to a 2D map whose unique interior balance point is an unstable spiral source (eigenvalue modulus $\sqrt{1 + a - a^2} > 1$), with an exact period-two orbit on the sphere. This proves that recurrent behavior under constant learning rate with weight decay is a structural consequence of discrete-time geometry, not stochasticity.

• Unified optimizer framework: A homogeneous-optimizer classification reveals that SGD/SGDM have homogeneity degree $\nu = 1$ (quadratic self-quenching), while Adam has $\nu = 0$ (linear self-quenching), providing a first-principles explanation for why adaptive methods are systematically more expansion-prone under normalization.

• Causal control demonstrated: Directly enforcing $B_t \equiv B$ via synthesized learning-rate schedules produces a sharply peaked accuracy curve with optimal performance at $B = 1$; a $\pm 2\%$ perturbation causes a >20 point accuracy drop on CIFAR-10.

Introduction and Theoretical Foundation

Normalization layers (BatchNorm, LayerNorm, etc.) induce positive scale invariance in neural network parameter blocks: the loss depends only on the direction $w/\|w\|$, not the magnitude. This symmetry fundamentally alters optimization dynamics. Rather than directly controlling progress, the parameter norm interacts with the learning rate to determine an effective directional stepsize, typically scaling as $\eta / \|w\|^2$.

The central feedback loop: gradient updates tend to increase the norm (suppressing effective stepsize), while weight decay and learning-rate schedules counteract this (shrinking norm or amplifying nominal step size). Prior work has offered:

• Equilibrium/mixing views: effective rate stabilizes [16, 22, 23, 30]
• Periodic destabilization: under constant schedules [24]
• Regime-based analyses: from convergence to chaos on the sphere [14]

The unresolved question: What is the exact, per-step law governing this feedback loop under arbitrary schedules?

Key Theoretical Setup

For a scale-invariant block with SGD + coupled weight decay:

$$w_{t+1} = a_t w_t - \eta_t \nabla_w \mathcal{L}(w_t), \quad a_t := 1 - \eta_t \lambda_t > 0$$

Using polar decomposition $w_t = r_t u_t$, scale invariance implies the rescaled gradient $\bar{g}(u) = r \nabla_w \mathcal{L}(r u)$ is independent of $r$ and tangent to the sphere ($\langle u, \bar{g}(u) \rangle = 0$).

Methodology

Theorem 2.1 (Exact Schedule Law)

Defining the effective directional stepsize $\Phi_t := \eta_t / (a_t r_t^2)$ and schedule factor:

$$B_t := \frac{\eta_{t+1}}{\eta_t a_t a_{t+1}}$$

The dynamics decompose into:

1. Polar dynamics: $r_{t+1}^2 = r_t^2 a_t^2 (1 + \Phi_t^2 \|\bar{g}_t\|^2)$ and $u_{t+1} = \frac{u_t - \Phi_t \bar{g}_t}{\|u_t - \Phi_t \bar{g}_t\|}$
2. Exact recurrence: $\Phi_{t+1} = \frac{B_t \Phi_t}{1 + \Phi_t^2 \|\bar{g}_t\|^2}$
3. Contraction criterion: If $B_t \leq 1$, $\Phi_{t+1} \leq \Phi_t$ unconditionally; if $B_t > 1$, contraction iff $\Phi_t \|\bar{g}_t\| \geq \sqrt{B_t - 1}$

The identity is purely algebraic—no linearization, no continuous-time limit, no assumptions beyond scale invariance—and holds under arbitrary time-varying schedules and stochasticity.

Stochastic Extension (Theorem 2.3)

With $X_t := \Phi_t^2 \|\hat{g}_t\|^2$, $\beta_t := \log B_t$, $q_t := \mathbb{E}_t[\log(1 + X_t)]$:

• Exact conditional drift: $\mathbb{E}_t[\log \Phi_{t+1} - \log \Phi_t] = \beta_t - q_t$
• Martingale decomposition: $\log \frac{\Phi_T}{\Phi_0} = \sum_{t=0}^{T-1}(\beta_t - q_t) - \sum_{t=0}^{T-1} D_t$ where $D_t$ is a martingale-difference sequence
• Concentration bound (under sub-Gaussian increments): with probability $\geq 1 - \delta$,

$$\left| \log \frac{\Phi_T}{\Phi_0} - \sum_{t=0}^{T-1}(\beta_t - q_t) \right| \leq \sqrt{2 \log \frac{2}{\delta} \sum_{t=0}^{T-1} v_t}$$

Solved Model (Theorem 3.4)

For isotropic covariance ($\Sigma = I$) with constant schedule, the full high-dimensional dynamics collapse to a 2D map:

$$q_{t+1} = \frac{q_t + \Phi_t(1 - q_t^2)}{\sqrt{1 + \Phi_t^2(1 - q_t^2)}}, \quad \Phi_{t+1} = \frac{\Phi_t}{a^2(1 + \Phi_t^2(1 - q_t^2))}$$

Fixed point: $q_\star = \sqrt{\frac{1+a}{2}}$, $\Phi_\star = \frac{\sqrt{2(1+a)}}{a}$—which lies exactly on the switching surface $\Phi_t \|\bar{g}_t\| = \sqrt{B - 1}$.

Period-two orbit: $u_\pm = q_\star \beta \pm \sqrt{\frac{1-a}{2}} e$ for any unit $e \perp \beta$, satisfying $u_+ \mapsto u_- \mapsto u_+$.

Instability: The Jacobian has complex-conjugate eigenvalues with modulus $\sqrt{1 + a - a^2} > 1$, making the fixed point an unstable spiral source.

Homogeneous-Optimizer Framework (Theorem 4.1)

For updates $w_{t+1} = a_t w_t - \eta_t p_t$ where $p_t = r_t^{-\nu} \bar{p}_t$ for homogeneity degree $\nu \geq 0$:

Defining $\Psi_t := \eta_t / (a_t r_t^{1+\nu})$ and $\tilde{B}_t^{(\nu)} := \frac{\eta_{t+1}/\eta_t}{a_t^\nu a_{t+1}}$:

$$\Psi_{t+1} = \frac{\tilde{B}_t^{(\nu)} \Psi_t}{\|u_t - \Psi_t \bar{p}_t\|^{1+\nu}}$$

The denominator exponent $1 + \nu$ quantifies self-quenching strength: larger $\nu$ = stronger suppression.

• SGD: $\nu = 1$ (quadratic self-quenching)
• SGDM: $\nu = 1$ with one extra vector (momentum), introducing a radial amplification channel
• Adam: $\nu = 0$ (linear self-quenching), with preconditioning inducing nonzero radial components

Empirical Validation / Results

Exact-Map Validation

• Recurrence residual $< 2 \times 10^{-15}$ (float64 machine precision)
• Constant schedules ($B_t > 1$) sustain recurrence; step decay produces single-step contraction shocks; cosine decay sweeps $B_t$ smoothly across the boundary

Neural Network Validation

• BN MLP (MNIST) and BN ConvNet (CIFAR-10): expansion fraction closely tracks $B_t$ across schedules
• Recurrence residual $\sim 10^{-6}$ (float32 precision)

Target-$B_t$ Intervention (Causal Control)

Synthesizing schedules that enforce $B_t \equiv B$:

$$\eta_{t+1} = \frac{B \eta_t (1 - \lambda \eta_t)}{1 + \lambda B \eta_t (1 - \lambda \eta_t)}$$

Key result: Performance peaks sharply at $B = 1$; a $\pm 2\%$ perturbation causes >20 point accuracy drop on CIFAR-10.

Optimizer Mechanisms

Architecture Stress Test (LayerNorm Transformers)

• GPT2-small (4-block, $d_{model} = 256$) on WikiText and GPT2 (12-block, $d_{model} = 768$) on OpenWebText
• Median ratio residual: $1.19 \times 10^{-7}$ for SGD/SGDM
• Constant/step schedules sustain expansion fractions of 0.999–1.000; cosine drives them to $\sim 10^{-4}$
• Adam shows intermediate expansion (0.44–0.49) under constant/step, suppressed under cosine (0.033 on WikiText, 0.095 on OpenWebText)

Theoretical and Practical Implications

Theoretical Implications

1. Unification of existing observations: Equilibrium, oscillation, and regime-based perspectives are unified under a single exact law; the switching surface $\Phi_t \|\bar{g}_t\| = \sqrt{B - 1}$ reappears as both the local contraction boundary and the global fixed-point condition.

2. Discrete-time geometry is fundamental: The spiral-source instability shows that recurrent behavior under constant schedules is intrinsic to discrete-time scale-invariant optimization, not an artifact of stochasticity or continuous-time approximation.

3. Optimizer classification principle: The homogeneity degree $\nu$ provides a first-principles taxonomy of optimizers, explaining why adaptive methods (Adam, $\nu = 0$) are systematically more expansion-prone than gradient methods (SGD/SGDM, $\nu = 1$).

Practical Implications

1. $B_t$ as a control coordinate: Directly controlling $B_t$ enables precise schedule design; optimal performance occurs at the predicted boundary $B = 1$.

2. Operational diagnostics: The contraction condition $B_t \leq 1 + \Phi_t^2 \|\bar{g}_t\|^2$ can be evaluated after the backward pass and before the update, enabling real-time monitoring and intervention.

3. Schedule interpretation: Common schedules correspond to structured trajectories of $B_t$:

   • Constant LR + weight decay → sustained expansion pressure ($B_t > 1$)
   • Step decay → transient contraction shocks
   • Cosine decay → smooth sweep across the boundary

Conclusion

This paper establishes that scale-invariant optimization is governed by a single scalar quantity $B_t$ that captures all schedule and weight-decay forcing, balanced against geometric self-quenching. The exact recurrence holds to machine precision across dynamical systems, neural networks (MLP, CNN, GPT2), and optimizers (SGD, SGDM, Adam).

Key takeaways:

1. The effective stepsize in normalized networks is endogenous—it evolves according to an exact discrete-time law that separates schedule forcing from geometric response
2. The balance point between forcing and self-quenching is intrinsically unstable in discrete time, producing persistent recurrent behavior under constant schedules
3. Optimizer stability under normalization is determined by a single homogeneity exponent $\nu$: adaptive methods are structurally weaker in self-quenching
4. $B_t$ serves as both a diagnostic and a causal control coordinate for training dynamics

Future directions: Extension to decoupled weight decay (which lacks explicit closed-form solutions), and deeper analysis of the interaction between stochastic noise accumulation and the near-critical band near the switching surface.