Summary (Overview)

• Extends CompleteP framework to spectral optimizers: The paper demonstrates that the CompleteP desiderata (stable initialization, stable feature updates) hold for SCION and UNCONSTRAINED SCION, establishing depth-wise learning rate transfer for constrained spectral optimization methods.
• Confirms $L^{-\alpha}$ residual scaling: Under residual scaling with $\alpha \in [1/2, 1]$ and learning rate $\gamma \propto L^{\alpha-1}$, both SCION variants achieve depth-wise hyperparameter transfer, with $\alpha = 1$ providing complete feature learning.
• Proves width-wise transfer for SCION: The paper extends prior width-transfer results (Pethick et al., 2025b) to the constrained setting, showing the optimal learning rate is independent of hidden-layer width.
• Empirical validation across architectures: Experiments on nanoGPT (FineWeb), DeiT (ImageNet), and MLP-Mixer (CIFAR-10) confirm depth-wise and width-wise transfer with feature learning.
• Introduces PROJECTED SCION: A spectral clipping variant that admits width-transfer with theoretical stability guarantees, though a matching lower bound remains open.

Introduction and Theoretical Foundation

Background and Motivation

Deep neural networks have driven significant performance gains across domains including medicine, aeronautics, and nuclear engineering. Performance scales with model size, but so does the cost of training and hyperparameter tuning. The Maximal Update Parametrization ($\mu P$) framework (Yang & Hu, 2021) provides techniques ensuring optimal hyperparameters remain invariant as model width increases, enabling practitioners to tune small models and transfer hyperparameters to larger ones (Figure 1).

However, two gaps exist:

1. $\mu P$ focuses on width, not depth scaling
2. Existing analyses cover coordinate-wise optimizers (SGD, Adam), not spectral-norm-based methods

Spectral Optimizers

Spectral-norm-based optimizers like SCION (Pethick et al., 2025b) and Muon (Jordan et al., 2024b) have reemerged as strong alternatives. Unlike SGD or Adam, these methods:

• Precondition updates to weight matrices in a layer-wise manner using matrix-level operations (e.g., matrix sign)
• Use Linear Minimization Oracles (LMOs) over norm-ball constraints
• Have fundamentally different update structures, making standard $\mu P$ analysis inapplicable

CompleteP Framework

CompleteP (Dey et al., 2026) identifies three desiderata for joint width-and-depth-wise transfer:

1. Stable Initialization: Hidden layers and output remain stable at initialization as $L$ and $d_\ell$ grow
2. Stable Feature Updates: Each residual block contributes with order $1/L$ to feature movements
3. Non-lazy Training: Features evolve non-linearly (avoiding the Neural Tangent Kernel regime)

The residual architecture is parametrized as:

$$h^{(\ell)} = Z^{(\ell)}h^{(\ell-1)} + L^{-\alpha}\mathcal{F}_\ell(h^{(\ell-1)}; \theta_\mathcal{F}^{(\ell)})$$

where $\alpha \in [1/2, 1]$ controls residual branch scaling.

Methodology

Optimization Framework

The training objective is:

$$\min_{\boldsymbol{\Theta} \in \mathcal{X}} \mathcal{L}(\boldsymbol{\Theta})$$

where $\mathcal{X}$ is either unconstrained or constrained to:

$$\mathcal{D} = \prod_{\ell=0}^{L+1}\{\theta^{(\ell)}: \|\theta^{(\ell,k)}\| \leq \rho^{(\ell,k)}, \forall k\}$$

SCION Update Rules

SCION (constrained) uses the Stochastic Conditional Gradient (SCG) update:

$$\Theta_{t+1} \leftarrow (1 - \gamma_t)\Theta_t + \gamma_t \text{lmo}(M_t)$$

UNCONSTRAINED SCION uses:

$$\Theta_{t+1} \leftarrow \Theta_t + \gamma_t \text{lmo}(M_t)$$

The LMO minimizes a linear approximation under norm-ball constraints:

$$\text{lmo}(\nabla_{\theta^{(\ell)}}\mathcal{L}) \in \arg\min_{\nu \in \mathcal{D}} \langle \nabla_{\theta^{(\ell)}}\mathcal{L}, \nu\rangle$$

Table 1: Update rules for Sign → Spectral → Sign configuration

Key Scaling Rules

For depth-wise transfer, the learning rates must scale as:

$$\gamma = \gamma_{\text{base}}L^{\alpha-1} \quad \text{and} \quad \gamma_Z = \gamma_{Z_{\text{base}}}L^{-1}$$

Embedding and un-embedding learning rates need no depth scaling:

$$\gamma_{W^{(0)}} = \gamma_{W^{(0)}\text{base}} \quad \text{and} \quad \gamma_{W^{(L+1)}} = \gamma_{W^{(L+1)}\text{base}}$$

Modeling Assumptions

The analysis relies on several key assumptions:

• Weight initialization: $\|W^{(\ell)}\|_{\mathcal{S}_\infty} = \Theta(\sqrt{d_\ell/d_{\ell-1}})$
• Activation scaling: $\|\phi(h^{(\ell)})\|_2 = \Theta(\|h^{(\ell)}\|_2)$
• Alignment: $\|W^{(\ell)}\phi(h^{(\ell-1)})\|_2^2 = \Theta(\|W^{(\ell)}\|_{\mathcal{S}_\infty}^2\|\phi(h^{(\ell-1)})\|_2^2)$
• Non-cancellation: Updates and current quantities do not perfectly cancel

Empirical Validation / Results

Depth-wise Transfer Results

NanoGPT on FineWeb (Figures 7-8): Both SCION and UNCONSTRAINED SCION with $\alpha = 1$ and $\alpha = 0.5$ exhibit depth-aligned loss curves, confirming depth-wise hyperparameter transfer.

MLP-Mixer on CIFAR-10 (Figures 9-10): Minimizers are largely aligned across depths, with small deviations falling within flat regions.

DeiT on ImageNet (Figure 11): Achieved 82.3% top-1 accuracy in 200 epochs, matching prior results. Optimal base learning rate is invariant with respect to depth.

Complete Feature Learning

The paper validates that $\alpha = 1$ is the only value yielding complete feature learning:

• Figure 5: Best loss per learning rate shows $\alpha = 1$ provides the best performance across depths
• Figure 6: Relative distance from linearization shows $\alpha = 1$ achieves non-lazy regime, while $\alpha = 1/2$ approaches linearization at rate $1/\sqrt{L}$

The non-laziness criterion is measured as:

$$\frac{|\Delta_\theta h^{(\ell)} - \Delta_\theta h^{\text{lin},\theta}|}{|\Delta_\theta h^{\text{lin},\theta}|}$$

Width-wise Transfer Results

Proposition 3 establishes that both SCION and UNCONSTRAINED SCION satisfy Desideratum 3 (Feature Learning), with feature updates stable as width grows:

$$\|h^{(\ell)}\|_2^2 = \Theta(d_\ell) \quad \text{and} \quad \|\Delta h^{(\ell)}\|_2^2 = \Theta(d_\ell)$$

Figure 12 empirically confirms width transfer for SCION, UNCONSTRAINED SCION, and PROJECTED SCION on shallow nanoGPT.

PROJECTED SCION

Proposition 4 establishes width-wise stability through spectral clipping:

$$\mathrm{spectral\_clip}_{[a,b]}(W) = U \cdot \text{clip}_{[a,b]}(\Sigma) \cdot V^\top$$

The feature updates satisfy $\|\Delta h^{(\ell)}\|_2^2 = \mathcal{O}(d_\ell)$, though a matching lower bound remains open.

Radius Transfer

Preliminary empirical evidence (Figures 13-14) suggests the optimal ratio of radii $\eta = \rho_{\text{embedding}}/\rho_{\text{hidden}}$ transfers across depth and width, enabling a practical protocol:

1. Apply scaling rules with parametrized radii
2. Sweep on small model to find $(\gamma^*, a^*)$
3. Deploy to large model directly

Theoretical and Practical Implications

Theoretical Contributions

1. Extension of CompleteP to spectral optimizers: The paper provides new proofs tailored to SCION's LMO-based update dynamics, which differ fundamentally from coordinate-wise analyses in existing CompleteP formulations.

2. Depth-wise transfer: Confirms the $1/L$ residual scaling rule applies to constrained spectral optimizers, with $\alpha = 1$ enabling complete feature learning.

3. Width-wise transfer: Extends prior results to the constrained setting, proving learning rate independence from hidden-layer width for SCION with weight decay.

Practical Implications

• Reduced tuning costs: Practitioners can tune small models and transfer hyperparameters to larger ones, bypassing expensive tuning at scale.
• Layer-wise scaling guide: The paper provides concrete guidance for scaling learning rates per layer type (Table 1).
• Broader applicability: Results extend to Muon and other spectral optimizers, with implications for industrial-scale LLM training (Kimi, DeepSeek, Trinity series).

Conclusion

Main Takeaways

The paper establishes that SCION and UNCONSTRAINED SCION satisfy all CompleteP desiderata under $L^{-\alpha}$ residual scaling with $\alpha \in [1/2, 1]$ and learning rate $\gamma \propto L^{\alpha-1}$. The setting $\alpha = 1$ yields the strongest performance and best transfer, consistent with CompleteP findings for Adam-based optimizers.

Future Directions

1. Theoretical investigation of radius transfer: Rigorous proof of the empirically observed radius transfer property
2. Adversarial robustness: Radius transfer has implications for robustness against adversarial attacks and Lipschitz continuity control
3. Alternative weight decay parametrizations: Extending beyond the Frank-Wolfe step formulation
4. PROJECTED SCION lower bound: Establishing the matching lower bound for general spectral clipping to complete the $\Theta(d_\ell)$ guarantee