Summary (Overview)

  • CAST (Clipping with Adaptive Scales and Thresholds) is introduced as a history-adaptive extension of Soft Spectral Clipping (SSC) for LLM pretraining, addressing the limitation that SSC's effectiveness depends on gradient scale and clipping threshold.
  • CAST uses two complementary histories: Historical Scaling (HS) leverages AdamW's previous second moment to set row and column scales for each gradient matrix, while Output Feedback uses an EMA of the clipped output's estimated spectral norm to adapt each matrix's clipping threshold.
  • In a mixture-of-experts (MoE) model with 19.08B total parameters trained for 83.89B tokens, CAST achieves lower training and mean validation losses than nine gradient-processing baselines.
  • CAST records the fewest relative gradient spikes (1 vs. 6–422 for baselines) and reaches the best baseline's final smoothed training loss in 7.2% fewer steps.
  • The method incurs only ~2% per-step overhead compared to simpler baselines, with an estimated 5.5% total time saving.

Introduction and Theoretical Foundation

Gradient clipping controls large gradients during language model pretraining. When applied before AdamW, it also changes the moment estimates used by later updates, meaning the clipping rule affects both the current update and the optimizer's state over time.

Key limitation of existing approaches: Global norm clipping rescales the whole gradient by a single factor, which limits the influence of unusually large gradients but preserves direction. However, a matrix gradient can be large mainly along a few directions—rescaling the entire matrix weakens other directions even when their magnitudes are moderate.

Spectral clipping controls gradient magnitude in each singular direction. SPECTRA (Jiang et al., 2026) develops a smooth spectral transformation applicable before or after a base optimizer. The soft spectral clipping (SSC) map for a singular value σ≥0\sigma \geq 0 with threshold τ>0\tau > 0 is:

ϕτ(σ)=σ1+(σ/τ)2.(1)\phi_{\tau}(\sigma) = \frac{\sigma}{\sqrt{1 + (\sigma / \tau)^{2}}}.\tag{1}

For matrix B∈Rq×d\boldsymbol{B} \in \mathbb{R}^{q \times d} with q≤dq \leq d, its matrix form is:

Hτ(B)=τ(τ2I+BB⊤)−1/2B.(2)H_{\tau}(B) = \tau (\tau^{2} I + B B^{\top})^{-1/2} B.\tag{2}

The core insight: A clipping threshold is meaningful only relative to the gradient scale. Persistent differences in row and column magnitudes affect the spectrum on which clipping acts. An unusually large value in one part of a matrix may be routine in another, and the overall scale changes over training. A fixed threshold applies different clipping strengths as the gradient scale changes. CAST responds to these changes using each matrix's history.

Methodology

Algorithm 1: Soft Spectral Clipping (SSC)

The SSC routine bounds the largest squared singular value using the Gram matrix. If the bound is at or below the squared threshold, it returns the input unchanged. Otherwise, it approximates HτH_{\tau} with K Newton–Schulz iterations. A Boolean flag aa records whether clipping was applied.

Algorithm 2: CAST Gradient Preprocessing

CAST makes two changes around SSC:

1. Historical Scaling (HS): Using the bias-corrected previous second moment V^=V/(1−β2k)\widehat{V} = V/(1-\beta_2^k), the row, column, and overall means are:

ri=1n∑jV^ij,cj=1m∑iV^ij,b0=1mn∑i,jV^ij.(4)\begin{array}{l} r_{i} = \frac{1}{n} \sum_{j} \widehat{V}_{ij}, \quad c_{j} = \frac{1}{m} \sum_{i} \widehat{V}_{ij}, \\ b_{0} = \frac{1}{mn} \sum_{i,j} \widehat{V}_{ij}. \end{array}\tag{4}

With ρ=0.01\rho = 0.01 and ϵs=10−8\epsilon_s = 10^{-8}, the scale is defined as:

b=max⁡{b0,(ϵs/ρ)2},Sij=max⁡(ri,ρb)max⁡(cj,ρb)b,Z=G⊘S.(5)\begin{array}{c} b = \max\{b_0, (\epsilon_s / \rho)^2\}, \\ S_{ij} = \sqrt{\frac{\max(r_i, \rho b) \max(c_j, \rho b)}{b}}, \\ Z = G \oslash S. \end{array}\tag{5}

After clipping, the original coordinates are restored with Gout=S⊙ZoutG_{\text{out}} = S \odot Z_{\text{out}}.

2. Adaptive Thresholds: Each matrix maintains a positive scalar history hh of its clipped output's estimated spectral norm:

τt=λht−1,λ=1.3.(6)\tau_t = \lambda h_{t-1}, \qquad \lambda = 1.3.\tag{6}

The history is updated via:

st=PowerTop(Zout,t,3),ht=βhht−1+(1−βh)st,βh=0.95.(7)\begin{array}{l} s_t = \text{PowerTop}(Z_{\text{out},t}, 3), \\ h_t = \beta_h h_{t-1} + (1 - \beta_h) s_t, \quad \beta_h = 0.95. \end{array}\tag{7}

Implementation details:

  • First 40 optimizer updates use global clipping at norm 1 to initialize moments/histories
  • Selected matrices: two-dimensional tensors in the dense optimizer group (excluding embeddings/output matrices)
  • Other tensors (MoE experts, embeddings) use an output-norm controller with threshold λf\lambda f
  • Distributed implementation all-gathers gradient fragments, applies transformation, writes local optimizer slices
  • Complexity: O(max⁡(m,n)q2+Kq3+pmn)O(\max(m,n)q^2 + Kq^3 + pmn) with K=5K=5, p=3p=3

Empirical Validation / Results

Experimental Setup

  • Model: Shortcut MoE with 10 blocks, hidden size 1536, 16 attention heads; 19.08B total parameters, 647.6M activated backbone
  • Training: Sequence length 8192, global batch size 2048, 5,000 steps (83.89B nominal tokens), BF16, expert parallelism of 8
  • Optimizer: AdamW with (β1,β2)=(0.9,0.95)(\beta_1, \beta_2) = (0.9, 0.95), ϵ=10−16\epsilon = 10^{-16}, weight decay 0.1, learning rate 8×10−48 \times 10^{-4}

Pretraining Results (Table 1)

MethodTrain loss ↓Validation loss ↓CAST lower
CAST1.65161.8738—
GlobalGC1.67181.896219/20
AdaGC1.66251.887220/20
SSC-constant1.66381.886420/20
SSC-cosine1.67001.894719/20
GradientStabilizer1.68281.903418/20
GradPower1.68231.913920/20
ClipByValue1.67891.902220/20
AGC1.66411.885619/20
ZClip1.67111.893719/20

CAST's mean validation loss was 1.26×10−21.26 \times 10^{-2} below SSC-constant and 2.10×10−22.10 \times 10^{-2} below SSC-cosine.

Ablation Study (Table 2)

ConditionTrain lossValidation lossGate fraction
CAST1.65161.87380.884
w/o HS1.65411.87470.403
w/ τ = 0.11.69371.92151.000
w/ cosine τ1.67671.90341.000

Both components contribute: removing HS increases losses, and fixed/cosine thresholds perform significantly worse than the adaptive spectral EMA.

Training Stability (Table 3)

MethodMedian normP99 normMax normSteps with norm > 1Relative spikes
CAST0.15100.37020.547601
AdaGC0.13430.34851.054116
SSC-constant0.15910.47231.1829122
SSC-cosine0.16530.49961.1922221
GradPower0.11420.69874.914425422
ZClip0.15401.2761918.78625275

CAST combines the fewest relative spikes with the lowest mean validation loss.

Training Efficiency (Table 4)

MeasureCASTSSC-constantClipByValue
Mean (s)44.40843.53343.580
Tokens/s/device590360226015

CAST takes 2.0% longer per step than SSC-constant but reaches AdaGC's final loss at step 4,639 (7.2% fewer steps), yielding an estimated 5.5% total time saving.

Theoretical and Practical Implications

Theoretical contributions:

  • CAST demonstrates that gradient clipping thresholds should be history-adaptive rather than fixed or schedule-based, responding to each matrix's actual scale changes during training
  • The use of AdamW's second moment for coordinate standardization provides a principled link between the optimizer's state and the clipping geometry
  • Output feedback via spectral norm EMA creates a self-regulating mechanism where the threshold follows the actual post-clipping spectrum

Practical implications:

  • CAST achieves better quality with fewer gradient spikes, suggesting improved training stability that could reduce the need for intervention
  • The 7.2% step reduction translates to meaningful wall-clock savings even with the ~2% per-step overhead
  • The method is applicable to large MoE models (19.08B parameters) with expert parallelism, demonstrating scalability
  • The component design (HS + output feedback) allows partial adoption for different use cases

Conclusion

CAST uses two complementary histories to guide spectral clipping in pretraining: AdamW's previous second moment sets the gradient scale, and an EMA of the clipped output's estimated spectral norm sets each matrix's threshold. In 5,000-step MoE pretraining, CAST achieved lower training and mean validation losses than nine baselines and three component variants, with the fewest relative gradient spikes. CAST reached AdaGC's final smoothed loss in 7.2% fewer steps, with an estimated 5.5% time saving including per-step overhead.

Limitations: Full-matrix computation and communication times depend on tensor shape; reported timings reflect the specific cluster environment. Changes in hardware or training settings may alter the exact speedup, though the shared experimental setup supports fair comparison.

Related papers