Summary

• Spectral perspective on long-context extension: This paper identifies two essential spectral properties governing long-range information retrieval in Gated DeltaNet (GDN): a sufficiently broad slow spectral band aligned with the target dependency length, and the preservation of fast-decaying modes for state clearing and context switching.
• SpectralShift method: A spectral reparameterization approach that (1) reparameterizes alpha projections with an optimal scaling factor $s = (L_{\text{ref}}/L_{\text{tar}})^{0.5}$ to reshape the decay spectrum, and (2) applies learning-rate scaling to alpha projections to preserve the state spectrum during continual pretraining.
• Consistent improvements: Across multiple extension settings (8K→32K/64K/128K), SpectralShift consistently outperforms naive context extension baselines on long-context benchmarks (RULER, DROP, RACE) while maintaining comparable general capabilities.
• Broad applicability: The method is compatible with different positional encoding strategies (DroPE, YaRN, ABF) and extends effectively to pure linear attention models, not just hybrid architectures.
• Theoretical grounding: The paper provides formal proofs (Lemma 1, Theorems 1-2) establishing the optimal scaling factor and the relationship between learning-rate scaling and spectral preservation.

Introduction and Theoretical Foundation

Background

Modern LLMs require increasingly long context windows for applications like long-document understanding and retrieval-intensive reasoning. While full softmax attention provides direct token-to-token interactions, its quadratic computational complexity limits scalability. Linear attention architectures based on recurrent state updates offer a linear-complexity alternative by compressing historical information into fixed-size states.

The Core Challenge

Unlike full attention where historical tokens remain explicitly accessible, recurrent models must preserve useful information through a finite-dimensional state transition process. Long-context capability is fundamentally constrained by whether the recurrent state contains sufficient capacity to retain information over the required dependency distance. Existing context extension methods typically apply continued pretraining directly without modifying linear attention layers, overlooking the spectral properties of linear attention state dynamics.

Theoretical Foundation

The paper analyzes GDN through the lens of finite-time information decay. For positions $1 \leq i < j \leq L$ with distance $\ell = j - i$, the finite-time transition matrix is $\Phi_{i \rightarrow j} = \mathbf{A}_j \mathbf{A}_{j-1} \cdots \mathbf{A}_{i+1}$. The finite-time decay rate is defined as:

$$\gamma_r(i, j) = -\frac{1}{\ell} \log \sigma_r(\Phi_{i \rightarrow j}),$$

which decomposes into a head-wise component $\gamma_\alpha$ and a singular-value component $\gamma_{\mathbf{R}, r}$:

$$\gamma_r = \underbrace{-\frac{1}{\ell} \sum_{t=i+1}^{j} \log \alpha_t}_{\text{head-wise component } \gamma_\alpha} + \underbrace{-\frac{1}{\ell} \log \sigma_r(\mathbf{R}_{i \rightarrow j})}_{\text{singular-value component } \gamma_{\mathbf{R}, r}},$$

where $\mathbf{R}_{i \rightarrow j} = \prod_{t=i+1}^{j} (\mathbf{I} - \beta_t \mathbf{k}_t \mathbf{k}_t^\top)$.

The slow spectral band is defined as modes satisfying:

$$\mathcal{S}_\ell = \{r: \log \sigma_r(\Phi_{i \rightarrow j}) \geq -1\}.$$

The quantity $|\mathcal{S}_\ell|$ measures the width of the slow spectral band—the number of state directions whose effective timescales match the target dependency length.

Key Analytical Findings

1. Slow Propagation Capacity: Models with better long-context retrieval have more slow modes ($M_\ell^{\text{slow}}$) distributed across more layer-heads ($H_\ell^{\text{slow}}$).

2. Slow-Band Utilization: Retrieval performance depends on whether information is effectively written into slow-transition subspaces. The write alignment metric:

$$E_{\text{write}} = \frac{\|\mathbf{V}_{\mathcal{S}_\ell}^\top \mathbf{k}_i\|_2^2}{\|\mathbf{k}_i\|_2^2}$$

measures the proportion of the needle key contained in the slow input subspace. Higher $E_{\text{write}}$ correlates with better NIAH scores.

Methodology

Alpha Reparameterization

The alpha projection is decomposed into a shared magnitude component and an input-dependent deviation:

$$\mathbf{a}_t^{\text{ref}} = \mathbf{W}_\alpha^{\text{ref}} \mathbf{h}_t = \underbrace{\mu_\alpha^{\text{ref}} \mathbf{J} \mathbf{h}_t}_{\mathbf{c}_t} + \underbrace{(\mathbf{W}_\alpha^{\text{ref}} - \mu_\alpha^{\text{ref}} \mathbf{J}) \mathbf{h}_t}_{\boldsymbol{\xi}_t},$$

where $\mathbf{J}$ is the all-ones matrix. The reparameterization scheme is:

$$\mathbf{a}_t(s_1) = \mathbf{c}_t + s_1 \boldsymbol{\xi}_t, \quad 0 < s_1 \leq 1$$

Lemma 1 establishes that reducing $s_1$ induces opposite local forgetting adjustments based on the sign of $\xi_t$: positive deviations produce larger retention, while negative deviations produce smaller retention. The optimal scaling factor is derived as $s_1 = (L_{\text{ref}}/L_{\text{tar}})^{0.5}$.

Length-Scaled Continual Pretraining

Theorem 1 shows that the learning-rate scaling factor $s_2$ directly controls the spectral change shared by all modes within a head:

$$|\gamma_\alpha^{(u+1)} - \gamma_\alpha^{(u)}| \leq C_\alpha^{(u)} s_2 \eta_\alpha^{\text{ref}} + O(s_2^2 \eta_\alpha^{\text{ref}}).$$

For simplicity, $s_2 = s_1 = (L_{\text{ref}}/L_{\text{tar}})^{0.5}$.

SpectralShift Algorithm

Algorithm 1 SpectralShift
Require: Checkpoint Θ^ref, reference length L_ref, target length L_tar, learning rate η^tar
1: s ← (L_ref / L_tar)^(1/2)
// 1. Spectral-based initialization
2: if L_ref < L_tar then
3:   Extract {W_α^ref, Θ_others^ref}
4:   μ_α^ref ← Mean(W_α^ref)
5:   W_α* ← μ_α^ref J + s(W_α^ref - μ_α^ref J)
6:   Θ^tar* ← {W_α*, Θ_others^ref}
7: end if
8: Set η_α^tar ← s · η^tar
// 2. Target-length continual pretraining
9: for each training step do
10:   Execute forward and backward
11:   Update W_α* with η_α^tar
12:   Update Θ_others^ref with η^tar
13: end for
return Θ^tar

Theorem 2 characterizes the joint spectral effect: any mode satisfying $D_r \geq \gamma_r^{\text{ref}} - 1/\ell$ belongs to the final slow spectral band, where $D_r = \delta_r^{\text{init}} + \delta^{\text{CPT},\alpha} + \delta_r^{\text{CPT},\Delta}$.

Empirical Validation / Results

Main Results (Table 3)

The model is a 1.5B-A0.6B GDN-MoE pretrained on Dolma3 corpus with 8K context, then extended to 32K/64K/128K.

Key results for 128K extension (two-stage 10B+10B curriculum):

• SpectralShift: General Avg 54.8, RULER-128K: 44.6
• Baseline: General Avg 53.7, RULER-128K: 43.1

For 128K extension (20B curriculum):

• SpectralShift: General Avg 54.0, RULER-128K: 42.2
• Baseline: General Avg 52.9, RULER-128K: 41.9

Ablation Study (Table 4)

The best configuration (ID ⑤: α reparameterization + α LR scaling + factor s^0.5) achieves:

• General Avg: 54.3, RULER-128K: 42.9
• Compared to baseline (ID ①): General Avg 54.0, RULER-128K: 42.8

Removing either component degrades performance. Full scaling (s) shows degraded general capability.

Positional Encoding Compatibility (Table 5)

SpectralShift improves long-context performance across DroPE, YaRN, and ABF positional encodings, with consistent improvements at maximum evaluation lengths.

Pure GDN Model (Table 6)

Improvements become increasingly pronounced as target context length grows.

Theoretical and Practical Implications

Theoretical Contributions

1. Spectral characterization of long-context retrieval: The paper provides a rigorous framework connecting the spectral properties of GDN's transition matrix to long-range information retrieval capability, introducing the slow spectral band concept.
2. Optimal scaling derivation: The theoretical derivation of $k = 0.5$ in the power-law family $s_k = (L_{\text{ref}}/L_{\text{tar}})^k$ provides a principled basis for spectral initialization.
3. Training dynamics analysis: Theorems 1 and 2 establish formal relationships between learning-rate scaling and spectral preservation during continual pretraining.

Practical Implications

1. Efficient context extension: SpectralShift requires no architectural changes, only initialization and learning-rate adjustments, making it a drop-in improvement for existing GDN-based models.
2. Broad compatibility: Works with various positional encodings and pure linear attention models, suggesting applicability to modern hybrid architectures.
3. Balanced capability preservation: Unlike aggressive spectral scaling, the moderate $s^{0.5}$ factor preserves general capabilities while improving long-context performance.

Conclusion

This work addresses the context window extension problem for Gated DeltaNet from a spectral dynamics perspective. The key insight is that effective long-context modeling requires both expanding slow modes to match target dependency lengths and preserving fast-decaying modes for state clearing. SpectralShift achieves this through alpha projection reparameterization with optimal scaling and learning-rate adjustment during continual pretraining.

The method consistently improves long-context performance across various extension settings, positional encodings, and model architectures, while maintaining comparable general capabilities. The spectral insights and method developed here have potential for broad application to context extension in modern hybrid architectures, facilitating more capable long-context foundation models.

Limitations: The method is validated only on a 1.5B MoE model; validation on larger-scale models (e.g., Qwen3.5 scale) is left for future work due to computational constraints.