Summary (Overview)

• This paper presents the first systematic study of massive activations (MAs) in layer-interleaved hybrid linear attention (HLA) large language models, uncovering two novel architecture-aligned morphologies: pre-attention spikes (PAS) and inter-spike plateaus (ISP).
• MAs consistently spike immediately before full attention layers (PAS) and can persist through intervening linear attention layers (ISP), with the transition between these morphologies governed by the hybridization ratio $\rho = L/L_{FA}$.
• The findings are validated across five linear attention architectures (RetNet, HGRN, GLA, DeltaNet, GDN), six hybridization configurations, five data domains, and 12 large-scale open-source models spanning 1.2B to 397B parameters.
• Controlled pretraining of GDN-based hybrids up to 1.3B parameters shows that both morphologies emerge early during optimization and respond asymmetrically to output gating: full attention output gating strongly attenuates magnitudes without eliminating layerwise organization, while removing GDN gates has modest effects.
• A systematic-outlier lifecycle account unified by "cancellation timing" explains both morphologies: PAS follows a localized write–sink–cancel process, while ISP reflects delayed cancellation; at the full attention limit, this recovers the stable MA morphology of full attention LLMs.

---

Introduction and Theoretical Foundation

Background: The Hybrid Linear Attention (HLA) Paradigm

Softmax attention enables expressive content-dependent interactions but incurs quadratic computational costs with context length. Linear attention reduces these costs via efficient recurrent computation but has limited modeling capacity, particularly for recall-intensive tasks. Hybrid linear attention (HLA) LLMs address this trade-off by interleaving linear attention layers with full attention layers, combining recurrent efficiency with attention's modeling capacity.

Layer-interleaved hybrid architectures have become increasingly common in modern LLMs, including Qwen3-Next, Qwen3.5, Kimi Linear, Kimi K3, Zamba, and Nemotron-H.

Massive Activations (MAs) as a Lens

Massive activations (MAs) are a sparse set of hidden-state entries that exceed typical activation values by several orders of magnitude and concentrate at specific token positions (Sun et al., 2024). Prior work attributes their emergence and coupling with attention sinks to structural properties of softmax-based Transformers. However, their behavior in HLA LLMs was previously unexplored, raising the central research question:

How does hybridization shape MA dynamics in HLA LLMs, and what do these dynamics reveal about their internal computation?

Mathematical Foundations

An HLA LLM comprises $L$ pre-normalized residual blocks. For input hidden states $\mathbf{X}^{(0)} \in \mathbb{R}^{T \times \bar{d}}$, the $\ell$-th block computes:

$$\mathbf{H}^{(\ell)} = \mathbf{X}^{(\ell-1)} + \operatorname{Mixer}^{(\ell)}\left(\operatorname{Norm}_{\operatorname{mix}}^{(\ell)}\left(\mathbf{X}^{(\ell-1)}\right)\right)$$ $$\mathbf{X}^{(\ell)} = \mathbf{H}^{(\ell)} + \mathrm{FFN}^{(\ell)}\left(\mathrm{Norm}_{\mathrm{ffn}}^{(\ell)}\left(\mathbf{H}^{(\ell)}\right)\right)$$

where $\operatorname{Mixer}^{(\ell)}$ is full attention for $\ell \in \mathcal{I}_{FA}$ and linear attention otherwise. The hybridization ratio is defined as:

$$\rho = L / L_{FA}$$

where $L_{FA} = |\mathcal{I}_{FA}|$ is the number of full attention layers. A $\rho{:}1$ configuration contains one full attention layer per $\rho$ sequence-mixing layers; $\rho = 1$ recovers a full attention model.

Full attention computes (omitting multi-head notation):

$$[\mathbf{Q}, \mathbf{K}, \mathbf{V}] = \mathbf{X}[\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V], \qquad \mathrm{FA}(\mathbf{X}) = \mathrm{softmax}\left(\frac{\mathbf{QK}^\top}{\sqrt{d_h}} + \mathbf{M}\right)\mathbf{V}\mathbf{W}_O$$

Linear attention replaces pairwise interactions with a fixed-size recurrent state:

$$[\mathbf{q}_t, \mathbf{k}_t, \mathbf{v}_t] = \mathbf{x}_t[\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V], \quad \mathbf{S}_t = \mathbf{F}_t\mathbf{S}_{t-1} + \mathbf{k}_t^\top\mathbf{v}_t, \quad \mathbf{y}_t = \mathbf{q}_t\mathbf{S}_t\mathbf{W}_O$$

---

Methodology

Key Methodological Challenge

The authors identify a critical fragility in prior MA detection methods when applied to HLA LLMs: magnitude ranking loses alignment with attention sinks. In full attention LLMs, magnitude ranking works as a proxy for tracing MAs; in HLA models, sink tokens exhibit lower and less stable magnitude rankings across depth, and the maximally activated token switches more frequently between adjacent layers.

Attention-Sink-Guided Tracking

To address this, the authors develop a sink-conditioned activation tracing procedure:

1. Identify the dominant consensus sink token for each input $x$:

$$t_x^{\star} = \arg\max_{1 \leq t < T} \frac{1}{|\mathcal{I}_{FA}|H|\mathcal{Q}_t|} \sum_{\ell \in \mathcal{I}_{FA}} \sum_{h=1}^{H} \sum_{q \in \mathcal{Q}_t} A_{x,q,t}^{(\ell,h)}, \qquad \mathcal{Q}_t = \{q: q > t\}$$

2. Trace the maximum absolute hidden-state activation of this fixed token across depth:

$$m_{x,t_x^{\star}}^{(\ell)} = \|\mathbf{X}_{x,t_x^{\star},:}^{(\ell)}\|_\infty = \max_j |X_{x,t_x^{\star},j}^{(\ell)}|$$

This preserves token identity while allowing the maximally activated feature to evolve across depth.

Quantitative Metrics

Sink–spike alignment rate (measures PAS localization):

$$\mathrm{Align}(\mathcal{D}) = \frac{1}{|\mathcal{D}||\mathcal{I}_{FA}|} \sum_{x \in \mathcal{D}} \sum_{f \in \mathcal{I}_{FA}} \mathbf{1}\left[f-1 \in \arg\max_{\ell \in \mathcal{B}_f} m_{x,t_x^*}^{(\ell)}\right]$$

Inter-spike retention score (measures ISP persistence):

$$\operatorname{ISR}(\mathcal{D}) = \frac{1}{|\mathcal{D}|} \sum_{x \in \mathcal{D}} \frac{1}{K-1} \sum_{i=1}^{K-1} \frac{1}{|\mathcal{J}_i|} \sum_{\ell \in \mathcal{J}_i} \min\left(1, \frac{m_{x,t_x^{\star}}^{(\ell)}}{\min\left(m_{x,t_x^{\star}}^{(p_i)}, m_{x,t_x^{\star}}^{(p_{i+1})}\right)}\right)$$

Experimental Setup

• M-A-P Model Suite: Five linear attention architectures (RetNet, HGRN, GLA, DeltaNet, GDN) at 340M and 1.3B scales, multiple hybridization ratios, unified pretraining recipe.
• Evaluation domains: WikiText-103 (general prose), Scientific Papers, GSM8K (mathematical reasoning), CodeSearchNet (Python code), FLORES-200 (multilingual).
• Large-scale models: Kimi Linear, Qwen3.5, Nemotron-H, Zamba2 (1.2B–397B parameters).
• Controlled pretraining: 24-layer GDN models trained from scratch at 340M and 1.3B scales using Flash Linear Attention framework, with targeted interventions (full attention placement, output gating).

---

Empirical Validation / Results

Pre-Attention Spikes (PAS) Across Architectures

Across all five linear attention backbones under a fixed 12:1 hybridization ratio, sink-associated activations develop pronounced local maxima immediately before full attention layers. The PAS morphology recurs across all evaluation domains.

Table 1: Sink–spike alignment rates in M-A-P HLA models (%) (1.3B/340M, macro-average)

Inter-Spike Plateaus (ISP) Across Hybridization Ratios

As full attention becomes denser (decreasing $\rho$), intervening activations remain progressively more elevated, bridging successive PAS into sustained inter-spike plateaus (ISP). The onset varies by architecture: ISP emerges at 6:1 in DeltaNet but primarily at 3:1 in GDN.

Table 2: Inter-spike retention scores in M-A-P HLA models (%) (1.3B/340M, macro-average)

Large-Scale Pretrained Models

Evaluation of 12 checkpoints from Kimi Linear, Qwen3.5, Nemotron-H, and Zamba2 yields four key findings:

1. Consistency across post-training stages: Base and Instruct checkpoints retain aligned PAS locations and ISP boundaries.
2. Recurrence across model scales: PAS and ISP remain aligned with full attention placement despite magnitude variation.
3. Recurrence across sequence mixers: Both linear attention AND state-space hybrids exhibit PAS and ISP.
4. Consistency across domains: PAS locations and ISP spans remain stable while magnitudes vary.

Controlled Pretraining Results

• PAS emerges early: Visible after just 1B training tokens, becoming progressively more pronounced and stable.
• Full attention output gating strongly attenuates PAS magnitudes but does not eliminate the layerwise organization.
• Removing GDN output gates produces only moderate increases in PAS magnitude.
• The asymmetry indicates full attention plays a central role in organizing MA dynamics, while GDN gating mainly modulates propagation.

---

Theoretical and Practical Implications

The Systematic-Outlier Lifecycle Account

The authors propose a unified mechanism-level account distinguishing the two morphologies through cancellation timing:

1. PAS (localized write–sink–cancel): A pre-attention layer writes a large outlier into the residual stream; during full attention, the token acts as an attention sink receiving disproportionate attention; a subsequent opposite-signed update at the same coordinate substantially cancels the outlier.

2. ISP (delayed cancellation): A large outlier is written before full attention but remains prominent across intervening linear attention layers, followed by a later opposite-signed update as the plateau dissipates.

3. Full attention limit: As full attention becomes denser, increasing persistence connects successive PAS through ISP, consistent with progressively deferred cancellation. At the limit, this recovers the stable MA morphology of full attention LLMs.

This framework establishes a continuum of increasing outlier persistence: PAS → ISP → persistent full attention MAs.

Practical Implications

• Architecture-aware analysis: The finding that magnitude ranking loses alignment with attention sinks in HLA models necessitates sink-conditioned tracing for reliable MA identification in hybrid architectures.
• Gating as a design lever: Full attention output gating is a powerful mechanism for attenuating MA magnitudes, though it cannot eliminate the underlying layerwise organization—relevant for quantization and optimization of hybrid models.
• Interpretability tool: MAs serve as an informative probe for understanding hybrid attention computation and the division of labor between full and linear attention layers.

---

Conclusion

This paper presents the first systematic study of massive activations in layer-interleaved HLA LLMs, uncovering two architecture-aligned morphologies:

• Pre-attention spikes (PAS): MAs consistently spike immediately before full attention layers.
• Inter-spike plateaus (ISP): MAs persist through intervening linear attention layers as full attention becomes denser.

Key takeaways:

1. Both morphologies recur across architectures, hybridization configurations, model scales, and input domains.
2. They emerge early during pretraining and are attenuated, not eliminated, by full attention output gating.
3. A shared systematic-outlier lifecycle distinguished by cancellation timing unifies both morphologies: prompt cancellation produces PAS, delayed cancellation sustains ISP, and the full attention limit recovers stable MA morphology.

Future directions include determining what regulates cancellation timing and whether transient PAS and persistent ISP support distinct computational roles in hybrid models.