Summary (Overview)

• The paper proposes two novel intervention-based metrics—RoPE Frequency Importance Score (RFIS) and RoPE Positional Dependence (RPD)—to establish a complete head-level taxonomy of retrieval and positional heads in modern RoPE-based Transformers.
• The authors identify a Global Positional Band (GPBand), a salient mid-low-frequency band that separates retrieval from positional functions and follows the training-length positional scale, potentially explaining zero-shot length extrapolation failure.
• Two hybrid architecture design principles are derived: (i) positional modeling should operate only locally, while global access uses position-independent retrieval; (ii) retrieval and positional functions should be assigned at head granularity with layer-specific allocation.
• These principles are instantiated in the Head-wise Hybrid Architecture (HwH) using NoPE Full Attention (FA) for global retrieval and Gated DeltaNet (GDN) linear attention (LA) for local positional modeling, with FA-to-LA ratio less than 1:3.
• From-scratch pretraining at 380M and 1.4B scales shows HwH retains strong language modeling, improves retrieval, and substantially strengthens zero-shot long-context extrapolation over Transformer, pure-LA, and layer-wise hybrid baselines.

---

Introduction and Theoretical Foundation

Background and Motivation

Hybrid architectures combining Full Attention (FA) and Linear Attention (LA) are increasingly prominent, yet their allocation remains largely heuristic. FA provides exact token-to-token access but incurs quadratic sequence complexity and a growing KV cache, while fixed-state LA offers recurrent efficiency but struggles with exact long-range retrieval. The central design question remains: which roles should be assigned to FA or LA, at what granularity, and with what allocation strategy?

Theoretical Foundation

The functional differentiation learned by modern RoPE-based Transformers offers a natural design reference. Prior work identified retrieval heads supporting long-range access, but these methods begin with pretrained Transformers and do not independently characterize the complementary function.

Full Attention with causal masking:

$$O = \mathrm{softmax} \left(\frac{QK^{\top}}{\sqrt{d_h}} + M\right) V \tag{1}$$

Linear Attention (Gated DeltaNet) with data-dependent decay and delta-rule update:

$$S_t = S_{t-1}\left(\alpha_t(I - \beta_t k_t k_t^{\top})\right) + \beta_t v_t k_t^{\top}, \qquad o_t = S_t q_t \tag{3}$$

Rotary Position Embedding (RoPE) partitions each head into $F = d_h/2$ two-dimensional groups with frequencies:

$$\omega_r = \theta^{-2r/d_h}, \quad R(\varphi) = \begin{bmatrix} \cos\varphi & -\sin\varphi \\ \sin\varphi & \cos\varphi \end{bmatrix} \tag{4}$$

The dot product decomposes into exact frequency contributions:

$$\tilde{q}_t^{\top}\tilde{k}_s = \sum_{r=0}^{F-1} (q_t^{(r)})^{\top} R((s-t)\omega_r) k_s^{(r)} \tag{5}$$

---

Methodology

Behavioral Probes

The authors use QRscore to measure retrieval tendency and introduce LDscore to measure local-streaming tendency. However, behavioral evidence alone cannot classify all heads mechanistically, motivating intervention-based metrics.

RoPE Frequency Importance Score (RFIS)

RFIS removes one frequency contribution from a head's logits while holding others fixed, measuring the change in the complete attention distribution via Jensen-Shannon divergence:

$$I_{\ell,h,i,r}^{\mathrm{RFIS}} = \frac{D_{\mathrm{JS}}\left(\mathbf{p}_{\ell,h,i}, \mathbf{p}_{\ell,h,i}^{(-r)}\right)}{\ln 2}, \qquad \mathrm{RFIS}_{\ell,h,r} = \mathbb{E}_{x\sim\mathcal{D}} \mathbb{E}_{i\sim\mathcal{Q}(x)} \left[ I_{\ell,h,i,r}^{\mathrm{RFIS}} \right] \tag{10}$$

where the removal intervention is:

$$\mathbf{z}_{\ell,h,i}^{(-r)} = \mathbf{z}_{\ell,h,i} - \mathbf{c}_{\ell,h,i,r}^{\mathrm{RoPE}}, \qquad \mathbf{p}_{\ell,h,i}^{(-r)} = \mathrm{softmax}\left(\mathbf{z}_{\ell,h,i}^{(-r)}\right) \tag{8}$$

RoPE Positional Dependence (RPD)

RPD replaces the relative rotation of one frequency with the identity while preserving learned Q/K coordinates:

$$c_{\ell,h,i,j,r}^{\mathrm{NoPE}} = \mathbf{q}_{\ell,h,i,r}^{\top} \mathbf{k}_{\ell,g(h),j,r} \tag{11}$$

The intervention:

$$\mathbf{z}_{\ell,h,i}^{(r \to \mathrm{NoPE})} = \mathbf{z}_{\ell,h,i} - \mathbf{c}_{\ell,h,i,r}^{\mathrm{RoPE}} + \mathbf{c}_{\ell,h,i,r}^{\mathrm{NoPE}} \tag{12}$$

Both metrics are faithful intervention-based metrics bounded in [0, 1].

Head-wise Hybrid Architecture (HwH)

HwH instantiates the two principles as architectural inductive bias:

• NoPE FA heads provide position-independent global retrieval
• LA (GDN) heads provide local positional modeling

The standard configuration (HwH-std) follows the observed layer-specific distribution:

• First and last layers: only GDN
• Middle half: NoPE-FA:GDN = 1:3
• Remaining shallow and deep layers: 1:7

---

Empirical Validation / Results

Language Modeling

At 380M parameters, HwH-std achieves the best perplexity (27.41 on Wiki, 34.84 on LAMBADA) and highest average commonsense accuracy (42.78%). At 1.4B, HwH-std remains competitive with the best baselines.

Table 1: Language modeling perplexity (↓) and commonsense-reasoning accuracy (%, ↑)

Retrieval and Length Extrapolation

Table 2: Real-world retrieval and RULER NIAH at the 1.4B scale

Key findings:

• Transformer scores zero on all NIAH tasks at 4K (no zero-shot length extrapolation)
• HwH-std extrapolates nearly perfectly through 4K (twice the training length) and achieves strongest performance beyond 4K
• HwH-std achieves the best overall retrieval performance

Ablations

Function-specific components:

• GDN is stronger than SWA in commonsense reasoning; SWA slightly improves retrieval and substantially improves extrapolation
• Replacing NoPE FA with RoPE FA sharply degrades retrieval and extrapolation, confirming NoPE FA's advantage for position-independent global retrieval

Layer-specific allocation:

• Removing NoPE FA from first/last layers markedly improves extrapolation through 4K
• Reducing outer-layer ratio from 1:3 to 1:7 further improves extrapolation beyond 4K and strengthens retrieval

---

Theoretical and Practical Implications

Theoretical Contributions

1. Complete mechanism-level taxonomy: RFIS and RPD establish a complete two-type taxonomy of retrieval and positional heads in RoPE Transformers, moving beyond behavioral probes that cannot fully classify heads.

2. GPBand as functional boundary: The training-length-related salient mid-low-frequency band marks the mechanism-level boundary between retrieval and positional dependence. This global positional fitting is identified as a potential cause of extrapolation failure.

3. RoPE Transformers as implicit functional hybrids: Modern Transformers naturally learn to separate retrieval and positional functions across heads, providing evidence-grounded design principles for hybrid architectures.

Practical Implications

1. Principled hybrid design: The two principles offer a systematic alternative to heuristic FA/LA allocation in hybrid architectures.

2. Improved length extrapolation: HwH achieves zero-shot extrapolation to twice the training length, addressing a critical limitation of standard Transformers.

3. Efficiency: With FA-to-LA ratio less than 1:3, HwH maintains strong performance while reducing the quadratic attention cost.

4. Component role clarity: The work clarifies that LA (GDN) should handle local positional modeling while NoPE FA should handle global retrieval, informing future component selection.

---

Conclusion

The paper establishes a complete head-level taxonomy of retrieval and positional modeling in modern RoPE-based Transformers, identifying the training-length-related GPBand as its functional boundary. The derived hybrid-design principles—local positional modeling with position-independent global retrieval, and head-granularity with layer-specific allocation—are validated through from-scratch HwH experiments.

Future directions include:

• Extending analyses to function-aligned GQA
• Other modalities and larger scales
• Improved components
• Hybrid designs beyond multi-head Transformers

The work demonstrates that principled hybrid architecture design, grounded in mechanistic understanding of functional differentiation, is a promising route toward future foundation models.