HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs

Authors: Ambroise Borbely, Mathieu Salzmann, Olivier Verscheure

Summary (Overview)

  • HiSA (Hidden Sparse Attention) introduces a novel sparse attention mechanism that operates in the hidden (feature) dimension rather than over tokens, computing only a partial query–key dot product based on a static, parameter-free rule derived from RoPE rotation angles.
  • The method keeps query and key projections intact but drops score parts sij(l)s_{ij}^{(l)} when the RoPE-induced rotation Δθl\Delta\theta_l exceeds an angle budget Φ\Phi (default 90°), meaning fast frequency pairs see only nearby keys while slow pairs see far keys.
  • HiSA achieves lower validation loss than dense attention while computing only ~29–34% of the dot products in pretraining experiments with 135M and 360M parameter models at 4K context.
  • The method extends to FlashAttention-3 kernels (Flash-HiSA), providing training at dense speed, faster decoding (+42% tokens/s at 16K context, batch 128), and reduced KV-cache memory (~35% smaller).
  • An interpretability analysis shows that dense attention learns to suppress the same score parts that HiSA drops, providing theoretical justification for the approach.

Introduction and Theoretical Foundation

Background and Motivation

The attention layer in large language models compares every query with every key via a full dot product over the head dimension. Most prior work on attention efficiency keeps this dot product intact and instead reduces the set of keys a query attends to (e.g., sliding windows, KV-cache eviction, token selection). This paper investigates an orthogonal question: does attention require computing the complete query–key dot product?

Theoretical Basis: RoPE Structure

Rotary Position Embeddings (RoPE) provide a natural structure for this investigation. RoPE rotates each two-dimensional frequency pair of queries and keys by an angle proportional to position. The key insight is:

  • High-frequency pairs complete many rotations within a few tokens, making their score parts highly sensitive to exact distances
  • Low-frequency pairs barely rotate over long contexts, allowing them to capture long-range dependencies

The attention score decomposes exactly as:

sij=1d∑l=0P−1∥qi(l)∥∥kj(l)∥cos⁡(φij(l)+Δθl),(1)s_{ij} = \frac{1}{\sqrt{d}} \sum_{l=0}^{P-1} \left\| \mathbf{q}_i^{(l)} \right\| \left\| \mathbf{k}_j^{(l)} \right\| \cos \left(\varphi_{ij}^{(l)} + \Delta\theta_l\right), \tag{1}

where Δ=i−j\Delta = i - j is the distance between query position ii and key position jj, θl=b−2l/d\theta_l = b^{-2l/d} is the frequency of pair ll, and φij(l)\varphi_{ij}^{(l)} is the angle between the query and key pair before rotation.

The paper hypothesizes that models do not need precise positional patterns at long range—exactly what fast pairs encode there—while slow pairs should be kept for long-range dependencies.

Methodology

The HiSA Rule

HiSA defines sparsity using a simple criterion: the score part sij(l)s_{ij}^{(l)} is kept only while the rotation induced by RoPE stays below a fixed angle budget Φ\Phi:

sijHiSA=1d∑l=0P−11[Δθl<Φ]⟨Riθlqi(l),Rjθlkj(l)⟩(3)s_{ij}^{\mathrm{HiSA}} = \frac{1}{\sqrt{d}} \sum_{l=0}^{P-1} \mathbf{1}\left[\Delta\theta_l < \Phi\right] \left\langle R_{i\theta_l}\mathbf{q}_i^{(l)}, R_{j\theta_l}\mathbf{k}_j^{(l)}\right\rangle \tag{3}

Key properties:

  • Static, parameter-free sparsity: the pattern depends only on positions and RoPE frequencies, not on content
  • No learned parameters, no router, no top-k selection
  • With Φ=∞\Phi = \infty, HiSA reduces to dense attention

Each pair ll has its own window: Δmax⁡,l=Φ/θl\Delta_{\max,l} = \Phi/\theta_l positions. The reach (maximum distance any pair is active) is:

R=ΦθP−1=Φb(d−2)/d(4)R = \frac{\Phi}{\theta_{P-1}} = \Phi b^{(d-2)/d} \tag{4}

Compute and Memory Analysis

The fraction of the dense causal dot product computed over a sequence of length TT is:

ρ(T)=1P∑l=0P−1∑Δ=0T−1(T−Δ)1[Δ<Δmax⁡,l]T(T+1)/2(5)\rho(T) = \frac{1}{P} \sum_{l=0}^{P-1} \frac{\sum_{\Delta=0}^{T-1} (T-\Delta) \mathbf{1}[\Delta < \Delta_{\max,l}]}{T(T+1)/2} \tag{5}

Flash-HiSA: Block-Level Implementation

For GPU efficiency, Flash-HiSA applies the exact rule block-by-block:

  • Queries and keys are cut into blocks of S=128S=128 tokens
  • Features are chunked into groups of 16 (8 RoPE pairs)
  • Each K-tile is either multiplied in full or skipped; boundary chunks are zeroed in Q

Experimental Setup

  • Models: SmolLM-135M and SmolLM-360M architectures (head dimension d=64d=64, 32 RoPE pairs)
  • Data: OpenWebText, ~20 tokens per parameter
  • Training: AdamW, cosine schedule, peak learning rate selected for dense model
  • Context: 4K for pretraining, extended to 16K for long-context experiments

Empirical Validation / Results

Pretraining Results (Table 1)

At 4K context, HiSA (Φ=90∘\Phi = 90^\circ) outperforms dense attention:

ModelAttentionVal. Lossρ (%)
135M, b=104b=10^4Dense2.906100.0
135M, b=104b=10^4HiSA2.88029.3
135M, b=2⋅104b=2\cdot10^4Dense2.908100.0
135M, b=2⋅104b=2\cdot10^4HiSA2.86934.2
360M, b=104b=10^4Dense2.587100.0
360M, b=104b=10^4HiSA2.57529.3

The advantage holds at every position in the context window (0-512, 512-2K, 2K-4K).

Implementation Comparison (Table 2)

AttentionKernelVal. LossTokens/sρ (%)
DenseSDPA (FlashAtt. 2)2.906205k100.0
DenseFlashAtt. 32.904236k100.0
HiSATriton (exact rule)2.880119k29.3
HiSA block ruleTriton2.888191k34.8
Flash-HiSAFlashAtt. 32.892238k34.8

Long-Context Extension (Tables 3 & 4)

After extending to 16K context with b=2⋅104b = 2\cdot10^4:

  • HiSA maintains its advantage on PG19 and OpenWebText while computing only ~20% of dot products
  • Flash-HiSA trains at 94% of dense FlashAttention-3 speed while computing 22% of dot products

Decoding Performance (Table 5)

TAttentionBatch 32 (Tok/s)Batch 128 (Tok/s)KV Cache (GiB)
4KDense (FA3)7.3k20.4k11.2
4KFlash-HiSA7.8k23.2k7.7
16KDense (FA3)4.7k9.2k45.0
16KFlash-HiSA5.6k13.0k29.1

Converting a Trained Dense Model (Table 6)

A dense model converted to HiSA and continued training for 1.35B tokens matches dense continued training (2.836 vs 2.838 val loss) while computing only 29.3% of dot products.

Theoretical and Practical Implications

Interpretability: Why HiSA Works

The paper demonstrates that dense attention learns to suppress the score parts that HiSA drops. Using a measure comparing the effect of true score parts against those computed with unrelated keys:

Δoi=∑juj(vj−oi)1+∑juj,(10)\Delta\mathbf{o}_i = \frac{\sum_j u_j(\mathbf{v}_j - \mathbf{o}_i)}{1 + \sum_j u_j}, \tag{10}

where uj=aij(e−sij(l)−1)1[i−j∈B]u_j = a_{ij}(e^{-s_{ij}^{(l)}} - 1)\mathbf{1}[i-j \in \mathcal{B}].

Key findings:

  • At initialization, all score parts have equal influence (ratio ≈ 1)
  • During training, models learn to use score parts at small rotation angles (ratio > 1)
  • Models learn to suppress score parts at large rotation angles (ratio < 1)
  • Suppression grows with training: 87–95% of cells beyond 180° are below the uninformative reference in SmolLM-135M

Practical Implications

  1. Training efficiency: HiSA trains at dense speed (with FlashAttention-3) while computing ~30% of dot products
  2. Inference efficiency: Faster decoding (+42% at 16K, batch 128) and 35% smaller KV cache
  3. Model conversion: Existing dense models can be converted to HiSA without quality loss
  4. No architectural changes: Query/key projections, values, and softmax remain unchanged

Conclusion

HiSA demonstrates that attention does not require the complete query–key dot product. By exploiting RoPE structure, the method creates static, parameter-free sparsity in the hidden dimension that:

  • Improves validation loss while computing only ~1/3 of dot products
  • Maintains advantages through long-context extension
  • Enables faster decoding and reduced memory footprint
  • Is theoretically justified by evidence that dense models learn to suppress the same score parts

Future directions include scaling to larger models, downstream task benchmarking, and improved kernel design.

Impact: Cheaper training and inference lower the cost and energy of running language models, with no specific societal consequences beyond those of LLMs in general.

Related papers