# HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs

> HiSA sparsifies the hidden dimension of RoPE attention, computing only ~30% of dot products while achieving lower validation loss than dense attention.

- **Source:** [arXiv](https://arxiv.org/abs/2611.26363)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/0JZu70
- **Whiteboard:** https://picx.dev/p/0JZu70/image

## Summary

# HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs

**Authors:** Ambroise Borbely, Mathieu Salzmann, Olivier Verscheure

## Summary (Overview)

- **HiSA (Hidden Sparse Attention)** introduces a novel sparse attention mechanism that operates in the hidden (feature) dimension rather than over tokens, computing only a partial query–key dot product based on a static, parameter-free rule derived from RoPE rotation angles.
- The method keeps query and key projections intact but drops score parts $s_{ij}^{(l)}$ when the RoPE-induced rotation $\Delta\theta_l$ exceeds an angle budget $\Phi$ (default 90°), meaning fast frequency pairs see only nearby keys while slow pairs see far keys.
- HiSA achieves **lower validation loss than dense attention** while computing only ~29–34% of the dot products in pretraining experiments with 135M and 360M parameter models at 4K context.
- The method extends to **FlashAttention-3 kernels** (Flash-HiSA), providing training at dense speed, faster decoding (+42% tokens/s at 16K context, batch 128), and reduced KV-cache memory (~35% smaller).
- An interpretability analysis shows that **dense attention learns to suppress the same score parts that HiSA drops**, providing theoretical justification for the approach.

## Introduction and Theoretical Foundation

### Background and Motivation

The attention layer in large language models compares every query with every key via a full dot product over the head dimension. Most prior work on attention efficiency keeps this dot product intact and instead reduces the set of keys a query attends to (e.g., sliding windows, KV-cache eviction, token selection). This paper investigates an orthogonal question: **does attention require computing the complete query–key dot product?**

### Theoretical Basis: RoPE Structure

Rotary Position Embeddings (RoPE) provide a natural structure for this investigation. RoPE rotates each two-dimensional frequency pair of queries and keys by an angle proportional to position. The key insight is:

- **High-frequency pairs** complete many rotations within a few tokens, making their score parts highly sensitive to exact distances
- **Low-frequency pairs** barely rotate over long contexts, allowing them to capture long-range dependencies

The attention score decomposes exactly as:

$$
s_{ij} = \frac{1}{\sqrt{d}} \sum_{l=0}^{P-1} \left\| \mathbf{q}_i^{(l)} \right\| \left\| \mathbf{k}_j^{(l)} \right\| \cos \left(\varphi_{ij}^{(l)} + \Delta\theta_l\right), \tag{1}
$$

where $\Delta = i - j$ is the distance between query position $i$ and key position $j$, $\theta_l = b^{-2l/d}$ is the frequency of pair $l$, and $\varphi_{ij}^{(l)}$ is the angle between the query and key pair before rotation.

The paper hypothesizes that **models do not need precise positional patterns at long range**—exactly what fast pairs encode there—while slow pairs should be kept for long-range dependencies.

## Methodology

### The HiSA Rule

HiSA defines sparsity using a simple criterion: the score part $s_{ij}^{(l)}$ is kept only while the rotation induced by RoPE stays below a fixed angle budget $\Phi$:

$$
s_{ij}^{\mathrm{HiSA}} = \frac{1}{\sqrt{d}} \sum_{l=0}^{P-1} \mathbf{1}\left[\Delta\theta_l < \Phi\right] \left\langle R_{i\theta_l}\mathbf{q}_i^{(l)}, R_{j\theta_l}\mathbf{k}_j^{(l)}\right\rangle \tag{3}
$$

Key properties:
- **Static, parameter-free sparsity**: the pattern depends only on positions and RoPE frequencies, not on content
- **No learned parameters, no router, no top-k selection**
- With $\Phi = \infty$, HiSA reduces to dense attention

Each pair $l$ has its own window: $\Delta_{\max,l} = \Phi/\theta_l$ positions. The **reach** (maximum distance any pair is active) is:

$$
R = \frac{\Phi}{\theta_{P-1}} = \Phi b^{(d-2)/d} \tag{4}
$$

### Compute and Memory Analysis

The fraction of the dense causal dot product computed over a sequence of length $T$ is:

$$
\rho(T) = \frac{1}{P} \sum_{l=0}^{P-1} \frac{\sum_{\Delta=0}^{T-1} (T-\Delta) \mathbf{1}[\Delta < \Delta_{\max,l}]}{T(T+1)/2} \tag{5}
$$

### Flash-HiSA: Block-Level Implementation

For GPU efficiency, Flash-HiSA applies the exact rule block-by-block:
- Queries and keys are cut into blocks of $S=128$ tokens
- Features are chunked into groups of 16 (8 RoPE pairs)
- Each K-tile is either multiplied in full or skipped; boundary chunks are zeroed in Q

### Experimental Setup

- Models: SmolLM-135M and SmolLM-360M architectures (head dimension $d=64$, 32 RoPE pairs)
- Data: OpenWebText, ~20 tokens per parameter
- Training: AdamW, cosine schedule, peak learning rate selected for dense model
- Context: 4K for pretraining, extended to 16K for long-context experiments

## Empirical Validation / Results

### Pretraining Results (Table 1)

At 4K context, HiSA ($\Phi = 90^\circ$) outperforms dense attention:

| Model | Attention | Val. Loss | ρ (%) |
|-------|-----------|-----------|-------|
| 135M, $b=10^4$ | Dense | 2.906 | 100.0 |
| 135M, $b=10^4$ | HiSA | **2.880** | 29.3 |
| 135M, $b=2\cdot10^4$ | Dense | 2.908 | 100.0 |
| 135M, $b=2\cdot10^4$ | HiSA | **2.869** | 34.2 |
| 360M, $b=10^4$ | Dense | 2.587 | 100.0 |
| 360M, $b=10^4$ | HiSA | **2.575** | 29.3 |

The advantage holds at every position in the context window (0-512, 512-2K, 2K-4K).

### Implementation Comparison (Table 2)

| Attention | Kernel | Val. Loss | Tokens/s | ρ (%) |
|-----------|--------|-----------|----------|-------|
| Dense | SDPA (FlashAtt. 2) | 2.906 | 205k | 100.0 |
| Dense | FlashAtt. 3 | 2.904 | 236k | 100.0 |
| HiSA | Triton (exact rule) | **2.880** | 119k | 29.3 |
| HiSA block rule | Triton | 2.888 | 191k | 34.8 |
| Flash-HiSA | FlashAtt. 3 | 2.892 | **238k** | 34.8 |

### Long-Context Extension (Tables 3 & 4)

After extending to 16K context with $b = 2\cdot10^4$:
- HiSA maintains its advantage on PG19 and OpenWebText while computing only ~20% of dot products
- Flash-HiSA trains at 94% of dense FlashAttention-3 speed while computing 22% of dot products

### Decoding Performance (Table 5)

| T | Attention | Batch 32 (Tok/s) | Batch 128 (Tok/s) | KV Cache (GiB) |
|---|-----------|-------------------|--------------------|----------------|
| 4K | Dense (FA3) | 7.3k | 20.4k | 11.2 |
| 4K | Flash-HiSA | 7.8k | 23.2k | 7.7 |
| 16K | Dense (FA3) | 4.7k | 9.2k | 45.0 |
| 16K | Flash-HiSA | **5.6k** | **13.0k** | **29.1** |

### Converting a Trained Dense Model (Table 6)

A dense model converted to HiSA and continued training for 1.35B tokens **matches dense continued training** (2.836 vs 2.838 val loss) while computing only 29.3% of dot products.

## Theoretical and Practical Implications

### Interpretability: Why HiSA Works

The paper demonstrates that **dense attention learns to suppress the score parts that HiSA drops**. Using a measure comparing the effect of true score parts against those computed with unrelated keys:

$$\Delta\mathbf{o}_i = \frac{\sum_j u_j(\mathbf{v}_j - \mathbf{o}_i)}{1 + \sum_j u_j}, \tag{10}$$

where $u_j = a_{ij}(e^{-s_{ij}^{(l)}} - 1)\mathbf{1}[i-j \in \mathcal{B}]$.

Key findings:
- At initialization, all score parts have equal influence (ratio ≈ 1)
- During training, models learn to **use** score parts at small rotation angles (ratio > 1)
- Models learn to **suppress** score parts at large rotation angles (ratio < 1)
- Suppression grows with training: 87–95% of cells beyond 180° are below the uninformative reference in SmolLM-135M

### Practical Implications

1. **Training efficiency**: HiSA trains at dense speed (with FlashAttention-3) while computing ~30% of dot products
2. **Inference efficiency**: Faster decoding (+42% at 16K, batch 128) and 35% smaller KV cache
3. **Model conversion**: Existing dense models can be converted to HiSA without quality loss
4. **No architectural changes**: Query/key projections, values, and softmax remain unchanged

## Conclusion

HiSA demonstrates that **attention does not require the complete query–key dot product**. By exploiting RoPE structure, the method creates static, parameter-free sparsity in the hidden dimension that:
- Improves validation loss while computing only ~1/3 of dot products
- Maintains advantages through long-context extension
- Enables faster decoding and reduced memory footprint
- Is theoretically justified by evidence that dense models learn to suppress the same score parts

**Future directions** include scaling to larger models, downstream task benchmarking, and improved kernel design.

**Impact**: Cheaper training and inference lower the cost and energy of running language models, with no specific societal consequences beyond those of LLMs in general.

---

_Markdown view of https://picx.dev/p/0JZu70, served by PicX — AI-generated visual whiteboard summaries of research papers._
