HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs
Authors: Ambroise Borbely, Mathieu Salzmann, Olivier Verscheure
Summary (Overview)
- HiSA (Hidden Sparse Attention) introduces a novel sparse attention mechanism that operates in the hidden (feature) dimension rather than over tokens, computing only a partial query–key dot product based on a static, parameter-free rule derived from RoPE rotation angles.
- The method keeps query and key projections intact but drops score parts when the RoPE-induced rotation exceeds an angle budget (default 90°), meaning fast frequency pairs see only nearby keys while slow pairs see far keys.
- HiSA achieves lower validation loss than dense attention while computing only ~29–34% of the dot products in pretraining experiments with 135M and 360M parameter models at 4K context.
- The method extends to FlashAttention-3 kernels (Flash-HiSA), providing training at dense speed, faster decoding (+42% tokens/s at 16K context, batch 128), and reduced KV-cache memory (~35% smaller).
- An interpretability analysis shows that dense attention learns to suppress the same score parts that HiSA drops, providing theoretical justification for the approach.
Introduction and Theoretical Foundation
Background and Motivation
The attention layer in large language models compares every query with every key via a full dot product over the head dimension. Most prior work on attention efficiency keeps this dot product intact and instead reduces the set of keys a query attends to (e.g., sliding windows, KV-cache eviction, token selection). This paper investigates an orthogonal question: does attention require computing the complete query–key dot product?
Theoretical Basis: RoPE Structure
Rotary Position Embeddings (RoPE) provide a natural structure for this investigation. RoPE rotates each two-dimensional frequency pair of queries and keys by an angle proportional to position. The key insight is:
- High-frequency pairs complete many rotations within a few tokens, making their score parts highly sensitive to exact distances
- Low-frequency pairs barely rotate over long contexts, allowing them to capture long-range dependencies
The attention score decomposes exactly as:
where is the distance between query position and key position , is the frequency of pair , and is the angle between the query and key pair before rotation.
The paper hypothesizes that models do not need precise positional patterns at long range—exactly what fast pairs encode there—while slow pairs should be kept for long-range dependencies.
Methodology
The HiSA Rule
HiSA defines sparsity using a simple criterion: the score part is kept only while the rotation induced by RoPE stays below a fixed angle budget :
Key properties:
- Static, parameter-free sparsity: the pattern depends only on positions and RoPE frequencies, not on content
- No learned parameters, no router, no top-k selection
- With , HiSA reduces to dense attention
Each pair has its own window: positions. The reach (maximum distance any pair is active) is:
Compute and Memory Analysis
The fraction of the dense causal dot product computed over a sequence of length is:
Flash-HiSA: Block-Level Implementation
For GPU efficiency, Flash-HiSA applies the exact rule block-by-block:
- Queries and keys are cut into blocks of tokens
- Features are chunked into groups of 16 (8 RoPE pairs)
- Each K-tile is either multiplied in full or skipped; boundary chunks are zeroed in Q
Experimental Setup
- Models: SmolLM-135M and SmolLM-360M architectures (head dimension , 32 RoPE pairs)
- Data: OpenWebText, ~20 tokens per parameter
- Training: AdamW, cosine schedule, peak learning rate selected for dense model
- Context: 4K for pretraining, extended to 16K for long-context experiments
Empirical Validation / Results
Pretraining Results (Table 1)
At 4K context, HiSA () outperforms dense attention:
| Model | Attention | Val. Loss | ρ (%) |
|---|---|---|---|
| 135M, | Dense | 2.906 | 100.0 |
| 135M, | HiSA | 2.880 | 29.3 |
| 135M, | Dense | 2.908 | 100.0 |
| 135M, | HiSA | 2.869 | 34.2 |
| 360M, | Dense | 2.587 | 100.0 |
| 360M, | HiSA | 2.575 | 29.3 |
The advantage holds at every position in the context window (0-512, 512-2K, 2K-4K).
Implementation Comparison (Table 2)
| Attention | Kernel | Val. Loss | Tokens/s | ρ (%) |
|---|---|---|---|---|
| Dense | SDPA (FlashAtt. 2) | 2.906 | 205k | 100.0 |
| Dense | FlashAtt. 3 | 2.904 | 236k | 100.0 |
| HiSA | Triton (exact rule) | 2.880 | 119k | 29.3 |
| HiSA block rule | Triton | 2.888 | 191k | 34.8 |
| Flash-HiSA | FlashAtt. 3 | 2.892 | 238k | 34.8 |
Long-Context Extension (Tables 3 & 4)
After extending to 16K context with :
- HiSA maintains its advantage on PG19 and OpenWebText while computing only ~20% of dot products
- Flash-HiSA trains at 94% of dense FlashAttention-3 speed while computing 22% of dot products
Decoding Performance (Table 5)
| T | Attention | Batch 32 (Tok/s) | Batch 128 (Tok/s) | KV Cache (GiB) |
|---|---|---|---|---|
| 4K | Dense (FA3) | 7.3k | 20.4k | 11.2 |
| 4K | Flash-HiSA | 7.8k | 23.2k | 7.7 |
| 16K | Dense (FA3) | 4.7k | 9.2k | 45.0 |
| 16K | Flash-HiSA | 5.6k | 13.0k | 29.1 |
Converting a Trained Dense Model (Table 6)
A dense model converted to HiSA and continued training for 1.35B tokens matches dense continued training (2.836 vs 2.838 val loss) while computing only 29.3% of dot products.
Theoretical and Practical Implications
Interpretability: Why HiSA Works
The paper demonstrates that dense attention learns to suppress the score parts that HiSA drops. Using a measure comparing the effect of true score parts against those computed with unrelated keys:
where .
Key findings:
- At initialization, all score parts have equal influence (ratio ≈ 1)
- During training, models learn to use score parts at small rotation angles (ratio > 1)
- Models learn to suppress score parts at large rotation angles (ratio < 1)
- Suppression grows with training: 87–95% of cells beyond 180° are below the uninformative reference in SmolLM-135M
Practical Implications
- Training efficiency: HiSA trains at dense speed (with FlashAttention-3) while computing ~30% of dot products
- Inference efficiency: Faster decoding (+42% at 16K, batch 128) and 35% smaller KV cache
- Model conversion: Existing dense models can be converted to HiSA without quality loss
- No architectural changes: Query/key projections, values, and softmax remain unchanged
Conclusion
HiSA demonstrates that attention does not require the complete query–key dot product. By exploiting RoPE structure, the method creates static, parameter-free sparsity in the hidden dimension that:
- Improves validation loss while computing only ~1/3 of dot products
- Maintains advantages through long-context extension
- Enables faster decoding and reduced memory footprint
- Is theoretically justified by evidence that dense models learn to suppress the same score parts
Future directions include scaling to larger models, downstream task benchmarking, and improved kernel design.
Impact: Cheaper training and inference lower the cost and energy of running language models, with no specific societal consequences beyond those of LLMs in general.
Related papers
- InfuseLoop: Cross-Trajectory Latent Injection for Scalable Parallel Looped Transformers
InfuseLoop injects cross-trajectory latent states into parallel looped transformers, eliminating training spikes and enabling quality gains at 8 loops with minimal overhead.
- Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
Looped Transformers perform state-dependent computation, not fixed one-loop-one-step algorithms, and a learned linear layer can steer frozen shared weights via attention routing.
- SWE-growth: Teacher-Guided Adaptive Self-Play for Software Engineering Agents
SWE-Growth uses a frozen teacher to guide self-play curricula, improving software agents to 31.7% on SWE-bench Pro and 17.2% on TerminalBench, surpassing student-only training.