Summary of "Sliding-Window Beats Linear Attention"

Overview

• Core finding: Sliding Window Attention (SWA) with attention sinks outperforms post-trained linear attention methods on both short-context knowledge/reasoning tasks and long-context reasoning tasks, while requiring zero post-training tokens and offering higher speed and lower memory cost.
• Key comparison: The paper directly compares SWA (window size 64–512, 4 sinks) against state-of-the-art linear attention post-training methods (LoLCATs, Liger-GLA, MOHAWK, DiJiang, ARWKV, QLinAtt, QRWKV6/7, Hedgehog, Mamba2, Lamba, SUPRA) across models from 1.3B to 70B parameters.
• Short-context results: SWA recovers 93.2% of MMLU baseline performance and 99.0% of average benchmark performance—matching or exceeding all linear attention methods without any fine-tuning.
• Long-context results: SWA dramatically outperforms linear attention methods; e.g., at 4K context, SWA recovers 17.2–23% of full-attention accuracy on S-NIAH tasks versus ≤5.8% for LoLCATs and ≤0.8% for Liger-GLA.
• Efficiency: SWA is faster than all linear attention variants and has lower or comparable memory cost at window sizes ≤512.

Introduction and Theoretical Foundation

Problem Statement

• Quadratic scaling: Standard Self-Attention (SA) has $\mathcal{O}(DL^2)$ complexity and requires a growing KV cache of size $\mathcal{O}(L)$ per token at inference, leading to increasing memory and compute costs as context length grows.
• Proposed solutions: Two main alternatives—Sliding Window Attention (SWA) and Linear Attention (LA)—aim to reduce this cost.

Self-Attention Formulation

The standard self-attention output at position $t$ is:

$$\mathbf{x}_t = \frac{\sum_{i=1}^{t} \exp(\mathbf{q}_t \mathbf{k}_i^\top / \sqrt{d}) \mathbf{v}_i}{\sum_{i=1}^{t} \exp(\mathbf{q}_t \mathbf{k}_i^\top / \sqrt{d})}, \quad t \in [1, \ldots, L]$$

Sliding Window Attention (SWA)

• Definition: SWA restricts attention to the previous $w$ tokens (window size), with effective receptive field growing as $l \cdot w$ after $l$ layers (similar to CNNs).
• Attention sinks: Transformers learn to assign high attention to the first few tokens (typically 4) as "sinks" for unnecessary attention. Without including these sinks, performance catastrophically degrades once they fall outside the window. The fix: attend to the first $s=4$ tokens in addition to the sliding window.

SWA formulation:

$$\mathbf{x}_t = \frac{\sum_{i=\max(1, t-w+1)}^{t} \exp(\mathbf{q}_t \mathbf{k}_i^\top / \sqrt{d}) \mathbf{v}_i}{\sum_{i=\max(1, t-w+1)}^{t} \exp(\mathbf{q}_t \mathbf{k}_i^\top / \sqrt{d})}, \quad t \in [1, \ldots, L]$$

Linear Attention (LA)

• Key idea: Approximate $\exp(\mathbf{q}_t \mathbf{k}_i^\top) \approx \phi(\mathbf{q}_t) \phi(\mathbf{k}_i)^\top$ using a kernel $\phi$, enabling reformulation:

$$\mathbf{x}_t = \frac{\phi(\mathbf{q}_t) \sum_{i=1}^{t} \phi(\mathbf{k}_i)^\top \mathbf{v}_i}{\phi(\mathbf{q}_t) \sum_{i=1}^{t} \phi(\mathbf{k}_i)^\top} = \frac{\phi(\mathbf{q}_t) \mathbf{s}_t}{\phi(\mathbf{q}_t) \mathbf{z}_t}$$

with recurrent updates:

$$\mathbf{s}_t = \mathbf{s}_{t-1} + \phi(\mathbf{k}_t)^\top \mathbf{v}_t, \quad \mathbf{z}_t = \mathbf{z}_{t-1} + \phi(\mathbf{k}_t)^\top$$

This gives $\mathcal{O}(1)$ inference cost per token with fixed-size state.

• Kernel requirements: Expressiveness, spikiness, monotonicity. Hedgehog kernel uses learnable projection + dual-sided exponential:

$$\phi(x) \leftarrow (\exp(f(x)), \exp(-f(x)))$$

where $f$ is a linear projection from $D$ to $D/2$.

Post-Training Linear Attention

• Instead of training from scratch, existing pretrained models can be linearized via post-training (e.g., LoLCATs with only 40M tokens using LoRA). LoLCATs combines linear attention with a small SWA component.

Methodology

Experimental Setup

• Models tested: Phi-1.5-1.3B, Mistral-7B-v0.1, Llama2.0-7B, Llama3.0-8B (base and Instruct), Llama-3.1- -8B/70B,[mm]
• Qwen2.-5-7B/32B/72B-Instruct, QwQ-32B
• SWA configuration: Window sizes 64–512, always with 4 attention sinks; no post-training.
• Benchmarks:
  • Short-context: MMLU (5-shot), ARC-C, ARC-E, HellaSwag, PIQA, WinoGrande
  • Long-context: Single Needle-in-a-Haystack (S-NIAH-1/2/3), BABILong
• Baselines: Pre-trained teachers (full attention) and various linear attention post-training methods (LoLCATs, Liger-GLA, MOHAWK, DiJiang, ARWKV, QLinAtt, QRWKV6/7, Hedgehog, Mamba2, Lamba, SUPRA).

Efficiency Comparison

• Backends: FA and SWA use FlashAttention; linear attention uses ThunderKittens; LoLCATs uses fused SWA+linear kernel (window=256).
• Hardware: NVIDIA RTX PRO 6000 Blackwell Max-Q; 4-layer Transformer (embedding 1024, 16 heads, dim 64), batch-size 1, float16.

Empirical Validation / Results

Short-Context Knowledge and Reasoning

Key findings:

• SWA achieves the best average performance in 9 out of 11 cases.
• Only exceptions: LoLCATs on Phi-1.5 (62.5 vs 62.4 for SWA) and QRWKV6 on Qwen2.5-32B (77.3 vs 76.6 for SWA).
• SWA recovers 93.2% of MMLU and 99.0% of average baseline performance with zero training tokens.

Long-Context Reasoning (S-NIAH)

Key findings: At all window sizes and context lengths, SWA achieves equal or higher scores than linear attention methods. At 4K context, SWA recovers 17.2–23% of full attention accuracy vs. ≤5.8% for LoLCATs and ≤0.8% for Liger-GLA.

BABILong Results

Key findings: At short contexts (0K, 1K), LoLCATs slightly outperforms SWA (56 vs 55; 22 vs 20). At longer contexts (2K, 4K), SWA clearly dominates (19 vs 10; 15 vs 3). SWA recovers 25% of baseline at 4K vs. only 5% for LoLCATs.

Speed and Memory

• Speed: FA slows down with context length; SWA, linear attention, and LoLCATs maintain flat speed. SWA is the fastest method (window=64 > window=512 > linear > LoLCATs).
• Memory: FA grows linearly; SWA plateaus after reaching window size. SWA(64) has the lowest memory, followed by Linear, LoLCATs, then SWA(512).

Theoretical and Practical Implications

Key Insights

1. SWA with sinks is a strong, training-free baseline that has been overlooked in linear attention comparisons# research. It achieves comparable or better performance than sophisticated post-training methods.
2. Linear attention's advantage is primarily# not clear-cut: The O(1) inference cost comes at the price expense of significant performance degradation, especially on long-context tasks.
3. Training efficiency: SWA requires zero post-training tokens, while linear attention methods need 20M–100B tokens to reach similar (or lower) performance.

Practical Recommendations

• For practitioners needing fixed-memory inference, SWA with sinks (window 64–512## 512) is recommended over linear attention post-training.
• SWA's simplicity (no kernel engineering, no fine-tuning) makes it a more practical choice choice for deployment.

Conclusion

The paper demonstrates that Sliding Window Attention with attention sinks outperforms post-trained linear attention methods across short- and long-context benchmarks, while being faster and more memory-efficient. The authors recommend SWA as a## a simple, effective alternative to linearpackage linear attention for practitioners.

Future Work

• Investigate post-training with SWA and compare scaling laws against linear attention.
• Explore hybrid models with partial full-attention layers.
• Extend# analysis to larger models, agentic tasks, and multi-modal/video generation models (where SWA already shows promise in diffusion models).

---

Note: This summary preserves all key quantitative results and mathematical formulations from the original paper. The tables and equations are reproduced to maintain fidelity to the source data.