Summary (Overview)

• Core Problem: Hybrid LLMs combine softmax attention with linear attention, which compresses history into fixed-size recurrent states. However, existing serving systems store these states densely, causing memory traffic that grows with batch size and dominates decoding cost.
• Key Insight: The paper identifies a phenomenon called the linear attention sink — queries, keys, and values of linear layers concentrate in a stable, low-dimensional subspace, making each recurrent state effectively low-rank.
• Main Contribution: SINKSTATE is a serving system that maintains recurrent states in an adaptive low-rank SVD form throughout inference, using local SVD updates, asynchronous CPU consolidation, and optimized fused kernels.
• Results: On four hybrid models (4B–120B parameters), SINKSTATE reduces peak recurrent-state memory by up to 2.6× (1.9× average), speeds up decoding by up to 1.58× (1.25× average), and serves up to 2× larger batches while maintaining accuracy.

---

Introduction and Theoretical Foundation

Background

Hybrid models (e.g., Kimi K3, Qwen3.8, Nemotron 3 Ultra, GLM-5.3-Flash) combine linear attention layers with softmax attention to balance model quality and inference efficiency. Linear attention compresses the entire history into a fixed-size recurrent state $H_t$, avoiding the quadratic cost of full attention.

The Linear Attention Sink

The paper's central observation is that the queries $q_t$, keys $k_t$, and values $v_t$ of linear attention layers concentrate in a stable, low-dimensional subspace. This holds even for randomly sampled inputs, making it semantics-independent. This "sink" causes the state $H_t$ to be effectively low-rank:

• The top-1 direction captures 39–74% of the stacked $q_t$, $k_t$, $v_t$ matrices (44–95× more than isotropic vectors)
• The top-8 directions capture 73–88% (8–14× more)
• The dominant subspace remains stable as context grows to 64k tokens, retaining 70.8–87.8% of energy

Theoretical Foundation

The state update follows the linear attention recurrence:

$$H_{t} = \alpha_{t} H_{t-1} + v_{t} k_{t}^{\top}, \tag{1}$$ $$y_{t} = H_{t} q_{t}, \tag{2}$$

where $\alpha_t \in (0,1)$ is a scalar decay. Variants like GDN use the delta rule:

$$H_t = \alpha_t H_{t-1}(I - \beta_t k_t k_t^\top) + \beta_t v_t k_t^\top$$

and KDA replaces the scalar $\alpha_t$ with a diagonal matrix $\text{Diag}(\alpha_t)$.

The paper leverages the singular value decomposition (SVD): $H = U\Sigma V^\top$ with singular values $\sigma_0 \geq \sigma_1 \geq \cdots$. Truncating small singular values gives the best low-rank approximation in Frobenius norm.

Key Observations

1. Observation 1: Incoming keys and values are mostly captured by the leading components of the state. The top-1 column captures 18–65% of $v_t$ and $k_t$ (23–83× more than isotropic), while the last columns capture only 0.03–0.1%.
2. Observation 2: Without consolidation, similar directions accumulate in the frozen part of the factorization, inflating the stored rank by 1.19–1.75× compared to exact SVD.

---

Methodology

1. Local SVD Update (§4.2)

SINKSTATE maintains the factorization $H_t \approx U_t \Sigma_t V_t^\top$ through:

• Buffering: Caches $K$ new token updates (keys and values) before compacting, amortizing SVD cost across steps.
• Top-subspace SVD: Splits factors into an active part (top-$r$ components) and a frozen part (the rest). Only the active part is merged with buffered updates for a local SVD.

The local SVD uses Brand's incremental SVD reformulation. For the concatenated factors $U_C \Sigma_C V_C^\top$, the paper orthogonalizes into $Q_L$ and $Q_R^\top$:

$$U_C \Sigma_C V_C^\top = Q_L C Q_R^\top$$

where only the small core matrix $C$ requires SVD. Components below threshold $\varepsilon \sigma_0$ are dropped.

Staggered compaction: Layers compact at different steps (only $1/K$ of layers per step) to avoid latency spikes.

2. Asynchronous CPU Consolidation (§4.3)

The frozen part accumulates redundant components over time. SINKSTATE:

1. Downloads the frozen part to the CPU
2. Runs exact SVD with OpenMP parallelism across heads
3. Drops components below $\varepsilon \sigma_0$
4. Uploads results back via pointer swapping (no data copy)

A three-stage CPU pipeline (download → SVD → upload) with dedicated threads overlaps consolidation with GPU decoding. Layers are processed in round-robin order with a drop-and-retry policy for stale results.

3. Kernel Design (§4.4)

• Memory layout: CSR-like format with offset arrays; separate contiguous regions for active and frozen parts; dense fallback when $r > 64$ for a $128 \times 128$ state (since $256r > 128^2$)
• Kernel fusion: Single decoding kernel appends key-value pairs and computes $y_t$ directly from factors without materializing dense states
• Load balancing: One program per component row for row-independent operations
• CUDA Graphs support: Via device-side pointer table and counter

---

Empirical Validation / Results

Setup

• Models: Nemotron-3-Nano-4B, Nemotron-3-Super-120B-A12B (Mamba2 layers); Qwen3.5-9B, Qwen3-Next-80B-A3B (GDN layers)
• Hardware: NVIDIA RTX PRO 6000 Blackwell GPU (96GB), 2× Intel Xeon 6515P CPUs
• Baselines: Mamba-SSM and FLA (dense-state implementations)
• Configurations: $\varepsilon = 3 \times 10^{-3}$ (Nemotron), $\varepsilon = 10^{-2}$ (Qwen), buffer size $K = 8$, top-8 active subspace

End-to-End Performance

• Speedup up to 1.58× (Nemo-4B on MATH-500 and IFEval)
• Accuracy changes: −1.6 to +2.6 points, improving in 12 of 24 model-benchmark pairs (+0.3 average)

Kernel Speedup

• State-kernel time reduced by 1.64–2.41× on Nemo-4B, 1.03–1.95× on Qwen-9B
• SINKSTATE serves batches of B=1280 on both models vs. baselines' max of B=640 (Nemo-4B) and B=768 (Qwen-9B) — up to 2× larger

Accuracy on Long Contexts

• RULER (16K contexts): at most 0.1% accuracy drop
• MATH-500 and GPQA Diamond with thinking mode (up to 32K tokens): at most 0.5% loss

Ablation Results

Rank Distribution

Ranks vary widely across heads: 22% retain ≤8 components, 13% retain >32, some use dense states. The distribution remains stable during decoding.

---

Theoretical and Practical Implications

Theoretical Implications

• Linear attention sink: The finding that queries, keys, and values concentrate in a stable, low-dimensional subspace independent of input semantics provides a theoretical explanation for the low effective rank of recurrent states observed in prior work.
• Adaptive rank: The paper demonstrates that ranks must be adapted per-state and per-head — a fixed low-rank approximation would either waste memory or lose information.
• On-the-fly factorization: This is the first work to maintain recurrent states in low-rank SVD form throughout decoding, showing that incremental SVD techniques can be made practical for LLM serving.

Practical Implications

• Memory efficiency: Recurrent states in Qwen3.8-2.4T-A95B occupy 6,144× the KV-cache memory per token; SINKSTATE's 1.6–2.6× reduction is significant for serving large models on limited GPU memory.
• Throughput: Up to 1.58× decoding speedup and 2× larger batch sizes directly translate to higher serving throughput and lower cost per token.
• Generality: The approach works across different linear attention variants (Mamba2, GDN, KDA) and model sizes (4B–120B), suggesting broad applicability to the growing family of hybrid models.

---

Conclusion

SINKSTATE addresses the redundancy in recurrent states of hybrid LLMs by exploiting the linear attention sink — the concentration of queries, keys, and values in a stable low-dimensional subspace. The system maintains each state in an adaptive low-rank SVD form throughout inference using:

1. Local SVD updates with buffering and staggered compaction
2. Asynchronous CPU consolidation to remove redundant frozen components
3. Optimized fused kernels with compact memory layout

The evaluation demonstrates up to 2.6× peak memory reduction, 1.58× decoding speedup, and 2× larger batch sizes across four hybrid models, while maintaining accuracy on benchmarks including long-context retrieval (RULER) and long reasoning tasks.

Future Directions

The paper suggests that the linear attention sink could be further exploited for other optimizations, such as more aggressive compression schemes, better scheduling of consolidation work, or extending the approach to other state-based architectures beyond linear attention variants.