# LINEARKV: ONE CACHED STATE SUFFICES FORPOSITION-INDEPENDENT CACHING IN HYBRID LLMS

> LinearKV shows that initializing recurrent layers from a single cached state, not exact composition, robustly enables position-independent caching in hybrid LLMs, recovering up to 92% of full quality.

- **Source:** [arXiv](https://arxiv.org/abs/2608.11231)
- **Published:** 2026-08-15
- **Permalink:** https://picx.dev/p/zKK3jH
- **Whiteboard:** https://picx.dev/p/zKK3jH/image

## Summary

# LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs

## Summary (Overview)

- **Core contribution**: LinearKV is a training-free position-independent caching (PIC) framework for hybrid LLMs (models mixing full-attention and recurrent layers like Mamba-2 and Gated DeltaNet). It enables cache reuse across requests without requiring full prefill.

- **Key insight**: A **decoupled initialization** — full-attention (FA) layers concatenate cached KV entries as in standard PIC, while each recurrent (linear) layer is initialized from a **single cached state** rather than composing all K matched chunk states. Surprisingly, this simple approach is both more robust and cheaper than the algebraically "exact" composition.

- **Counterintuitive finding**: The mathematically principled approach of composing all K cached states into the exact full-prefix state (as done in concurrent work HYPIC) is **architecture-fragile** — it collapses on Mamba-2 models (recovering only 46.6% of full quality under EPIC) while a single-block initializer recovers 86.8%, yet the two tie on GDN models.

- **Compatibility**: LinearKV reuses existing PIC selectors (CacheBlend, EPIC, ProphetKV) unchanged — only the linear-state initializer differs — making it a drop-in enhancement for hybrid architectures.

- **Efficiency**: The single-state initializer cuts time-to-first-token (TTFT) to 0.46× full prefill (vs. 5–17% additional overhead for exact composition), validated across LongBench QA and RULER at 8K–32K context lengths.

## Introduction and Theoretical Foundation

### Background

Long-context LLM applications (multi-turn conversations, agentic workflows, long-document understanding) incur prefill costs that grow with prompt length. Standard prefix-based caching limits hit rates because cached entries are only reusable when tokens follow the same prefix. **Position-independent caching (PIC)** relaxes this by allowing independently cached chunks to be reused at any position, with selective recomputation of a small token subset to restore cross-chunk attention.

### The Hybrid LLM Challenge

Hybrid LLMs interleave a few full-attention (FA) layers with a majority of recurrent layers (Mamba-2, Gated DeltaNet). This creates two challenges:

- **(C1) Reuse mismatch**: FA layers expose per-token KV caches that concatenate naturally, but recurrent layers summarize each chunk into a single fixed-size state with no token-indexed entries to concatenate or selectively repair.
- **(C2) Generalization**: Hybrids vary in recurrence family (Mamba-2 vs. GDN) and recurrent-to-FA ratio, so a solution must generalize across architectures.

### Theoretical Foundation

A recurrent layer compresses its processed prefix into a fixed-size state $S_i^{\ell} \in \mathbb{R}^{d_k \times d_v}$, updated token-by-token through the general affine recurrence:

$$
S_i^{\ell} = T_i^{\ell} S_{i-1}^{\ell} + u_i^{\ell}, \tag{1}
$$

where $T_i^{\ell}$ transports the previous state and $u_i^{\ell}$ is a token-dependent outer-product update. The two families instantiate $T_i^{\ell}$ differently:

$$
\text{Mamba-2:} \quad S_i = a_i S_{i-1} + u_i, \quad T_i = a_i I, \tag{2}
$$

$$
\text{GDN:} \quad S_i = T_i S_{i-1} + u_i, \quad T_i = \alpha_i (I - \beta_i k_i k_i^{\top}), \tag{3}
$$

where $a_i \in (0,1]$ is a per-head scalar decay, $\alpha_i = \exp(g_i)$ is a learned gate, and $I - \beta_i k_i k_i^{\top}$ is a token-dependent rank-1 correction. Mamba-2 transports all state directions with the same scalar, while GDN applies direction-dependent dense transitions — a difference that shapes error propagation.

## Methodology

### LinearKV Framework

The framework follows a four-step pipeline (Algorithm 1):

1. **Offline caching**: Each reusable chunk $C$ is prefilled independently, storing its per-chunk FA KV and each recurrent layer's end state $S_j^{\text{local}}$.

2. **Decoupled initialization**: FA layers concatenate cached KV along the token axis; each recurrent layer folds the K cached states into one initial state:

$$
S_{\text{init}} = f(S_1^{\text{local}}, \dots, S_K^{\text{local}}), \tag{4}
$$

3. **Selector-driven recomputation**: Any existing PIC selector $\Pi(q, C_{1:K}, r)$ outputs repair positions $\mathcal{U} = \{i_1 < \cdots < i_m\}$ with $m \approx rN$; LinearKV replays these in context order, advancing each recurrent state from $S_{\text{init}}$ while overwriting FA KV at those positions.

4. **Serve**: The query is prefilled against the repaired cache and decoded.

### Two Initializer Choices

**Exact composition** uses the chunk-level affine operator. Since the recurrence is linear in state, each chunk applies:

$$
S_{\text{out}} = T_{C_j} S_{\text{in}} + S_j^{\text{local}}, \tag{5}
$$

where $T_{C_j} = \prod_{t \in C_j} T_t$ is the cumulative transition. Telescoping over K chunks:

$$
S_{\text{init}}^{\text{exact}} = \sum_{j=1}^{K} \left(\prod_{m > j} T_{C_m}\right) S_j^{\text{local}}, \tag{6}
$$

**Last-block initialization** simply uses the final matched chunk's state: $S_{\text{init}} = S_K^{\text{local}}$.

### Error Analysis

The exactness of composition is conditional: each chunk's operator is built from isolated prefill (missing earlier context), so it differs from the full-prefill operator. The error follows:

$$
e_j = T_{C_j} e_{j-1} + \delta_j, \tag{7}
$$

where $e_j$ is the gap between composed and true full-prefix state after chunk $j$, and $\delta_j$ is the fresh mismatch from mis-conditioned operators. Mamba-2's scalar decay only retains/sums transported error, while GDN's dense gated transition can suppress specific directions.

## Empirical Validation / Results

### Setup

- **Models**: Granite-4.0-H-Tiny (7B, Mamba-2, 90% recurrent), OLMo-Hybrid-7B-Instruct (7B, GDN, 75% recurrent), Qwen3.6-27B (27B, GDN, 75% recurrent)
- **Selectors**: CacheBlend, EPIC, ProphetKV (all plugged in unchanged)
- **Benchmarks**: LongBench QA (5 datasets, token-F1), RULER (8K/32K, string-match recall), LongBench summarization (ROUGE-L)
- **Budget**: matched recompute ratio r≈0.2

### Key Results

**Finding 1 — Framework transfers**: Every existing selector transfers to hybrid models with decoupled initialization. On GDN models, the initializer choice is immaterial (paired EX/LB gap ≤0.013 Avg-F1), yet both recover up to 92% of full quality (OLMo, ProphetKV, RULER-8K).

**Finding 2 — Mamba-2 collapse**: On Granite, exact composition collapses under every selector while last-block stays robust:

| Selector | EX Avg-F1 (% of full) | LB Avg-F1 (% of full) |
|----------|----------------------|----------------------|
| CacheBlend | 0.063 | 0.118 |
| EPIC | 0.145 (46.6%) | 0.270 (86.8%) |
| ProphetKV | 0.125 | 0.205 |

The RULER results confirm the split: Granite Avg 0.030/0.110/0.362 (EX) vs. 0.375/0.596/0.622 (LB) for CacheBlend/EPIC/ProphetKV.

**Finding 3 — Efficiency**: Last-block is uniformly faster than exact composition in all 27 measured pairs (by 5–17%), cutting TTFT to 0.46–0.62× full recompute at 32K.

**Finding 4 — Ablations**:
- More recompute cannot rescue a bad initializer: exact composition stays flat at 41–52% of full quality across all budgets on Granite, while last-block reaches 76–89%.
- Under a HYPIC-style seam selector, last-block lifts Granite from 47.6% to 67.8% of full quality — a 20pp gain from the initializer alone.

## Theoretical and Practical Implications

### Theoretical Implications

- **Challenges the "exactness" assumption**: The paper demonstrates that algebraic exactness in composing cached states is conditional on operator correctness, which fails when chunks are prefilled in isolation. This has broader implications for any caching scheme relying on compositional algebra over independently-trained components.

- **Recurrence family shapes error propagation**: Mamba-2's scalar decay retains and sums transported error, while GDN's dense gated transitions can suppress specific directions. This provides a testable prediction (Eq. 7) for future hybrid architectures.

- **Single-source construction matters**: A random single block works as well as the last block (Appendix A), isolating single-source construction as the operative factor — not which specific block is chosen.

### Practical Implications

- **Drop-in enhancement**: LinearKV requires no retraining and leaves selector interfaces untouched, making it directly deployable with existing PIC infrastructure.
- **Quality-latency tradeoff**: The last-block initializer wins on both axes — better quality and lower latency — making it the clear default for hybrid LLM serving.
- **Architecture-aware caching**: The results suggest that caching strategies should be tuned to recurrence family, not just model scale or layer count.

## Conclusion

LinearKV shows that **one cached state suffices** for position-independent caching in hybrid LLMs. The decoupled-initialization framework reuses existing PIC selectors unchanged, while the single-block initializer proves robust across architectures — matching exact composition on GDN models and far exceeding it on Mamba-2, where exact composition collapses from mis-conditioned deep operators. The approach reduces TTFT to 0.46× full prefill while recovering up to 92% of full quality.

**Future directions** include batched serving scenarios, host-to-device cache transfer optimization, and validating the error-propagation predictions on more hybrid architectures as they emerge.

---

_Markdown view of https://picx.dev/p/zKK3jH, served by PicX — AI-generated visual whiteboard summaries of research papers._