Summary (Overview)

• GATESIGHT is a novel query-agnostic KV cache eviction method for hybrid LLMs that uses the recurrent memory signals of Gated DeltaNet (GDN) layers to predict which attention KV entries will be needed in the future.
• The paper presents three key observations: (1) GDN memory dynamics are structured and stable across documents, (2) GDN write/forget signals anticipate future KV utility before any query exists, and (3) this relationship is distributed across heads/layers and nonlinear.
• GATESIGHT uses lightweight per-head MLPs that read native GDN signals (write gate, decay, write magnitude) plus age features—no attention scores are computed for selection.
• With only 10% of the attention KV cache, GATESIGHT outperforms four other query-agnostic baselines on three of four model–benchmark pairs and leads KVzip on all four, by 9.5 to 47.2 points.
• Scoring overhead is minimal: 0.246s for a 256K-token context on Qwen3.8-27B (~0.5% of prefill time).

Introduction and Theoretical Foundation

Background

Hybrid language models interleave Gated DeltaNet (GDN) layers with full-attention layers. While GDN layers compress context into fixed-size recurrent states, the attention layers still maintain a KV cache that grows linearly with context length, dominating serving memory. For Qwen3.8-27B, the attention KV cache exceeds GDN states beyond ~2.4K tokens and is ~42× larger at 100K tokens.

The Core Question

Does the recurrent memory already know what the attention memory will need?

This question is motivated by the observation that GDN layers make per-token decisions about how strongly to write tokens into memory and how quickly to forget them—decisions that may carry information about which KV entries attention layers will later need.

Problem Setting

The paper focuses on contexts that are prefilled once and whose cache is reused for multiple subsequent questions (a pattern adopted by inference frameworks). Since the cache must be compacted at prefill time, eviction scores cannot depend on future queries.

Formal Definition of Future KV Utility

For a context of $n$ tokens, with $M = 4$ sampled continuations $z^{(1)}, \ldots, z^{(M)}$ of up to 128 tokens:

$$\Delta r(t, i) = \sum_{j \in \mathcal{Q}(h)} \frac{a_j(t, i)}{1 - a_j(t, i)} W_O^{(j)} \big(v_i - o_j(t)\big), \qquad U(i) = \max_{t \in z^{(1..M)}} \big\| \Delta r(t, i) \big\|$$

where $a_j(t, i)$ is the attention weight of query $t$ on entry $i$, $o_j(t)$ is the output of head $j$, and $W_O^{(j)}$ is its slice of the output projection. The utility $U(i)$ measures the exact joint leave-one-out change of the ungated output of the layer.

Methodology

Native GDN Signals

A GDN layer updates its state using the gated delta rule:

$$S_t = \alpha_t \big(I - \beta_t k_t k_t^\top\big) S_{t-1} + \beta_t k_t v_t^\top, \quad \alpha_t = \exp(g_t), \; g_t \leq 0$$

where $k_t$ and $v_t$ are the key and value written by token $t$, $\beta_t \in (0, 1)$ is the write gate, and $g_t$ is the log-decay. The three native GDN signals are:

• Write gate: $\beta_t$
• Decay: $g_t$
• Write magnitude: $\|v_t\|$

GATESIGHT Estimator

For each attention layer $\ell$ and KV head $h$, the input feature vector for context entry $i$ is:

$$x_i^{(\ell)} = \left[ \{\beta_{i,m}^{(j)}, g_{i,m}^{(j)}, \|v_{i,m}^{(j)}\| \}_{j \in W(\ell), m=1..48}; \frac{i}{n}, \frac{n-i}{n}, \frac{\log(1 + n - i)}{\log(1 + n)} \right]$$

This gives $6 \times 48 \times 3 + 3 = 867$ dimensions: native signals from 48 heads of up to 6 preceding GDN layers, plus 3 normalized age features.

The estimator $f_\theta^{(\ell,h)}: \mathbb{R}^{867} \to [0,1]$ is a multilayer perceptron with one hidden layer of 64 units and sigmoid output. Qwen3.8-27B has $16 \times 4 = 64$ such MLPs (3.4M total parameters).

Training

• Target: Rollout utility of Eq. 2, max-normalized per (document, layer, KV head) to [0,1], trained with binary cross-entropy loss
• Data: 2,428 Nemotron documents (750–1,250 tokens each), 500 random positions per document
• Optimizer: AdamW, learning rate $2 \times 10^{-3}$, 12 epochs
• No task data, questions, or downstream benchmarks used

Per-Layer Allocation from Output Gate

The layer weight $w_\ell$ is the mean magnitude of the attention output gate (measured once offline). Layer $\ell$ receives $K_\ell \propto w_\ell$ entries per head, with the same model-wide total as flat allocation. The mean gate spans 0.022 to 0.459 across layers—more than an order of magnitude.

Deployment

Implemented in vLLM 0.28 as a plugin that changes no kernels. Each attention layer has its own block table; after prefill, the estimator scores the context once, kept entries move to the front of pages, and freed blocks return to the pool.

Empirical Validation / Results

Three Key Observations

Observation 1: GDN memory dynamics are structured and stable across documents.

• GDN heads span multiple orders of magnitude in memory half-life (from a few tokens to thousands)
• Ordering of heads by half-life agrees at Spearman 0.99–0.997 across document pairs
• Static half-life from checkpoint matches measured one at Spearman 0.942
• Slow-head set overlaps global set at Jaccard 0.80

Observation 2: GDN write and forget signals anticipate future KV utility.

• All signal summaries are positively correlated with utility $U$ in each of nine text subsets
• Restricting effective write to slow heads doubles correlation: from +0.10 to +0.23
• Entries in the highest decile of slow-head $\beta\|v\|$ reach top utility decile with probability 18.2% (1.82× chance rate)

Observation 3: The relation is distributed and nonlinear.

Table 1: Predicting U from native GDN signals (mean over layers and heads). $\rho$: Spearman; Ov@10: top-10% overlap (%).

Main Results

Qwen3.8-27B: GATESIGHT is more accurate than SnapKV, TriAttention, and KeyDiff at every kept fraction on both SCBench and RULER-4096.

Comparison with KVzip:

Table 3: Comparison with KVzip: task-averaged accuracy (%); Mean: nine kept fractions.

At $p = 0.1$, GATESIGHT leads KVzip by 9.5 points on SCBench and 47.2 points on RULER-4096 for Qwen3.8-27B.

Scoring Cost

Table 2: Wall-clock scoring time (s) per context in the same vLLM pipeline, Qwen3.8-27B.

GATESIGHT scoring is 0.5%–2.6% of prefill time, while KVzip's second pass takes 1.5–2.2× the prefill time.

Allocation Ablation

Table 4: Allocation ablation on Qwen3.8-27B: accuracy (%) over three SCBench and three RULER tasks.

Theoretical and Practical Implications

Theoretical Implications

• Cross-memory information transfer: The paper demonstrates that recurrent memory (GDN) and attention memory (KV cache) in hybrid models share predictive information—the recurrent state "knows" what attention will need, suggesting these memory systems are not independent.
• Stable functional roles: GDN heads have fixed roles determined by weights rather than content, analogous to the static retrieval/streaming head partition found in attention layers (DuoAttention).
• Nonlinearity matters for tail recovery: While linear methods capture the overall trend of utility prediction, nonlinear models (MLPs) recover significantly more of the high-utility tail, which is crucial under aggressive pruning budgets.

Practical Implications

• Zero-overhead scoring: GATESIGHT adds only 0.5%–2.6% overhead relative to prefill, making it practical for serving long contexts.
• Physical memory release: The method physically compacts the cache (not just logical masking), returning freed blocks to the pool—enabling more concurrent sessions.
• No query needed: Works in the shared-context multi-query setting where the same prefilled context serves many future questions.
• Generalizable: The same training recipe works for both dense attention (Qwen3.8-27B) and sparse attention (Qwen3.8-Flash-Next) models, and extends to contexts up to 262K tokens despite training on ~1K-token documents.

Conclusion

GATESIGHT demonstrates that the recurrent memory of hybrid language models contains predictive signals about which attention KV entries will be needed, even before any query exists. The method:

1. Reads native GDN signals (write gate, decay, write magnitude) that are already computed during prefill—no additional forward passes needed
2. Uses per-head MLPs to capture the distributed, nonlinear relationship between recurrent signals and KV utility
3. Allocates budgets via the attention output gate, whose mean differs by more than an order of magnitude across layers

The evaluation shows GATESIGHT is the most accurate query-agnostic method at aggressive pruning levels (10% cache retention), leading KVzip by 9.5–47.2 points across all four model–benchmark pairs, with negligible scoring overhead.

Future directions could include: extending to other hybrid architectures, exploring adaptive window sizes for GDN layer selection, and investigating whether similar cross-memory prediction works for other recurrent layer types (e.g., Mamba).