# Who Should Score the Cache? Mid-Depth Layers Are the Best Judges of KV-Cache Eviction

> The depth of the scoring layer in KV-cache eviction is a critical ignored hyperparameter, with mid-depth scoring beating default methods by up to 43 points on models up to 14B.

- **Source:** [arXiv](https://arxiv.org/abs/2611.02230)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/vkYAkf
- **Whiteboard:** https://picx.dev/p/vkYAkf/image

## Summary

## Summary (Overview)

- **Core finding**: The depth of the scoring layer in KV-cache eviction governs eviction quality across 21 models (0.5B–109B). Accuracy peaks at mid-depth (0.4–0.75 of network depth), collapses at shallow layers, and degrades at the last layer.
- **Predictive model**: A parameter-free "survival model" computes answer survival probability from prefill attention, reproducing measured depth curves and selecting the optimal scoring layer on 14 of 18 per-layer models.
- **Improved SnapKV**: Combining mid-depth scoring with a single shared token set across heads beats per-head SnapKV by +18 to +43 percentage points on models up to 14B under tight budgets.
- **Shared-cache penalty**: On shared-cache architectures (e.g., Gemma-4-E2B), the default writer-layer scoring is suboptimal when the writer is shallow, costing up to 28 points vs. a mid-depth reader at 10% budget.
- **Scale limits**: Gains are significant up to 14B; from 32B upward, the default already performs close to the best mid-depth layer, though wrong scorers remain catastrophic at all scales.

---

## Introduction and Theoretical Foundation

The KV cache is the dominant memory cost in long-context inference. Eviction methods rank cached tokens by attention scores and keep the top-k, dropping the rest. Existing methods (e.g., SnapKV, H2O, PyramidKV) all score each cache with the layer that owns it—layer 7 judges what layer 7 keeps. **No prior method varies or reports the scoring layer as a hyperparameter.**

The paper's central insight is that **a layer judges tokens well only once it has used the question to find the answer**, which requires several layers of computation. Shallow layers haven't done this retrieval work yet; the last layer has moved on to generating output. This produces a characteristic depth curve: a cliff on the shallow side, a peak at mid-depth, and a decline toward the end.

The problem is most acute in **shared-cache models** (CLA, YOCO, Gemma-3n/4), where one cache serves multiple reader layers and scores come by construction from the cache's *writer*—its shallowest reader.

**Key related work:**
- **Attention-scored eviction**: SnapKV, H2O, PyramidKV, CAKE, etc. (all use own-layer scoring)
- **Mid-layer token selection**: GemFilter (closest prior work, in recompression setting), ASL, FastKV, LazyLLM
- **Cross-layer KV sharing**: CLA, YOCO, Gemma-3n/4 line
- **Per-head differences**: RazorAttention, DuoAttention, HeadKV

---

## Methodology

### Scoring Convention
Importance of context token $j$ under scorer layer $\ell$ is the **mean softmax attention** that a window of queries places on token $j$, averaged over heads. Two windows are used:
- **Question-window**: queries from the question block (for depth curves and shared-cache results)
- **kvpress window**: last 64 prefilled tokens (for baseline comparisons)

### Eviction Procedure
- Budget $b$ keeps $k = \lfloor bT_c \rfloor$ of a prompt's $T_c$ context tokens in every layer's KV cache.
- Always-kept tokens: sink (first 4) and recency window (last 32) under question window.
- Compression happens **after prefill**, so savings apply to the cache that persists through decode.
- Evicted positions are masked with $-\infty$ additive mask (mathematically equivalent to deleting rows, preserves RoPE positions).

### Models and Benchmarks
- **21 models**: 18 per-layer cache + 3 shared-cache (Gemma-4-E2B, Gemma-4-E4B, Gemma-3n-E2B)
- **Primary set**: Qwen2.5-{0.5B, 1.5B, 7B}, SmolLM2-1.7B, Llama-3.2-3B, Llama-3.1-8B, Gemma-4-E2B
- **Benchmarks**: RULER (4K–32K), LongBench v1, DocFinQA (financial filings)
- **Statistics**: Two-stage cluster bootstraps (resample tasks, then prompts), n=200 (25 per task), exact sign-flip tests for primary comparisons

---

## Empirical Validation / Results

### The Depth Curve (Figure 1)

The central result: accuracy vs. scorer depth follows the same curve on every model—a shallow-side cliff, a peak at 0.4–0.75 depth, and a decline toward the last layer. The default (own-layer) loses to the best mid-depth layer on all seven primary models, significantly on four.

**Key quantitative results:**
- Shallow scoring is catastrophic: **−25 to −56 percentage points** vs. default
- Best mid-depth gains: +11.8pp (SmolLM2), +26.6pp (Qwen2.5-1.5B), +31.4pp (Qwen2.5-0.5B)
- Shared-cache: best reader recovers **+36.7pp** over the writer

### Improved SnapKV (Table 1)

| Scoring method | 1.5B @15% | 1.5B @10% | SmolLM2 @15% | 7B @10% | 14B @5% |
|---|---|---|---|---|---|
| No eviction (ceiling) | 91.1 | 90.8 | 76.4 | 93.0 | 91.7 |
| H2O | 18.4 | 16.6 | 8.8 | 21.6 | 21.7 |
| SnapKV (per-head, own layer) | 53.0 | 39.2 | 28.4 | 57.4 | 57.6 |
| AdaKV | 57.8 | 42.8 | 30.9 | 53.9 | 53.6 |
| PyramidKV | 33.1 | 22.6 | 20.2 | 28.0 | 34.8 |
| Own layer, single set | 63.1 | 49.5 | 48.5 | 77.1 | 56.0 |
| Mid-layer, per-head | 84.7 | 75.1 | 47.0 | 87.2 | 71.6 |
| **Mid-layer, single set (ours)** | **87.6** | **82.0** | **63.4** | **91.9** | **75.9** |
| **Δ (ours − SnapKV)** | **+34.6** | **+42.8** | **+34.9** | **+34.6** | **+18.3** |

The combination recovers **83–99%** of uncompressed accuracy where SnapKV keeps only 37–63%.

### The Survival Model

The probability that a gold answer survives the top-k cut:

$$\hat{S}_{\ell} = \frac{1}{n} \sum_{\text{prompts}} \frac{1}{|G|} \sum_{g \in G} \mathbf{1}\{\exists o \in O_g: o \subseteq K_{\ell}\}$$

where $G$ is the prompt's gold set and $O_g$ the token-index sets of gold $g$'s occurrences. Everything comes from prefill attention of the uncompressed model—**no generation, no adjustable constants**.

**Predictive performance:**
- Spearman correlation 0.90–1.00 with measured curves
- Mean absolute error 2.7–7.4pp
- Reproduces all five shared-cache readers' accuracies within 5.1pp (r=0.998)
- Pick is best measured layer on 14/18 models, never significantly below on the other 4

### Mechanism: The Question Matters

Without the question, every layer is nearly equally blind to the answer. With it, attention on the answer builds over depth and peaks at mid-depth:
- Gemma-4 readers place **2.4×** as much question-driven mass on the answer as the writer
- Without the question, the gap shrinks by a factor of **6.5**

### Scale and Task Effects (Table 2)

| Model | Budget | Best−own | Worst−own |
|---|---|---|---|
| Qwen2.5-14B | 5% | +17.7 [+5.6, +30.9] | −50.7 |
| Qwen2.5-32B | 5% | +3.9 [−4.2, +11.4] | −44.6 |
| Qwen3-0.6B | 15% | +20.4 [+5.1, +37.4] | −45.4 |
| Qwen3-14B | 5% | −2.1 [−6.9, +2.4] | −42.0 |
| Llama-3.2-1B | 10% | +24.4 [+5.7, +45.6] | −46.0 |
| Llama-3.1-70B | 5% | +2.4 [−8.5, +16.4] | −42.9 |
| Llama-4-Scout (109B) | 3% | +7.0 [−2.7, +18.1] | −50.0 |

**Key patterns**: Significant gains persist to 14B; undetectable from 32B up. The **cost of a wrong scorer never shrinks** (−25 to −56pp at all scales).

### Shared-Cache Penalty (Table 3)

On Gemma-4-E2B (writer at layer 14 of 35, depth 0.41):

| Scorer | Accuracy (%) | Δ vs. writer |
|---|---|---|
| No eviction (ceiling) | 90.3 | − |
| Writer, L14 (default) | 23.1 | − |
| Reader L19 | 51.2 | +28.2 [+13.7, +48.0] |
| Reader L24 | 48.3 | +25.2 [+9.5, +43.6] |
| Rank consensus (5 readers) | 45.3 | +22.2 [+7.6, +39.9] |

The penalty follows the writer's depth: Gemma-4-E4B (writer at depth 0.56) shows no significant penalty (+6.3pp, CI includes zero).

### Where the Advantage Does NOT Hold

- **LongBench (open-book QA)**: No mid-depth layer beats the default; shallow and last layers still lose 5–6 F1
- **Larger models (≥32B)**: Default already near-optimal
- **Newer Qwen3 family**: Margin falls with capability (significant at 0.6B, gone at 14B)

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **The depth curve as a universal phenomenon**: The finding that scoring quality follows a single-peaked curve across 21 models from 4 labs suggests a fundamental property of transformer information flow—retrieval happens at mid-depth, not at the output layer.

2. **The survival model as a mechanism**: Eviction quality is explained by top-k survival probability of answer tokens, computable from prefill attention alone. This provides a *predictive* theory, not just a descriptive one.

3. **Question-driven attention**: The mechanism test (removing the question flattens the depth signal) confirms that mid-depth layers are where question-conditioned retrieval happens.

### Practical Implications

1. **Immediate improvements with zero cost**: Selecting the scoring layer from calibration prompts requires no extra forward pass and adds <1% to prefill time. Gains of +18 to +43pp over SnapKV are achievable on models up to 14B.

2. **Architectural guidance for shared-cache designs**: The writer's depth should be chosen at mid-depth (0.5–0.7), not at the shallowest reader. This is actionable for future CLA-style architectures.

3. **Budget savings**: At 1% budget, Llama-3.1-8B's 32K decode-resident cache shrinks from 4.0GB to 45MB with 1.4× faster decode.

4. **Caution for deployment**: A wrong scorer loses 25–56pp at every scale, so deployments should verify the scoring layer rather than assume the default is optimal.

---

## Conclusion

The paper establishes that **the depth of the scoring layer is a critical, previously ignored hyperparameter in KV-cache eviction**. The depth curve—shallow cliff, mid-depth peak, end-of-network decline—is universal across 21 models, explained by the network's need to use the question to find the answer before it can judge token importance.

**Main takeaways:**
1. Score every cache from a mid-depth layer selected by the survival model, not from each layer's own cache.
2. Combine mid-depth scoring with a single shared token set across heads for best results.
3. Shared-cache architectures should place writers at mid-depth or allow mid-depth readers to score.

**Open problems:**
1. Task-family-dependent optimal scorers suggest a selector that detects task family at prefill could beat any single layer.
2. The writer-depth explanation rests on one sibling pair (Gemma-4-E2B vs. E4B); training the same model with a deeper writer would test it directly.
3. Question-free scoring signals are needed for prefix-cache scenarios where the question isn't available at scoring time.

---

_Markdown view of https://picx.dev/p/vkYAkf, served by PicX — AI-generated visual whiteboard summaries of research papers._
