Summary (Overview)
- Core finding: The depth of the scoring layer in KV-cache eviction governs eviction quality across 21 models (0.5B–109B). Accuracy peaks at mid-depth (0.4–0.75 of network depth), collapses at shallow layers, and degrades at the last layer.
- Predictive model: A parameter-free "survival model" computes answer survival probability from prefill attention, reproducing measured depth curves and selecting the optimal scoring layer on 14 of 18 per-layer models.
- Improved SnapKV: Combining mid-depth scoring with a single shared token set across heads beats per-head SnapKV by +18 to +43 percentage points on models up to 14B under tight budgets.
- Shared-cache penalty: On shared-cache architectures (e.g., Gemma-4-E2B), the default writer-layer scoring is suboptimal when the writer is shallow, costing up to 28 points vs. a mid-depth reader at 10% budget.
- Scale limits: Gains are significant up to 14B; from 32B upward, the default already performs close to the best mid-depth layer, though wrong scorers remain catastrophic at all scales.
Introduction and Theoretical Foundation
The KV cache is the dominant memory cost in long-context inference. Eviction methods rank cached tokens by attention scores and keep the top-k, dropping the rest. Existing methods (e.g., SnapKV, H2O, PyramidKV) all score each cache with the layer that owns it—layer 7 judges what layer 7 keeps. No prior method varies or reports the scoring layer as a hyperparameter.
The paper's central insight is that a layer judges tokens well only once it has used the question to find the answer, which requires several layers of computation. Shallow layers haven't done this retrieval work yet; the last layer has moved on to generating output. This produces a characteristic depth curve: a cliff on the shallow side, a peak at mid-depth, and a decline toward the end.
The problem is most acute in shared-cache models (CLA, YOCO, Gemma-3n/4), where one cache serves multiple reader layers and scores come by construction from the cache's writer—its shallowest reader.
Key related work:
- Attention-scored eviction: SnapKV, H2O, PyramidKV, CAKE, etc. (all use own-layer scoring)
- Mid-layer token selection: GemFilter (closest prior work, in recompression setting), ASL, FastKV, LazyLLM
- Cross-layer KV sharing: CLA, YOCO, Gemma-3n/4 line
- Per-head differences: RazorAttention, DuoAttention, HeadKV
Methodology
Scoring Convention
Importance of context token under scorer layer is the mean softmax attention that a window of queries places on token , averaged over heads. Two windows are used:
- Question-window: queries from the question block (for depth curves and shared-cache results)
- kvpress window: last 64 prefilled tokens (for baseline comparisons)
Eviction Procedure
- Budget keeps of a prompt's context tokens in every layer's KV cache.
- Always-kept tokens: sink (first 4) and recency window (last 32) under question window.
- Compression happens after prefill, so savings apply to the cache that persists through decode.
- Evicted positions are masked with additive mask (mathematically equivalent to deleting rows, preserves RoPE positions).
Models and Benchmarks
- 21 models: 18 per-layer cache + 3 shared-cache (Gemma-4-E2B, Gemma-4-E4B, Gemma-3n-E2B)
- Primary set: Qwen2.5-{0.5B, 1.5B, 7B}, SmolLM2-1.7B, Llama-3.2-3B, Llama-3.1-8B, Gemma-4-E2B
- Benchmarks: RULER (4K–32K), LongBench v1, DocFinQA (financial filings)
- Statistics: Two-stage cluster bootstraps (resample tasks, then prompts), n=200 (25 per task), exact sign-flip tests for primary comparisons
Empirical Validation / Results
The Depth Curve (Figure 1)
The central result: accuracy vs. scorer depth follows the same curve on every model—a shallow-side cliff, a peak at 0.4–0.75 depth, and a decline toward the last layer. The default (own-layer) loses to the best mid-depth layer on all seven primary models, significantly on four.
Key quantitative results:
- Shallow scoring is catastrophic: −25 to −56 percentage points vs. default
- Best mid-depth gains: +11.8pp (SmolLM2), +26.6pp (Qwen2.5-1.5B), +31.4pp (Qwen2.5-0.5B)
- Shared-cache: best reader recovers +36.7pp over the writer
Improved SnapKV (Table 1)
| Scoring method | 1.5B @15% | 1.5B @10% | SmolLM2 @15% | 7B @10% | 14B @5% |
|---|---|---|---|---|---|
| No eviction (ceiling) | 91.1 | 90.8 | 76.4 | 93.0 | 91.7 |
| H2O | 18.4 | 16.6 | 8.8 | 21.6 | 21.7 |
| SnapKV (per-head, own layer) | 53.0 | 39.2 | 28.4 | 57.4 | 57.6 |
| AdaKV | 57.8 | 42.8 | 30.9 | 53.9 | 53.6 |
| PyramidKV | 33.1 | 22.6 | 20.2 | 28.0 | 34.8 |
| Own layer, single set | 63.1 | 49.5 | 48.5 | 77.1 | 56.0 |
| Mid-layer, per-head | 84.7 | 75.1 | 47.0 | 87.2 | 71.6 |
| Mid-layer, single set (ours) | 87.6 | 82.0 | 63.4 | 91.9 | 75.9 |
| Δ (ours − SnapKV) | +34.6 | +42.8 | +34.9 | +34.6 | +18.3 |
The combination recovers 83–99% of uncompressed accuracy where SnapKV keeps only 37–63%.
The Survival Model
The probability that a gold answer survives the top-k cut:
where is the prompt's gold set and the token-index sets of gold 's occurrences. Everything comes from prefill attention of the uncompressed model—no generation, no adjustable constants.
Predictive performance:
- Spearman correlation 0.90–1.00 with measured curves
- Mean absolute error 2.7–7.4pp
- Reproduces all five shared-cache readers' accuracies within 5.1pp (r=0.998)
- Pick is best measured layer on 14/18 models, never significantly below on the other 4
Mechanism: The Question Matters
Without the question, every layer is nearly equally blind to the answer. With it, attention on the answer builds over depth and peaks at mid-depth:
- Gemma-4 readers place 2.4× as much question-driven mass on the answer as the writer
- Without the question, the gap shrinks by a factor of 6.5
Scale and Task Effects (Table 2)
| Model | Budget | Best−own | Worst−own |
|---|---|---|---|
| Qwen2.5-14B | 5% | +17.7 [+5.6, +30.9] | −50.7 |
| Qwen2.5-32B | 5% | +3.9 [−4.2, +11.4] | −44.6 |
| Qwen3-0.6B | 15% | +20.4 [+5.1, +37.4] | −45.4 |
| Qwen3-14B | 5% | −2.1 [−6.9, +2.4] | −42.0 |
| Llama-3.2-1B | 10% | +24.4 [+5.7, +45.6] | −46.0 |
| Llama-3.1-70B | 5% | +2.4 [−8.5, +16.4] | −42.9 |
| Llama-4-Scout (109B) | 3% | +7.0 [−2.7, +18.1] | −50.0 |
Key patterns: Significant gains persist to 14B; undetectable from 32B up. The cost of a wrong scorer never shrinks (−25 to −56pp at all scales).
Shared-Cache Penalty (Table 3)
On Gemma-4-E2B (writer at layer 14 of 35, depth 0.41):
| Scorer | Accuracy (%) | Δ vs. writer |
|---|---|---|
| No eviction (ceiling) | 90.3 | − |
| Writer, L14 (default) | 23.1 | − |
| Reader L19 | 51.2 | +28.2 [+13.7, +48.0] |
| Reader L24 | 48.3 | +25.2 [+9.5, +43.6] |
| Rank consensus (5 readers) | 45.3 | +22.2 [+7.6, +39.9] |
The penalty follows the writer's depth: Gemma-4-E4B (writer at depth 0.56) shows no significant penalty (+6.3pp, CI includes zero).
Where the Advantage Does NOT Hold
- LongBench (open-book QA): No mid-depth layer beats the default; shallow and last layers still lose 5–6 F1
- Larger models (≥32B): Default already near-optimal
- Newer Qwen3 family: Margin falls with capability (significant at 0.6B, gone at 14B)
Theoretical and Practical Implications
Theoretical Contributions
-
The depth curve as a universal phenomenon: The finding that scoring quality follows a single-peaked curve across 21 models from 4 labs suggests a fundamental property of transformer information flow—retrieval happens at mid-depth, not at the output layer.
-
The survival model as a mechanism: Eviction quality is explained by top-k survival probability of answer tokens, computable from prefill attention alone. This provides a predictive theory, not just a descriptive one.
-
Question-driven attention: The mechanism test (removing the question flattens the depth signal) confirms that mid-depth layers are where question-conditioned retrieval happens.
Practical Implications
-
Immediate improvements with zero cost: Selecting the scoring layer from calibration prompts requires no extra forward pass and adds <1% to prefill time. Gains of +18 to +43pp over SnapKV are achievable on models up to 14B.
-
Architectural guidance for shared-cache designs: The writer's depth should be chosen at mid-depth (0.5–0.7), not at the shallowest reader. This is actionable for future CLA-style architectures.
-
Budget savings: At 1% budget, Llama-3.1-8B's 32K decode-resident cache shrinks from 4.0GB to 45MB with 1.4× faster decode.
-
Caution for deployment: A wrong scorer loses 25–56pp at every scale, so deployments should verify the scoring layer rather than assume the default is optimal.
Conclusion
The paper establishes that the depth of the scoring layer is a critical, previously ignored hyperparameter in KV-cache eviction. The depth curve—shallow cliff, mid-depth peak, end-of-network decline—is universal across 21 models, explained by the network's need to use the question to find the answer before it can judge token importance.
Main takeaways:
- Score every cache from a mid-depth layer selected by the survival model, not from each layer's own cache.
- Combine mid-depth scoring with a single shared token set across heads for best results.
- Shared-cache architectures should place writers at mid-depth or allow mid-depth readers to score.
Open problems:
- Task-family-dependent optimal scorers suggest a selector that detects task family at prefill could beat any single layer.
- The writer-depth explanation rests on one sibling pair (Gemma-4-E2B vs. E4B); training the same model with a deeper writer would test it directly.
- Question-free scoring signals are needed for prefix-cache scenarios where the question isn't available at scoring time.
Related papers
- BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
BuggyMLE generates 6,600+ bug-injected ML repositories and a structural complexity metric (CCI) predicting solve rates with R²=0.951, enabling difficulty-graded benchmarks where a 9B model rivals frontier agents.
- InfuseLoop: Cross-Trajectory Latent Injection for Scalable Parallel Looped Transformers
InfuseLoop injects cross-trajectory latent states into parallel looped transformers, eliminating training spikes and enabling quality gains at 8 loops with minimal overhead.
- Do Halting Heads Really Save Compute in Looped Language Models?
Halting heads in looped language models save far less compute than reported because mixture-trained heads rarely stop tokens and batching overhead eliminates most remaining gains.