Still: Amortized KV Cache Compaction in a Single Forward Pass

Summary (Overview)

• Key contribution: Still is a per-layer Perceiver-based compactor that produces compact keys and values for KV caches in a single forward pass, combining amortized synthesis (learning the compactor once offline) with the expressiveness of latent-space synthesis rather than token selection.
• Architecture: One small Perceiver module per transformer layer, with learned latent queries that cross-attend the full KV cache and project to compact keys/values, operating in a position-free frame via RoPE inverse-rotation.
• Performance: On Qwen and Gemma models, Still dominates the speed–quality frontier across 8×–200× compression ratios and 8k–128k context lengths, exceeding the strongest amortized-selection baseline (KV-Distill) by 8–22 accuracy points on 16 of 18 matched-training RULER cells.
• Versatility: The compact cache supports both multiple-choice QA and free-form summarization (HELMET multi_lexsum, LongBench v1), and enables iterative chunked compaction for long-horizon deployment.
• Key design insight: Amortized synthesis avoids the subset-bound of token selection methods while eliminating per-context optimization, making iterative long-horizon compaction tractable.

---

Introduction and Theoretical Foundation

Background and Motivation

The KV cache is the memory bottleneck of long-horizon language model deployment. As LLMs operate over multi-day coding agents, multi-turn tool use, and repository-scale reasoning, the cache becomes the binding constraint on system capabilities. Current options are all-or-nothing:

• Lossless cache: grows linearly with context length
• Lossy alternatives (fine-tuning, RAG, document summaries): abandon the structure and fidelity of the model's internal representations

Design Space of KV Cache Compaction

Existing methods fall along two axes:

Theoretical Foundation

The authors draw an analogy to representation learning:

"Amortized variational inference and sparse autoencoders both replace repeated per-instance optimization over a fixed structure with a learned encoder. In each case, the lesson has been to learn the optimizer. We apply the same move to KV-cache synthesis."

This motivates the underexplored combination: amortized synthesis of layer-wise KV caches for frozen pretrained models.

---

Methodology

Problem Formulation

Let $f_\theta$ be a frozen autoregressive transformer with $L$ layers, $H$ KV-heads per layer, head dimension $d$, and rotary position embeddings. Given a prefix of $T$ tokens, $f_\theta$ produces per-layer KV caches. Still learns a per-layer compactor:

$$g_\phi^{(\ell)}: (K^{(\ell,h)}, V^{(\ell,h)})_{h=1}^{H} \mapsto (C_k^{(\ell,h)}, C_v^{(\ell,h)})_{h=1}^{H},$$

with $C_k^{(\ell,h)}, C_v^{(\ell,h)} \in \mathbb{R}^{t \times d}$ and $t \ll T$. The compact cache replaces the original prefix cache when running $f_\theta$ on subsequent tokens.

Per-layer Perceiver Compactor

Each transformer layer has its own Perceiver-style compactor with:

• Learned latent queries: $Z \in \mathbb{R}^{H \times t \times d_\ell}$ (per-head banks, shared linear projections)
• Input: per-head concatenation $X^{(h)} = [K^{(h)}; V^{(h)}] \in \mathbb{R}^{T \times 2d}$ with keys un-rotated into a position-free frame
• Processing: $B$ pre-norm blocks with cross-attention (latents → X), latent self-attention, and optional FFN
• Output heads: independent linear heads project final latent state to compact keys $C_k = Z_{out}W_{key}$ and values $C_v = Z_{out}W_{val}$

Canonical configuration (Qwen3-4B): $d_\ell = 256$, $B = 2$ blocks, ~50M parameters (~1% of base model) at $t = 128$.

Position-Free Compaction

Cached keys are RoPE-rotated, so the same content has different key vectors at different positions. Still operates in a position-free frame:

1. Cached keys are inverse-rotated before the compactor
2. The compactor uses its own RoPE inside cross-attention with latent queries at evenly spaced positions
3. Compact keys are re-rotated at chosen output positions before being written into the cache

Training

Data: Four-domain extractive multiple-choice QA dataset (Financial filings, Project Gutenberg, Legal, Code), ~120k items and ~1B context tokens at 8k context length. Questions are generated from random sub-chunks and verified by the frozen base model.

Loss: Forward KL divergence from full-context teacher to compact-cache student, masked to answer-side tokens:

$$\mathcal{L} = \mathbb{E}_{(c,p,a) \sim \mathcal{D}} \sum_{i \in \mathrm{ans}} \mathrm{KL}(f_\theta(\cdot | c, p, a_{<i}) \| f_\theta(\cdot | g_\phi(c), p, a_{<i}))$$

• KL evaluated on top 200 teacher-vocabulary tokens with gold answer token forced into support
• Answer tokens include rationale trace + MCQ answer token
• Only compactor parameters $\phi$ are trainable; $\theta$ remains frozen

Optimization: AdamW at learning rate $4 \times 10^{-5}$, effective batch 32, 1500 training steps on 8× H200.

Iterative Chunked Compaction

For long-horizon settings, Still supports a recurrent schedule with fixed local compression ratio $c$:

• Pass 0: prefill first $2ct$ tokens, compact first $ct$ KV entries to $t$ entries
• Each subsequent pass: prefill next $ct$ tokens conditioned on $N$ compacted chunks + one raw chunk (lookahead buffer)
• Retained cache after $T$ tokens: $T/c + ct$ entries (linear at compression rate $1/c$)

---

Empirical Validation / Results

3.1 Speed–Quality Frontier

Across compression ratios 8×–200× and contexts 8k–64k on Qwen3-4B:

• Selection methods (H2O, SnapKV, StreamingLLM): degrade under tight budgets
• Per-context synthesis (Attention Matching): accurate at low compression but slow
• Prompted summarization: preserves some content but passes through lossy text bottleneck
• Still: only method remaining accurate AND fast across the full sweep

A 256k-token single-pass Still at 100× compression reaches 40.7% compact accuracy on QuALITY-concat (vs. 55% full-context, 22% no-context).

3.2 Transfer Across Model Scales and Architectures

• Qwen3 dense family (4B–32B): compact cache stays usefully inside the no-context-to-full-context band at every size
• Qwen3-30B-A3B MoE: transfers without modification (compact-cache interface determined by attention-layer KV geometry)
• Gemma-3 4B (mixed sliding-window/global attention): transfer requires compacting only global-attention layers; matches/exceeds every fixed-cache baseline at 16k

3.3 Long-Context Generalization vs. KV-Distill

Matched training (RULER): Still exceeds KV-Distill by 8–22 accuracy points in 16 of 18 cells. The two exceptions (64k/K=82 and 128k/K=82) sit within seed-level noise.

Zero-shot transfer (mixed-domain MCQ → RULER): Same qualitative story with smaller margins.

3.4 Iterative Compaction at Fixed Compression Ratio

On Long-MCQ, with checkpoints trained at 8k/16k/32k iterative horizons deployed at 32k–128k:

Key finding: Failure is graceful only within the trained range — training horizon is the binding constraint, not cache budget.

3.5 Free-Form Summarization

HELMET multi_lexsum (utilization scores, higher = better):

Still recovers 74–95% of full-context gain across 8k–64k, and 59% at 128k.

LongBench v1 (GovReport/QMSum, 16k context, K=1024): Still wins 300/500 pairwise comparisons vs. KV-Distill, +5.4 mean judge score improvement.

---

Theoretical and Practical Implications

Design Space Insights

1. Amortized synthesis is the winning combination: Selection methods are subset-bound (quality ceiling falls with input information density); per-context synthesis is operationally impractical for iterative use. Still demonstrates that a learned compactor can achieve synthesis-level expressiveness with selection-level speed.

2. Synthesis is worth amortizing: The convergence of Still and DeepSeek-V4 on synthesis (rather than selection) is independent evidence for the design choice, despite V4 baking compression into pretraining.

3. The compact cache as working memory: Results support interpreting the compact state as a compressed working memory the model can attend to natively, without lossless recall of every original token.

Operational Implications

• Single forward pass compaction: Enables mid-trajectory compaction in long-horizon settings where per-context optimization costs compound
• Plug-in compatibility: Works with frozen open-weight checkpoints without modifying base model weights
• Transferability: Same recipe works across model scales (4B–32B), architectures (dense, MoE, mixed-attention), and context lengths

---

Conclusion

Main Takeaways

Still is a learned KV-state transform that is:

• Lightweight enough to call repeatedly during inference
• Expressive enough to beat token selection at high information density
• Stable enough to transfer across model scales and attention architectures
• Useful enough to support both answer selection and free-form summarization

"Amortization makes cache compaction tractable at long context; in the regimes we evaluate, synthesis gives the compact state enough expressive capacity to be worth amortizing."

Limitations

• Not lossless: full-context inference remains substantially better in hardest 128k settings
• No free extrapolation: iterative compaction degrades outside trained horizon; 1M-token deployment requires proportionally longer training horizons
• Fixed compression ratio only: retained cache grows linearly at rate $1/c$, not O(1) recurrent memory
• Weak on exact retrieval: needle-style tasks remain challenging even when semantic evidence is preserved
• Per-checkpoint training required: each base model needs its own compactor

Future Directions

• Curriculum schedules over horizon
• Training Still directly under the recurrence
• Constant-budget variants that reuse or merge compact slots in place
• Training mixtures covering the task surfaces on which the compact cache is expected to operate