# Still: Amortized KV Cache Compaction in a Single Forward Pass > Still is a per-layer Perceiver compactor that amortizes KV cache synthesis in one forward pass, beating selection baselines by 8-22 accuracy points across 8x-200x compression. - **Source:** [arXiv](https://arxiv.org/abs/2606.07878) - **Published:** 2026-09-05 - **Permalink:** https://picx.dev/p/VQ1nO6 - **Whiteboard:** https://picx.dev/p/VQ1nO6/image ## Summary # Still: Amortized KV Cache Compaction in a Single Forward Pass ## Summary (Overview) - **Key contribution**: Still is a per-layer Perceiver-based compactor that produces compact keys and values for KV caches in a single forward pass, combining amortized synthesis (learning the compactor once offline) with the expressiveness of latent-space synthesis rather than token selection. - **Architecture**: One small Perceiver module per transformer layer, with learned latent queries that cross-attend the full KV cache and project to compact keys/values, operating in a position-free frame via RoPE inverse-rotation. - **Performance**: On Qwen and Gemma models, Still dominates the speed–quality frontier across 8×–200× compression ratios and 8k–128k context lengths, exceeding the strongest amortized-selection baseline (KV-Distill) by 8–22 accuracy points on 16 of 18 matched-training RULER cells. - **Versatility**: The compact cache supports both multiple-choice QA and free-form summarization (HELMET multi_lexsum, LongBench v1), and enables iterative chunked compaction for long-horizon deployment. - **Key design insight**: Amortized synthesis avoids the subset-bound of token selection methods while eliminating per-context optimization, making iterative long-horizon compaction tractable. --- ## Introduction and Theoretical Foundation ### Background and Motivation The KV cache is the memory bottleneck of long-horizon language model deployment. As LLMs operate over multi-day coding agents, multi-turn tool use, and repository-scale reasoning, the cache becomes the binding constraint on system capabilities. Current options are all-or-nothing: - **Lossless cache**: grows linearly with context length - **Lossy alternatives** (fine-tuning, RAG, document summaries): abandon the structure and fidelity of the model's internal representations ### Design Space of KV Cache Compaction Existing methods fall along two axes: | Axis | Options | Examples | |------|---------|----------| | **Cache construction** | Selected from original tokens vs. synthesized via content-dependent combination | H2O, SnapKV (selection); Attention Matching, Cartridges (synthesis) | | **Computation timing** | Per-context at inference, amortized once offline, or trained-in during pretraining | Attention Matching (per-context); KV-Distill (amortized selection) | ### Theoretical Foundation The authors draw an analogy to representation learning: > "Amortized variational inference and sparse autoencoders both replace repeated per-instance optimization over a fixed structure with a learned encoder. In each case, the lesson has been to **learn the optimizer**. We apply the same move to KV-cache synthesis." This motivates the underexplored combination: **amortized synthesis of layer-wise KV caches for frozen pretrained models**. --- ## Methodology ### Problem Formulation Let $f_\theta$ be a frozen autoregressive transformer with $L$ layers, $H$ KV-heads per layer, head dimension $d$, and rotary position embeddings. Given a prefix of $T$ tokens, $f_\theta$ produces per-layer KV caches. Still learns a per-layer compactor: $$ g_\phi^{(\ell)}: (K^{(\ell,h)}, V^{(\ell,h)})_{h=1}^{H} \mapsto (C_k^{(\ell,h)}, C_v^{(\ell,h)})_{h=1}^{H}, $$ with $C_k^{(\ell,h)}, C_v^{(\ell,h)} \in \mathbb{R}^{t \times d}$ and $t \ll T$. The compact cache replaces the original prefix cache when running $f_\theta$ on subsequent tokens. ### Per-layer Perceiver Compactor Each transformer layer has its own Perceiver-style compactor with: - **Learned latent queries**: $Z \in \mathbb{R}^{H \times t \times d_\ell}$ (per-head banks, shared linear projections) - **Input**: per-head concatenation $X^{(h)} = [K^{(h)}; V^{(h)}] \in \mathbb{R}^{T \times 2d}$ with keys un-rotated into a position-free frame - **Processing**: $B$ pre-norm blocks with cross-attention (latents → X), latent self-attention, and optional FFN - **Output heads**: independent linear heads project final latent state to compact keys $C_k = Z_{out}W_{key}$ and values $C_v = Z_{out}W_{val}$ **Canonical configuration** (Qwen3-4B): $d_\ell = 256$, $B = 2$ blocks, ~50M parameters (~1% of base model) at $t = 128$. ### Position-Free Compaction Cached keys are RoPE-rotated, so the same content has different key vectors at different positions. Still operates in a **position-free frame**: 1. Cached keys are inverse-rotated before the compactor 2. The compactor uses its own RoPE inside cross-attention with latent queries at evenly spaced positions 3. Compact keys are re-rotated at chosen output positions before being written into the cache ### Training **Data**: Four-domain extractive multiple-choice QA dataset (Financial filings, Project Gutenberg, Legal, Code), ~120k items and ~1B context tokens at 8k context length. Questions are generated from random sub-chunks and verified by the frozen base model. **Loss**: Forward KL divergence from full-context teacher to compact-cache student, masked to answer-side tokens: $$ \mathcal{L} = \mathbb{E}_{(c,p,a) \sim \mathcal{D}} \sum_{i \in \mathrm{ans}} \mathrm{KL}(f_\theta(\cdot | c, p, a_{ "Amortization makes cache compaction tractable at long context; in the regimes we evaluate, synthesis gives the compact state enough expressive capacity to be worth amortizing." ### Limitations - **Not lossless**: full-context inference remains substantially better in hardest 128k settings - **No free extrapolation**: iterative compaction degrades outside trained horizon; 1M-token deployment requires proportionally longer training horizons - **Fixed compression ratio only**: retained cache grows linearly at rate $1/c$, not O(1) recurrent memory - **Weak on exact retrieval**: needle-style tasks remain challenging even when semantic evidence is preserved - **Per-checkpoint training required**: each base model needs its own compactor ### Future Directions - Curriculum schedules over horizon - Training Still directly under the recurrence - Constant-budget variants that reuse or merge compact slots in place - Training mixtures covering the task surfaces on which the compact cache is expected to operate --- _Markdown view of https://picx.dev/p/VQ1nO6, served by PicX — AI-generated visual whiteboard summaries of research papers._