# BellowsKV: One Latent for Many Layers, Expanded on Demand

> BellowsKV compresses KV caches 13-20x with near-dense accuracy by sharing a full-rank latent across layers and buying compression from bits rather than rank reduction.

- **Source:** [arXiv](https://arxiv.org/abs/2610.35023)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/9pIQ8m
- **Whiteboard:** https://picx.dev/p/9pIQ8m/image

## Summary

# BellowsKV: One Latent for Many Layers, Expanded on Demand

## Summary

- **Core Innovation**: BellowsKV is a training-free KV cache compression method that maps hidden states of $n$ adjacent layers to one shared latent vector per token, from which each layer reconstructs its own keys and values through layer-specific up-projections. This is a novel cross-layer approach that achieves compression ratios an order of magnitude beyond existing methods.
- **Key Insight**: The paper identifies three critical observations: (1) adjacent layers have highly similar hidden states but nearly orthogonal keys/values, (2) the shared latent representation is naturally robust to quantization without needing rotation, and (3) dimension reduction of the shared latent causes catastrophic accuracy loss while quantization does not.
- **Two Compression Operators**: BellowsKV-Quant stores the latent in low-precision (INT8/INT4/INT3) with negligible accuracy loss, and BellowsKV-Evict makes a single eviction decision per block of layers, exploiting the high agreement (Spearman correlation 0.76–0.80) between layers' token importance rankings.
- **SOTA Results**: BellowsKV retains ~99% of full-cache accuracy at 3.6× compression and 97–98% at 13–14× compression across Llama-3.1-8B, Mistral-7B, and Qwen2.5-7B—far surpassing existing cross-layer methods like xKV (at most 84% at 16×) and CommonKV (54% at 16×).
- **Architecture Agnostic**: The method works on standard pretrained models without retraining, requiring only a one-time offline calibration (~20 minutes per model, no gradients) and a small per-sequence "whisper" correction.

## Introduction and Theoretical Foundation

### Background
The paper addresses the growing bottleneck of KV cache memory in LLM serving. As context windows expand from 4K to 128K+ tokens and agents engage in long-horizon tasks, the KV cache has become a memory rival to model weights:

> On Llama-3.1-8B a single 131K-token sequence carries a 16.8 GB cache in 16-bit precision, slightly more than the model itself.

A batch of eight 131K-token sequences holds 134 GB of cache against 16 GB of weights—making compression by 4× (where most methods operate) insufficient; the paper targets an order of magnitude.

### Existing Approaches and Their Limits
Three single-layer compression families exist, each with inherent limitations:
1. **Quantization**: Degrades sharply below 2 bits, requiring complex outlier handling (rotation, sparse outliers).
2. **Token Eviction**: Irreversible—dropped tokens cannot be recovered even if later needed.
3. **Channel/rank reduction**: Removes components of keys/values that attention still uses.

A recent position paper argues that at matched memory budget, quantization dominates rank reduction (Salfati, 2026).

### The Cross-Layer Opportunity
Adjacent layers compute caches from nearly identical hidden states yet store separate caches. Prior cross-layer methods (xKV, CommonKV) work at 4× but degrade sharply at 16×. The paper investigates why and proposes a principled solution based on three measurements.

### Theoretical Foundation
The paper establishes three key observations through systematic measurement (Figure 2):

**Observation 1: Adjacent layers have similar hidden states but dissimilar caches.** Per-token cosine similarity between adjacent-layer keys/values is near zero, while hidden-state similarity reaches 0.87 (Llama) and 0.82 (Qwen). Substituting one shared key across even two layers catastrophically inflates perplexity. The redundancy lies in cache *inputs*, not caches themselves.

**Observation 2: Shared representation has no dominant outlier channels.** Uniform quantization fails when few channels dominate dynamic range. The shared latent, being a linear combination of all layers' channels, is naturally flat—INT4 error drops from 0.156 (raw cache) to 0.10 (shared latent) without rotation.

**Observation 3: Layers agree on token importance.** Spearman correlation between adjacent layers' token importance scores ranges 0.76–0.80, increasing with depth—enabling a single eviction decision per block.

## Methodology

### Formal Framework
For a decoder-only transformer with $L$ layers and hidden width $d$, at layer $\ell$ and position $t$:
$$k_{\ell,t} = W^K_\ell h_{\ell,t}, \quad v_{\ell,t} = W^V_\ell h_{\ell,t}$$

where $W^K_\ell, W^V_\ell \in \mathbb{R}^{d \times p}$.

### One Latent Per Block
For a block $B$ of $n$ adjacent layers, BellowsKV replaces the block's cached keys and values with a latent:
$$c_t = W^T_\downarrow h_{B,t}, \quad \hat{k}_{\ell_i,t} = W^{(i)T}_\uparrow c_t, \quad h_{B,t} = [h_{\ell_1,t}; \ldots; h_{\ell_n,t}] \in \mathbb{R}^{nd}$$

where $W_\downarrow \in \mathbb{R}^{nd \times R}$ is the block-shared down-projection and $W^{(i)}_\uparrow \in \mathbb{R}^{R \times p}$ is layer $\ell_i$'s up-projection.

### Calibration Objective
The projections minimize reconstruction error over calibration data:
$$J(W_\downarrow, W_\uparrow) = \mathbb{E}_H \|HW - HW_\downarrow W_\uparrow\|^2_F$$

where $G = \mathbb{E}[H^TH] \in \mathbb{R}^{nd \times nd}$ is the second-moment matrix. The closed-form solution uses $M = G^{1/2}W = U\Sigma V^T$ (thin SVD):
$$W^*_\downarrow = G^{-1/2}U_{:R}, \quad W^*_\uparrow = \Sigma_{:R}V^T_{:R}$$

with residual $\sum_{i>R} \sigma^2_i$ (Eckart–Young–Mirsky theorem). The shipped "balanced" split uses $W_\downarrow = G^{-1/2}U_{:R}\Sigma^{1/2}_{:R}$, $W_\uparrow = \Sigma^{1/2}_{:R}V^T_{:R}$ for energy-ordered latents.

### The Whisper Correction
A low-rank sketch of the key residual $E = K - \hat{K}$ (rank $r=32$), fitted per-sequence at prefill, stabilizes worst-case accuracy without hurting averages. Values are left uncorrected because value errors average out in attention's convex combination.

### Two Operators
- **BellowsKV-Quant**: Uniform integer quantization with group-wise scales (128 coordinates/group), no rotation needed (Observation 1). Preserves full rank $R = p$.
- **BellowsKV-Evict**: Single eviction decision per block using attention-mass scoring from the last 64 queries, max-pooled over 13 tokens, with sink (4) and recency (64) exemptions.

### Formal Guarantees
The optimization is a reduced-rank regression (Izenman, 1975), with the crossing-layer blocks of $G$ being the novel load-bearing ingredient—setting them to zero recovers per-layer low-rank methods (Palu, Eigen Attention).

## Empirical Validation / Results

### Protocol B: Cross-Layer Head-to-Head (Table 1)
On Llama-3.1-8B, Mistral-7B, and Qwen2.5-7B across 11 LongBench tasks:

| Method | Compression | Retention (Llama) | Retention (Mistral) | Retention (Qwen) |
|--------|-------------|--------------------|--------------------|-------------------|
| MiniCache fp16 | 1.31× | 74.8% | 69.4% | 31.4% |
| KVSharer fp16 | 1.33× | 88.2% | 91.1% | 49.7% |
| **BellowsQ (n=2, INT8)** | **3.64×** | **98.9%** | **99.6%** | **98.8%** |
| CommonKV | 3.95× | 94.8% | 96.5% | 92.3% |
| xKV | 4.00× | 98.2% | 99.4% | 99.1% |
| **BellowsQ (n=4, INT4)** | **13.84×** | **98.0%** | **98.1%** | **97.3%** |
| xKV | 16.00× | 83.5% | 79.5% | 54.4% |
| CommonKV | 15.78× | 53.6% | 52.4% | 29.8% |
| **BellowsQ (n=4, K3V2)** | **20.47×** | **97.6%** | **97.9%** | **96.8%** |

### Cutting Rank vs. Cutting Bits (Table 3)
At a fixed budget, xKV reaches 16× by cutting rank $p \to p/4$, losing to 83.5% retention. xKV-hold (same basis at near-full rank, INT4, 15.1×) recovers to 97.6–99.2% on Llama/Mistral, proving that **rank reduction is the cliff, not sharing**.

### Agent Workload (Protocol C, Table 2)
On BrowseComp-Plus (Qwen3-32B and Qwen3-30B-A3B, including a MoE model), BellowsQ retains near-dense accuracy (85.3–86.0 vs. dense 85.5–86.1) at 14.6×, while xKV collapses to 8.4–26.5 at 8× and CommonKV loses half by 8×.

### Eviction Results (Table 5)
At equal memory, block-level eviction beats per-layer eviction by 0.9–3.0 points at ~8× and 5.3–6.6 points at ~16×, with the advantage growing at higher compression.

### Decode Cost (Table 6)
Measured decode speed is 0.995× dense (predicted 1.00×), confirming that reconstruct-at-read adds no measurable cost. Freed bytes buy 2× batch under HuggingFace and 14.76× KV pool under vLLM.

## Theoretical and Practical Implications

### Theoretical Contributions
1. **What to share**: The paper reframes cross-layer compression from sharing caches to sharing the *stream* (hidden states behind caches), which is where redundancy actually lives.
2. **The rank-vs-bits principle**: At fixed memory budget, holding rank and cutting bits is dramatically better than cutting rank. This aligns with and strengthens the position paper claim (Salfati, 2026) with causal evidence through the xKV-hold transplant experiment.
3. **Quantization-robust representations**: The latent's channel-mixing property makes it naturally outlier-free, eliminating the need for rotation—a structural advantage over per-layer caches.
4. **Decode speed bound**: ${\text{speedup} = (d_{kv}/R) \times (2/\text{bytes per latent value})}$—sharing buys capacity, never speed.

### Practical Contributions
1. **One-shot calibration**: ~20 minutes per model, no gradients, factors folded into weights—no per-sequence recomputation of the projection.
2. **Composable operators**: Quant and Evict compose (BellowsQ × BellowsE reaches ~30× at 97.5–99.2% retention).
3. **Wide compatibility**: Works on Llama-3.1, Mistral, Qwen families, standard and MoE models, without architecture changes.

## Conclusion

BellowsKV establishes one central rule backed by systematic measurement:

> **"Hold the rank, buy bits first, evict last."**

The paper's key takeaway is that cross-layer KV cache compression previously failed at high ratios not because sharing is wrong, but because sharing was done wrong—factorizing was the wrong lever. By fitting layer-specific projections to a full-rank, shared latent, BellowsKV achieves near-dense accuracy at an order-of-magnitude compression, bringing practical 10–20× KV cache reduction within reach for the first time.

**Future directions** (implicit from limitations): The Qwen anomaly remains unexplained—xKV-hold's collapse at INT4 on Qwen2.5 (23.4% retention) suggests model-specific interactions between basis structure and quantization that warrant investigation.

---

_Markdown view of https://picx.dev/p/9pIQ8m, served by PicX — AI-generated visual whiteboard summaries of research papers._