# Transformer-FSQ: Discrete Latents for Extreme KV-Cache Compression

> Transformer-FSQ achieves up to 97.5× KV-cache compression with highest accuracy by replacing keys/values with a finite scalar quantisation bottleneck, enabling million-token contexts on consumer GPUs.

- **Source:** [arXiv](https://arxiv.org/abs/2611.09598)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/CnPk8A
- **Whiteboard:** https://picx.dev/p/CnPk8A/image

## Summary

## Summary (Overview)

- **Transformer-FSQ** introduces a novel KV-cache compression method that replaces standard key/value projections with a **finite scalar quantisation (FSQ) bottleneck**, storing bit-packed integer codes per token instead of floating-point vectors.
- The method achieves **extreme compression ratios (12× to 97.5×)** while maintaining the highest retrieval accuracy (RULER) compared to five baseline families (token eviction, scalar quantisation, vector quantisation, low-rank projection, latent attention) across Qwen3 models from 0.6B to 8B parameters.
- A **fused Triton decode kernel** reads packed codes directly, matching native SDPA throughput at 32× compression (1.01× on Qwen3-8B) and exceeding it at 97.5× (1.54×), making extreme compression practical for local devices.
- The approach uses **closed-form conversion** from pretrained transformers followed by finetuning that trains only inserted projections on 0.25 tokens per original parameter, with the backbone frozen.
- Transformer-FSQ **stacks with token eviction** (PyramidKV), yielding 1.7× greater aggregate projected capacity (3.8M tokens) on a 16 GiB GPU compared to the best baseline.

## Introduction and Theoretical Foundation

### Background and Motivation

The KV cache grows linearly with context length and quickly dominates memory on local consumer devices. For Qwen3-8B-Base, caching 144 KiB per token in half precision alongside 15.3 GiB of weights on a 16 GiB GPU leaves room for only 5,000 uncompressed tokens. The paper targets the **extreme compression regime (>16×)** needed to hold millions of tokens alongside model weights.

### Theoretical Basis

The paper examines two discrete latent approaches:

**Vector Quantisation (VQ):** Maps a continuous latent vector to the nearest codeword in a learned codebook:
$$Q_{\mathrm{VQ}}(\mathbf{z}) = \mathbf{c}_{k^{\star}}, \quad k^{\star} = \arg\min_{k} \|\mathbf{z} - \mathbf{c}_{k}\|_2^2 \tag{1}$$

VQ suffers from commitment losses, codebook updates, nearest-neighbour search, and centroid collapse.

**Finite Scalar Quantisation (FSQ):** Replaces the learned codebook with fixed scalar levels per coordinate:
$$\tilde{u} = \tanh\bigl(u + \operatorname{atanh}(\delta/h)\bigr)h - \delta, \quad \hat{u} = Q(u) = \tilde{u} + \operatorname{sg}\bigl(\lfloor\tilde{u}\rceil - \tilde{u}\bigr) \in \mathcal{L}_i \tag{2}$$

The complete representable set is the Cartesian product:
$$\mathcal{C}_{\mathrm{FSQ}} = \mathcal{L}_1 \times \dots \times \mathcal{L}_d, \quad |\mathcal{C}_{\mathrm{FSQ}}| = \prod_{i=1}^{d} L_i \tag{3}$$

FSQ requires **no stored codebook**, no commitment losses, and no codebook collapse—the grid is implicit and never materialised.

## Methodology

### Architecture

Transformer-FSQ compresses the shared residual representation before K/V expansion:
- **Encoder** $E \in \mathbb{R}^{G \times C}$ maps the residual state to a G-dimensional latent
- **FSQ quantiser** rounds coordinates to L levels (L=16, i.e., 4 bits per coordinate)
- **Shared decoder** $W_{kv} \in \mathbb{R}^{2n_{kv}d_h \times G}$ reconstructs all KV heads

$$z_t = Q_{\mathrm{FSQ}}(E x_t) \in \mathcal{L}_1 \times \dots \times \mathcal{L}_G \tag{4}$$

$$\hat{k}_t = \mathrm{RoPE}_t(N_k(W_k z_t)), \qquad \hat{v}_t = W_v z_t \tag{5}$$

### Key Design Innovation

**Value decoding after attention pooling:** Due to linearity,
$$\sum_t a_t^{(h)} W_v z_t = W_v \left(\sum_t a_t^{(h)} z_t\right) \tag{6}$$

Values are decoded once after attention pools the codes, while keys must be reconstructed per position due to RoPE.

### Closed-Form Conversion

The encoder is initialised from principal directions of calibration activations:
$$E_{i,:} = \frac{u_{i \bmod C}^{\top}}{3\sigma_{i \bmod C}}, \qquad i = 0, \ldots, G-1 \tag{7}$$

The decoder is fitted via ridge regression:
$$D = (Z^{\top}Z + \lambda I_G)^{-1}Z^{\top}X, \quad W_{kv} = [W^K; W^V]D^{\top} \tag{8}$$

### Byte Budgeting

All methods are compared at equal realised cache bytes, charging for all metadata, windows, and auxiliary state. Full cache size: $c_{\text{full}} = 2 \cdot n_{kv} \cdot d_h \cdot \text{sizeof}(\text{float16})$ bytes/token/layer (4096 B for Qwen3 models).

## Empirical Validation / Results

### RQ1: Accuracy Across Compression Regimes

- At ≤5× compression, VecInfer retains ≥93% of uncompressed RULER scores
- **From 12× to 97.5×, Transformer-FSQ is the most accurate method** across all model scales
- At 12× on Qwen3-0.6B, Transformer-FSQ outperforms token eviction by 0.12 RULER points

### RQ2: Natural Document Summarisation (LongBench)

- Transformer-FSQ scores highest from 32× to 97.5× on all three models
- On Qwen3-8B: 12.5 ROUGE-L at 32× and 11.4 at 97.5×, vs. Palu's 7.0 at 102.4×

### RQ3: Latent Geometry at Fixed Budgets

| Method | Training | Cache | Width | B/tok | RULER |
|---|---|---|---|---|---|
| **Transformer-FSQ** | Discrete (STE, L=16) | 4-bit implicit grid | 288 | 144 | **0.232 ± 0.022** |
| Continuous latent | Continuous (bf16) | 16-bit float | 72 | 144 | 0.009 ± 0.008 |
| Latent + post-hoc FSQ | Continuous (bf16) | 4-bit post-hoc | 288 | 144 | 0.175 ± 0.030 |
| Palu | Low-rank SVD | 4-bit SVD + scales | 144 | 144 | 0.000 ± 0.000 |

Training through the 4-bit grid (quantisation-aware finetuning) scores highest at both 144 B and 80 B budgets.

### RQ4: Decode Throughput

- At 32×: 1.38×, 1.34×, 1.01× native SDPA throughput on Qwen3-0.6B, 1.7B, 8B
- At 409.6×: 1.98×, 2.20×, 1.63× (up to 5.86× at batch ≥100k tokens)
- Native SDPA runs out of memory by 24,555 tokens at batch 16; Transformer-FSQ decodes to 150,000

### RQ5: Weight-Quantised Backbone

With 4-bit NF4 weights on Qwen3-8B, Transformer-FSQ finetuned on the frozen NF4 backbone recovers to 0.464 RULER (vs. 0.382 for post-hoc NF4 application), within 0.029 of the 16-bit backbone result.

### Baseline Limits on 16 GiB GPU

| Method | Family | Deepest | Max tokens (≥0.30 RULER) |
|---|---|---|---|
| PyramidKV | token eviction | 174× | 552k |
| KIVI | scalar quantisation | 5.2× | 684k |
| VecInfer | vector quantisation | 11.0× | 592k |
| Palu | low-rank projection | 10.2× | 1.42M |
| TransMLA | latent attention | 14.2× | 542k |
| **Transformer-FSQ** | **FSQ** | **410×** | **2.21M** |

## Theoretical and Practical Implications

- **Extreme compression is practical**: Transformer-FSQ demonstrates that KV-cache compression beyond 16× can maintain usable accuracy, enabling million-token contexts on consumer hardware.
- **Discrete latents outperform continuous**: Training through a low-bit grid (QAFT) outperforms both inference-only quantisation and narrow continuous bottlenecks at matched byte budgets.
- **Memory-bandwidth-bound decoding**: At extreme compression, saved cache reads dominate arithmetic costs, making compressed decoding faster than uncompressed native attention.
- **Composability**: Channel-width compression (FSQ) composes with sequence-length compression (eviction), enabling multiplicative capacity gains.
- **Zero metadata overhead**: FSQ's implicit codebook eliminates per-token/per-channel scales, a key advantage over scalar quantisation methods like KIVI.

## Conclusion

Transformer-FSQ demonstrates that **attention does not need full-precision keys and values in its cache**—a learned discrete state per token maintains accuracy at compression ratios where existing methods fail. Key findings:

1. **Wide low-bit latents** outperform narrow high-precision ones at the same byte budget
2. **Quantisation-aware finetuning** is essential for extreme compression
3. **Fused decode kernels** make compressed attention faster than uncompressed at extreme ratios

**Future directions** include:
- **Allocation**: Varying code width/levels per layer, sharing projections across layers
- **Training**: Block distillation and LoRA adapters on the frozen backbone
- **Dynamic compression**: Coarsening codes in place as sequences grow (light compression for short sequences, deeper for longer ones)
- **Evaluation**: Testing on reasoning, code generation, many-shot learning, and longer contexts beyond 8,192 tokens

---

_Markdown view of https://picx.dev/p/CnPk8A, served by PicX — AI-generated visual whiteboard summaries of research papers._