Summary (Overview)

• Transformer-FSQ introduces a novel KV-cache compression method that replaces standard key/value projections with a finite scalar quantisation (FSQ) bottleneck, storing bit-packed integer codes per token instead of floating-point vectors.
• The method achieves extreme compression ratios (12× to 97.5×) while maintaining the highest retrieval accuracy (RULER) compared to five baseline families (token eviction, scalar quantisation, vector quantisation, low-rank projection, latent attention) across Qwen3 models from 0.6B to 8B parameters.
• A fused Triton decode kernel reads packed codes directly, matching native SDPA throughput at 32× compression (1.01× on Qwen3-8B) and exceeding it at 97.5× (1.54×), making extreme compression practical for local devices.
• The approach uses closed-form conversion from pretrained transformers followed by finetuning that trains only inserted projections on 0.25 tokens per original parameter, with the backbone frozen.
• Transformer-FSQ stacks with token eviction (PyramidKV), yielding 1.7× greater aggregate projected capacity (3.8M tokens) on a 16 GiB GPU compared to the best baseline.

Introduction and Theoretical Foundation

Background and Motivation

The KV cache grows linearly with context length and quickly dominates memory on local consumer devices. For Qwen3-8B-Base, caching 144 KiB per token in half precision alongside 15.3 GiB of weights on a 16 GiB GPU leaves room for only 5,000 uncompressed tokens. The paper targets the extreme compression regime (>16×) needed to hold millions of tokens alongside model weights.

Theoretical Basis

The paper examines two discrete latent approaches:

Vector Quantisation (VQ): Maps a continuous latent vector to the nearest codeword in a learned codebook:

$$Q_{\mathrm{VQ}}(\mathbf{z}) = \mathbf{c}_{k^{\star}}, \quad k^{\star} = \arg\min_{k} \|\mathbf{z} - \mathbf{c}_{k}\|_2^2 \tag{1}$$

VQ suffers from commitment losses, codebook updates, nearest-neighbour search, and centroid collapse.

Finite Scalar Quantisation (FSQ): Replaces the learned codebook with fixed scalar levels per coordinate:

$$\tilde{u} = \tanh\bigl(u + \operatorname{atanh}(\delta/h)\bigr)h - \delta, \quad \hat{u} = Q(u) = \tilde{u} + \operatorname{sg}\bigl(\lfloor\tilde{u}\rceil - \tilde{u}\bigr) \in \mathcal{L}_i \tag{2}$$

The complete representable set is the Cartesian product:

$$\mathcal{C}_{\mathrm{FSQ}} = \mathcal{L}_1 \times \dots \times \mathcal{L}_d, \quad |\mathcal{C}_{\mathrm{FSQ}}| = \prod_{i=1}^{d} L_i \tag{3}$$

FSQ requires no stored codebook, no commitment losses, and no codebook collapse—the grid is implicit and never materialised.

Methodology

Architecture

Transformer-FSQ compresses the shared residual representation before K/V expansion:

• Encoder $E \in \mathbb{R}^{G \times C}$ maps the residual state to a G-dimensional latent
• FSQ quantiser rounds coordinates to L levels (L=16, i.e., 4 bits per coordinate)
• Shared decoder $W_{kv} \in \mathbb{R}^{2n_{kv}d_h \times G}$ reconstructs all KV heads

$$z_t = Q_{\mathrm{FSQ}}(E x_t) \in \mathcal{L}_1 \times \dots \times \mathcal{L}_G \tag{4}$$ $$\hat{k}_t = \mathrm{RoPE}_t(N_k(W_k z_t)), \qquad \hat{v}_t = W_v z_t \tag{5}$$

Key Design Innovation

Value decoding after attention pooling: Due to linearity,

$$\sum_t a_t^{(h)} W_v z_t = W_v \left(\sum_t a_t^{(h)} z_t\right) \tag{6}$$

Values are decoded once after attention pools the codes, while keys must be reconstructed per position due to RoPE.

Closed-Form Conversion

The encoder is initialised from principal directions of calibration activations:

$$E_{i,:} = \frac{u_{i \bmod C}^{\top}}{3\sigma_{i \bmod C}}, \qquad i = 0, \ldots, G-1 \tag{7}$$

The decoder is fitted via ridge regression:

$$D = (Z^{\top}Z + \lambda I_G)^{-1}Z^{\top}X, \quad W_{kv} = [W^K; W^V]D^{\top} \tag{8}$$

Byte Budgeting

All methods are compared at equal realised cache bytes, charging for all metadata, windows, and auxiliary state. Full cache size: $c_{\text{full}} = 2 \cdot n_{kv} \cdot d_h \cdot \text{sizeof}(\text{float16})$ bytes/token/layer (4096 B for Qwen3 models).

Empirical Validation / Results

RQ1: Accuracy Across Compression Regimes

• At ≤5× compression, VecInfer retains ≥93% of uncompressed RULER scores
• From 12× to 97.5×, Transformer-FSQ is the most accurate method across all model scales
• At 12× on Qwen3-0.6B, Transformer-FSQ outperforms token eviction by 0.12 RULER points

RQ2: Natural Document Summarisation (LongBench)

• Transformer-FSQ scores highest from 32× to 97.5× on all three models
• On Qwen3-8B: 12.5 ROUGE-L at 32× and 11.4 at 97.5×, vs. Palu's 7.0 at 102.4×

RQ3: Latent Geometry at Fixed Budgets

Training through the 4-bit grid (quantisation-aware finetuning) scores highest at both 144 B and 80 B budgets.

RQ4: Decode Throughput

• At 32×: 1.38×, 1.34×, 1.01× native SDPA throughput on Qwen3-0.6B, 1.7B, 8B
• At 409.6×: 1.98×, 2.20×, 1.63× (up to 5.86× at batch ≥100k tokens)
• Native SDPA runs out of memory by 24,555 tokens at batch 16; Transformer-FSQ decodes to 150,000

RQ5: Weight-Quantised Backbone

With 4-bit NF4 weights on Qwen3-8B, Transformer-FSQ finetuned on the frozen NF4 backbone recovers to 0.464 RULER (vs. 0.382 for post-hoc NF4 application), within 0.029 of the 16-bit backbone result.

Baseline Limits on 16 GiB GPU

Theoretical and Practical Implications

• Extreme compression is practical: Transformer-FSQ demonstrates that KV-cache compression beyond 16× can maintain usable accuracy, enabling million-token contexts on consumer hardware.
• Discrete latents outperform continuous: Training through a low-bit grid (QAFT) outperforms both inference-only quantisation and narrow continuous bottlenecks at matched byte budgets.
• Memory-bandwidth-bound decoding: At extreme compression, saved cache reads dominate arithmetic costs, making compressed decoding faster than uncompressed native attention.
• Composability: Channel-width compression (FSQ) composes with sequence-length compression (eviction), enabling multiplicative capacity gains.
• Zero metadata overhead: FSQ's implicit codebook eliminates per-token/per-channel scales, a key advantage over scalar quantisation methods like KIVI.

Conclusion

Transformer-FSQ demonstrates that attention does not need full-precision keys and values in its cache—a learned discrete state per token maintains accuracy at compression ratios where existing methods fail. Key findings:

1. Wide low-bit latents outperform narrow high-precision ones at the same byte budget
2. Quantisation-aware finetuning is essential for extreme compression
3. Fused decode kernels make compressed attention faster than uncompressed at extreme ratios

Future directions include:

• Allocation: Varying code width/levels per layer, sharing projections across layers
• Training: Block distillation and LoRA adapters on the frozen backbone
• Dynamic compression: Coarsening codes in place as sequences grow (light compression for short sequences, deeper for longer ones)
• Evaluation: Testing on reasoning, code generation, many-shot learning, and longer contexts beyond 8,192 tokens