Efficient Sequence ModelingIssue 14Nov 7 – 14, 2026

KV Cache Becomes a Translatable, Discretely Compressible Object

Highlights of This Issue

The most prominent signal this issue is that the KV cache has evolved from a 'compressed object' to a 'translatable, cross-model transferable object'. Translating KV Caches Between Hybrid-Attention Language Models for the first time extends KV cache translation from GQA to all objects of hybrid attention models—MLA latent, KDA/GDN recurrent states, and sparse attention indexer keys—and provides capture/injection points and mechanisms for each object: per-token linear mapping, closed-form bilinear correction copying, and a previously unrecognized mechanism—token identity decoding (recovering token identity from source layer-0 MLA latent), which improves needle recall from 0.09 to 0.99 (2-32K) and 0.83-1.00 (64K). This converges with existing GQA layer-wise linear mapping and one-to-one state transfer: the translated objects cover all cache states of current hybrid models. Meanwhile, Transformer-FSQ advances the compression form from continuous low-rank to 'wide and low-bit' discrete finite scalar quantization (FSQ): compressing once on a shared residual representation, storing as bit-packed integer codes, without per-token metadata, codebook-free and search-free, establishing a retrieval accuracy frontier at 12×–97.5× extreme compression ratios.

There are two substantial advances in the low-rank nature and placement of recurrent states. SinkState for the first time reveals that q/k/v in linear attention layers concentrate in a stable low-dimensional subspace (linear attention sink), and accordingly maintains recurrent states throughout decoding with adaptive low-rank SVD: local SVD merges only top subspaces, asynchronous CPU merging, per-state variable rank, avoiding full state read/write each step—this is a serving-oriented advancement of the static/periodic low-rank storage in Shallow Encoding Emits Deep Memories from Issue 13, achieving 1.25× decoding speedup and halved state memory. Looping Deeper Without Growing the KV Cache studies the placement of loop depth in causal encoder-decoder (CED) models: proving that loops must be placed on the encoder side to iteratively refine the global KV source, rather than re-reading static projections on the decoder side; the contraction fixed point induced by stochastic depth training enables lossless cross-round KV sharing, achieving zero-round KV growth loop depth at 552B scale, recovering 86–93% of the gains of looped Transformers while using only 1/6 of their global KV.

There are two system increments worth reading on indexers and speculative decoding. Subquadratic Attention with a Hierarchical Tree-Structured Indexer continues the hierarchical tree direction, using piecewise-affine approximations of trainable reference queries to construct level-wise compressed keys, with level-wise top-p routing making total cost sublinear in length, maintaining dense performance within 256K—but extrapolation beyond 1M relies only on roofline extrapolation, which should be compared with the measured degradation curves of Recursive Sparse Attention from Issue 13. HybridSpec for the first time advances speculative decoding verification length selection to 'two-stage verification at the router boundary', and quantifies that GDN verification length 4→16 increases verification time by 71% (dense targets only 4-6%), a previously unreported hybrid architecture characteristic.

Community and Updates

  • STAR-KV: Learning the Rank Is Only Half of KV Cache Compression proposes joint optimization of rank allocation and quantization with per-head fine-grained spectral low-rank thresholds, rotation, and mixed-precision quantization, complementing the discrete low-rank compression direction this issue; but as a technical review it lacks compute-matched comparisons and long-context degradation curves, serving only as an initial signal.
  • From Full to Hybrid Attention in LLMs is a snapshot review of the 2026 industry shift to hybrid attention, documenting the current state where about half of models retain some full attention and the other half are fully hybrid, along with design trade-offs in Gated DeltaNet/KDA/Mamba-2 layer mixing—valuable as background for layer-wise ratios in hybrid architectures, but with no technical increments.
  • ForeSpark proposes joint scheduling of draft/verification budgets in parallel speculative decoding, advancing budget allocation from fixed/heuristic to joint optimization; complements the verification-side optimization of ReplaySSM from Issue 12, but lacks long-context degradation evidence.

Open Questions

  1. The token identity decoding in Translating KV Caches breaks the needle recall barrier to 0.99, but its multi-round and cross-direction evidence is only single-seed—can token identity caching hold at longer contexts (>64K) and multi-agent rounds, and can it combine with the discrete bottleneck of Transformer-FSQ into a unified cross-model portable + compressible cache?
  2. Both SinkState's online adaptive low-rank and SEED's state sink decomposition from Issue 13 point to 'recurrent states are low-rank'—but between adaptive variable-rank closed-loop error and Issue 13's open-loop periodic compression, which is more stable for serving long CoT? The joint benefit of both on the same GDN backbone has not been tested.
  3. The lossless cross-round KV sharing in Looping Deeper relies on contraction fixed points under stochastic depth—will this property exhibit phase-specific blind spots at longer contexts (million-scale) or agentic long tasks, as in Periodic Weak Spots from Issue 11?
  4. The superlinear cost of GDN verification length reported in HybridSpec is measured under single-request greedy decoding—under multi-request batching and draft confidence degradation, does two-stage verification at the router boundary still outperform simple fixed-length pre-verification?

Papers in this issue

  1. Translating KV Caches Between Hybrid-Attention Language Models: MLA Latents, Linear-Attention States and Sparse-Attention Indexers

    KV-cache translation extends beyond GQA to all hybrid-attention cache objects, achieving native quality in hybrid-to-full-MLA direction with 18.7-46.8x faster switching.

    Editor's note

    For the first time extends KV cache translation from GQA to all objects of hybrid attention models—MLA latent, KDA/GDN recurrent states, and sparse attention indexer keys—and provides capture/injection points and mechanisms for each object; its token identity decoding (recovering token identity from source layer-0 MLA latent) improves needle recall from 0.09 to 0.99 (2-32K) and 0.83-1.00 (64K). Relative to existing GQA layer-wise linear mapping and one-to-one state transfer, the increment is covering all cache objects of current hybrid models and providing complete evidence for bidirectional, multi-round, and indexer rehearsal. Must-read for those doing cross-model cache reuse and hybrid architecture KV translation; limitation: multi-round part is only single-seed.

  2. Transformer-FSQ: Discrete Latents for Extreme KV-Cache Compression

    Transformer-FSQ achieves up to 97.5× KV-cache compression with highest accuracy by replacing keys/values with a finite scalar quantisation bottleneck, enabling million-token contexts on consumer GPUs.

    Editor's note

    For the first time advances KV cache compression to a 'wide and low-bit' discrete finite scalar quantization (FSQ) bottleneck: compressing once on a shared residual representation, storing as bit-packed integer codes, without per-token metadata, codebook-free and search-free, end-to-end trainable, with closed-form initialization and fused decoding kernels. Relative to DCKV's trainable transform coding from Issue 12 and iS-KV's online low-rank from Issue 9, the increment is providing a VQ-collapse-free discrete bottleneck and establishing a retrieval accuracy frontier at 12×–97.5× extreme compression. Those doing trainable KV compression should read; limitation: long-context evaluation only up to 8K/32K.

  3. SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving

    SINKSTATE exploits the linear attention sink to maintain recurrent states in adaptive low-rank SVD form, cutting peak memory by 2.6x and boosting decoding speed by 1.58x across hybrid LLMs.

    Editor's note

    For the first time reveals that q/k/v in linear attention layers concentrate in a stable low-dimensional subspace (linear attention sink), and accordingly proposes maintaining recurrent states throughout decoding with adaptive low-rank SVD: local SVD merges only top subspaces, asynchronous CPU merging, per-state variable rank, avoiding full state read/write each step. Relative to Shallow Encoding Emits Deep Memories from Issue 13 (state sink + low-rank storage), the increment is advancing low-rank maintenance from static/periodic compression to online adaptive serving systems, with evidence of end-to-end speedup across 4B–120B models (1.25× decoding, halved state memory). Those doing state representation on the hybrid inference side should read; but 16K NIAH is its long-context evidence upper bound and lacks compute-matched comparison with full attention.

  4. Looping Deeper Without Growing the KV Cache: Recurrent Depth for Causal Encoder-Decoders

    Looped CED places recurrence in the encoder and uses stochastic-depth training to achieve lossless cross-round KV sharing, enabling 36.8x smaller memory and 3.6x higher throughput.

    Editor's note

    For the first time systematically studies loop depth placement in causal encoder-decoder (CED) models: proving that loops must be placed on the encoder side to iteratively refine the global KV source rather than re-reading static projections on the decoder side, and proving that the contraction fixed point induced by stochastic depth training enables lossless cross-round KV sharing, achieving zero KV growth loop depth at 552B scale. Relative to Muir's sensitivity gating from Issue 13 and ReplaySSM's caching inputs rather than states from Issue 12, the increment is jointly optimizing the two axes of 'loop depth' and 'CED cross-layer KV sharing' and providing mechanistic evidence for encoder vs. decoder loops. Those doing layer-wise ratios in hybrid architectures should read; limitation: quality evaluation only up to 340M.

  5. Subquadratic Attention with a Hierarchical Tree-Structured Indexer

    ZHA is a hierarchical sparse attention method that achieves subquadratic scaling with query-dependent sparsity, matching dense performance on long-context tasks while attending to only 1–6% of the KV cache.

    Editor's note

    For the first time treats both selection cost and performance retention of hierarchical tree indexers as design goals: using piecewise-affine approximations of trainable reference queries to construct level-wise compressed keys, with level-wise top-p routing making total cost sublinear in length. Relative to the logarithmic selection cost of Recursive Sparse Attention from Issue 13, the increment is top-p adaptive selection, tree-depth robustness (L6), and 1B token distillation for pretrained models, with a prediction model across geometries; maintains dense performance within 256K. Limitation: extrapolation beyond 1M relies only on roofline models, no real degradation curves.

  6. HybridSpec: Throughput-Aware Speculative Decoding for Hybrid GDN-MoE Models

    HybridSpec achieves 1.75–1.88x speedup over autoregressive decoding for hybrid GDN–MoE models by selecting verification length on the GPU using draft acceptance estimates and observed expert routing.

    Editor's note

    For the first time advances speculative decoding verification length selection to 'two-stage verification at the router boundary': pausing before MoE router execution, using real expert assignments to correct the retention of an additional token, and reporting the hybrid-specific degradation curve where GDN verification length 4→16 increases verification time by 71% (dense targets only 4-6%). Relative to ReplaySSM's chunkwise triangular solving from Issue 12, the increment is two-stage verification + router correction + pooled throughput objective. Those doing speculative decoding and GDN-MoE inference should read; limitation: only single-request greedy decoding.

  7. Verify Before You Fetch: Reliable KV Cache Compression for LLM Inference

    VERKV combines Poisson sampling compression with a bounded shrinkage estimator and Bernstein certificates, achieving near-dense attention quality at 8-32x memory reduction via verified, minimal CPU-GPU KV fetching.

    Editor's note

    Unifies query-agnostic compression and query-aware verification/retrieval of KV cache management into a hybrid CPU/GPU framework, using bounded contraction estimators to approximate the attention denominator and drive certificate-based retrieval decisions, providing (ε,δ) approximation guarantees and minimizing CPU→GPU transfer. Relative to Certified Sparse Attention's per-query finite-sample certificates from Issue 11, the increment is integrating verification and adaptive retrieval into an offloading setting, achieving near-dense quality at high compression with low host traffic. Those doing reliable long-context KV management should read.

  8. AttentionIndex: Accelerating Learned Sparse Attention with a Causal Inverted-File Index

    AttentionIndex replaces exhaustive key scoring in learned sparse attention with a causal inverted-file index, delivering 1.3-1.8x prefill throughput and far higher recall than HiSA at one million tokens.

    Editor's note

    For the first time adapts inverted file index (IVF) to the selection bottleneck of learnable sparse attention indexers: causal incremental maintenance and learned routing representatives make new keys immediately retrievable, exposed only to clusters built from their own prefix. Relative to existing block-level routing HiSA, the increment is providing causal IVF's per-token incremental construction and compute-matched throughput comparison, with 1.3–1.8× prefilling throughput at 1M tokens and 17–23 percentage points higher recall at the same throughput. Those doing block-sparse indexer acceleration should read.