# Translating KV Caches Between Hybrid-Attention Language Models: MLA Latents, Linear-Attention States and Sparse-Attention Indexers > KV-cache translation extends beyond GQA to all hybrid-attention cache objects, achieving native quality in hybrid-to-full-MLA direction with 18.7-46.8x faster switching. - **Source:** [arXiv](https://arxiv.org/abs/2611.12062) - **Published:** 2026-11-14 - **Permalink:** https://picx.dev/p/13ymNv - **Whiteboard:** https://picx.dev/p/13ymNv/image ## Summary # Translating KV Caches Between Hybrid-Attention Language Models: MLA Latents, Linear-Attention States and Sparse-Attention Indexers **Authors:** Povilas Kvedaras, Tomas Rasymas (Hostinger Research) --- ## Summary (Overview) - **Core contribution:** This paper extends KV-cache translation—mapping a source model's cache into a target model's format so the target can continue from full context without re-reading—from simple grouped-query attention (GQA) to all cache objects found in modern hybrid-attention models: MLA latents, KDA/GDN recurrent states, and sparse-attention indexer keys. - **Per-object recipes:** The authors show that each cache object needs its own translation mechanism and capture point: token-wise linear maps for per-token objects (GQA K/V, MLA latents), state copy with closed-form bilinear correction for same-geometry recurrent states, replay of the target's own recurrence when the source has no state, and token identity decoding from the source's layer-0 MLA latent for exact long-range recall. - **Key results:** Hybrid→full-MLA direction reaches native quality within paired confidence intervals on every task metric; same-geometry GDN hybrids reach native level in both directions, with time-to-first-token 18.7–46.8× shorter than re-read at 64K tokens. Token identity raises needle recall from 0.09 to 0.99 at 2–32K and to 0.83–1.00 at 64K (8× the longest training prompt). - **Multi-turn behavior:** Segment-wise transfer stays at re-prefill level; naive re-translation collapses (reproducing KV-Lingo's finding on hybrid pairs). - **Sparse-attention rehearsal:** Predicted indexer keys recall needles as well as the target's own keys on a Qwen3.8-Flash-Next target (one seed, 400/1000 steps). --- ## Introduction and Theoretical Foundation ### Background and Motivation When a conversation moves from one language model to another (e.g., small→large, or between generations), the receiving model normally re-reads (prefills) the whole history. KV-cache translation avoids this: a small learned map converts the source model's key–value cache into the target's format, and the target continues as if it had read the prompt itself. **Why this matters:** Model switches are routine—routers send turns to cheaper/stronger models, agent pipelines hand conversations between models, and serving fleets migrate users to new generations. Without cache translation, the receiving model must either re-read the full history (cost grows with prompt length and model size) or receive a text summary (drops content, still needs prefill). ### The Problem: Frontier Caches Are No Longer Plain GQA Within one year, every open-weights family except Meta moved away from plain per-token K/V caches. Modern caches contain: - **MLA latents:** Compressed 512-d latent + 64-d rotary key per token/layer (DeepSeek, GLM, Kimi, Qwen) - **Sparse-attention indexer keys:** Small per-token keys that decide which tokens attention reads (DSA, QSA) - **Linear-attention states:** Fixed-size recurrent states (KDA, GDN, Mamba-2) with no per-token cache Each object breaks an assumption of the GQA recipe: a token-wise map cannot produce a recurrent state; MLA latents are normalized and rotated before caching; indexer keys determine which tokens are read at all. ### Theoretical Foundation The fidelity measure is KL divergence between the target's next-token distribution when reading its own cache vs. the translated cache: $$\mathrm{KL}(p_T(\cdot \mid \mathcal{C}_T) \parallel p_T(\cdot \mid \hat{\mathcal{C}}_T))$$ where $\mathcal{C}_T$ is the target's own cache and $\hat{\mathcal{C}}_T$ is the translated cache. Zero means the target cannot distinguish the translated cache from its own. **Capture points:** A cached object passes through transformations before storage. Three points are identified: ① raw projection output (pre-norm, un-rotated), ② post-norm, ③ the tensor the engine actually caches. --- ## Methodology ### Setting Two frozen models share a tokenizer. Source S has read a prompt of $n$ tokens $x_{1:n}$ and holds cache $\mathcal{C}_S(x_{1:n})$. A translator $f_\theta$ builds a cache in T's format: $$\hat{\mathcal{C}}_T = f_\theta(\mathcal{C}_S, x_{1:n})$$ Only $f_\theta$ is trained; S and T never change. ### Token-Wise Maps (Stage 1) For per-token objects, target layer $l$ and token $t$: $$\hat{y}_l(t) = W_l [x_{s_1}(t) \parallel \dots \parallel x_{s_\nu}(t)] + b_l$$ where $x_s(t)$ is the source's cached object at layer $s$ and token $t$, $\parallel$ is concatenation, and $s_1, \dots, s_\nu$ are the $\nu$ source layers assigned to target layer $l$ (chosen by relative depth). $W_l$ is fitted in closed form via least squares on paired source/target objects. ### Token Identity from MLA Layer-0 Latent A key insight: the layer-0 MLA latent is a deterministic function of the token alone (embedding → norm → kv_a_proj). A codebook with one row per vocabulary entry (163,840 rows) allows decoding token IDs from cached latents via nearest-neighbor search. The target's input embedding of the decoded ID is then appended to every map's input. ### State Strategies Four ways to give the target its recurrent state: 1. **Zero:** States start empty (baseline) 2. **Copy:** Source's final state written directly (when geometries match) 3. **Copy + correction:** $\hat{S}_h = A_h S_h B_h$ per value head, fitted by alternating ridge passes 4. **Replay:** Predict per-token inputs and run the target's own recurrence (when source has no state) ### Distillation (Stage 2) Fine-tunes all translator parameters end-to-end against the target reading its own cache: $$\mathcal{L}(\theta) = \frac{1}{|\mathrm{cont}|} \sum_i \mathrm{KL}\Big(p_T(\cdot \mid \mathcal{C}_T, \mathrm{cont}_{