Translating KV Caches Between Hybrid-Attention Language Models: MLA Latents, Linear-Attention States and Sparse-Attention Indexers

Authors: Povilas Kvedaras, Tomas Rasymas (Hostinger Research)

---

Summary (Overview)

• Core contribution: This paper extends KV-cache translation—mapping a source model's cache into a target model's format so the target can continue from full context without re-reading—from simple grouped-query attention (GQA) to all cache objects found in modern hybrid-attention models: MLA latents, KDA/GDN recurrent states, and sparse-attention indexer keys.

• Per-object recipes: The authors show that each cache object needs its own translation mechanism and capture point: token-wise linear maps for per-token objects (GQA K/V, MLA latents), state copy with closed-form bilinear correction for same-geometry recurrent states, replay of the target's own recurrence when the source has no state, and token identity decoding from the source's layer-0 MLA latent for exact long-range recall.

• Key results: Hybrid→full-MLA direction reaches native quality within paired confidence intervals on every task metric; same-geometry GDN hybrids reach native level in both directions, with time-to-first-token 18.7–46.8× shorter than re-read at 64K tokens. Token identity raises needle recall from 0.09 to 0.99 at 2–32K and to 0.83–1.00 at 64K (8× the longest training prompt).

• Multi-turn behavior: Segment-wise transfer stays at re-prefill level; naive re-translation collapses (reproducing KV-Lingo's finding on hybrid pairs).

• Sparse-attention rehearsal: Predicted indexer keys recall needles as well as the target's own keys on a Qwen3.8-Flash-Next target (one seed, 400/1000 steps).

---

Introduction and Theoretical Foundation

Background and Motivation

When a conversation moves from one language model to another (e.g., small→large, or between generations), the receiving model normally re-reads (prefills) the whole history. KV-cache translation avoids this: a small learned map converts the source model's key–value cache into the target's format, and the target continues as if it had read the prompt itself.

Why this matters: Model switches are routine—routers send turns to cheaper/stronger models, agent pipelines hand conversations between models, and serving fleets migrate users to new generations. Without cache translation, the receiving model must either re-read the full history (cost grows with prompt length and model size) or receive a text summary (drops content, still needs prefill).

The Problem: Frontier Caches Are No Longer Plain GQA

Within one year, every open-weights family except Meta moved away from plain per-token K/V caches. Modern caches contain:

• MLA latents: Compressed 512-d latent + 64-d rotary key per token/layer (DeepSeek, GLM, Kimi, Qwen)
• Sparse-attention indexer keys: Small per-token keys that decide which tokens attention reads (DSA, QSA)
• Linear-attention states: Fixed-size recurrent states (KDA, GDN, Mamba-2) with no per-token cache

Each object breaks an assumption of the GQA recipe: a token-wise map cannot produce a recurrent state; MLA latents are normalized and rotated before caching; indexer keys determine which tokens are read at all.

Theoretical Foundation

The fidelity measure is KL divergence between the target's next-token distribution when reading its own cache vs. the translated cache:

$$\mathrm{KL}(p_T(\cdot \mid \mathcal{C}_T) \parallel p_T(\cdot \mid \hat{\mathcal{C}}_T))$$

where $\mathcal{C}_T$ is the target's own cache and $\hat{\mathcal{C}}_T$ is the translated cache. Zero means the target cannot distinguish the translated cache from its own.

Capture points: A cached object passes through transformations before storage. Three points are identified: ① raw projection output (pre-norm, un-rotated), ② post-norm, ③ the tensor the engine actually caches.

---

Methodology

Setting

Two frozen models share a tokenizer. Source S has read a prompt of $n$ tokens $x_{1:n}$ and holds cache $\mathcal{C}_S(x_{1:n})$. A translator $f_\theta$ builds a cache in T's format:

$$\hat{\mathcal{C}}_T = f_\theta(\mathcal{C}_S, x_{1:n})$$

Only $f_\theta$ is trained; S and T never change.

Token-Wise Maps (Stage 1)

For per-token objects, target layer $l$ and token $t$:

$$\hat{y}_l(t) = W_l [x_{s_1}(t) \parallel \dots \parallel x_{s_\nu}(t)] + b_l$$

where $x_s(t)$ is the source's cached object at layer $s$ and token $t$, $\parallel$ is concatenation, and $s_1, \dots, s_\nu$ are the $\nu$ source layers assigned to target layer $l$ (chosen by relative depth). $W_l$ is fitted in closed form via least squares on paired source/target objects.

Token Identity from MLA Layer-0 Latent

A key insight: the layer-0 MLA latent is a deterministic function of the token alone (embedding → norm → kv_a_proj). A codebook with one row per vocabulary entry (163,840 rows) allows decoding token IDs from cached latents via nearest-neighbor search. The target's input embedding of the decoded ID is then appended to every map's input.

State Strategies

Four ways to give the target its recurrent state:

1. Zero: States start empty (baseline)
2. Copy: Source's final state written directly (when geometries match)
3. Copy + correction: $\hat{S}_h = A_h S_h B_h$ per value head, fitted by alternating ridge passes
4. Replay: Predict per-token inputs and run the target's own recurrence (when source has no state)

Distillation (Stage 2)

Fine-tunes all translator parameters end-to-end against the target reading its own cache:

$$\mathcal{L}(\theta) = \frac{1}{|\mathrm{cont}|} \sum_i \mathrm{KL}\Big(p_T(\cdot \mid \mathcal{C}_T, \mathrm{cont}_{<i}) \Big\| p_T(\cdot \mid \hat{\mathcal{C}}_T(\theta), \mathrm{cont}_{<i})\Big)$$

Experimental Pairs

---

Empirical Validation / Results

Replication of KV-Lingo (Part 1)

• Stage-1 $R^2$: 0.802/0.658 (K/V) vs. KV-Lingo's 0.815/0.693
• Val KL: 0.322 → 0.056 (closed form → distilled)
• Needle exact recall: 1.00 at 2–8K
• Chat judge at native level; CoQA F1 0.07 below native (answer length effect)

Pair B: Same-Geometry Hybrids (Native Level Both Directions)

Forward direction ties the target on all metrics:

• CoQA: +0.015 [−0.005, +0.035]
• ARC: +0.010 [−0.007, +0.030]
• Judge: +0.03 [−0.34, +0.42]
• Needle: 12/12 at 64K and 128K (16× training cap)

The bilinear state correction improves distributional fidelity: val KL −0.0035 [−0.0044, −0.0027].

Reverse asymmetry (behavioral): Qwen3.5 starts 32% of CoQA answers with leading-space tokens; Qwen3.8 never does. The reverse translator makes Qwen3.5 refuse 6/100 needles as "sensitive"—the only significant gap to native (needle −0.06 [−0.11, −0.02]). KL and accuracy do not detect this.

Pair A Reverse: Hybrid → Full MLA (Easier Direction)

From 7 MLA latents per token plus token IDs, a single linear map per target layer brings Kimi-VL within the paired CI of native on every task metric:

• Needle: 100/100 at 2–32K, 12/12 at 64K
• ARC: −0.033 [−0.077, +0.010]
• Judge: 4.17 vs 4.26 (−0.09 [−0.67, +0.49])

Pair A Forward: Full MLA → Hybrid (Harder Direction)

• Gold-in-answer at native level: 0.813 vs 0.803 (+0.010 [−0.030, +0.050])
• CoQA F1 trails (−0.200, mostly answer length: 9.5 vs 4.5 tokens)
• ARC: −0.050 [−0.087, −0.013]
• Chat judge: 5.15 vs 6.02 (−0.87)

Token Identity Closes the Recall Gap

Without token identity: needle recall 9/100 at 2–32K, 0/12 at 64K (though ~60% of digits correct—locates the fact but cannot copy it).

With token identity (decoded from layer-0 MLA latent or passed as IDs):

• Recall: 0.99 (+0.90 [+0.84, +0.96]) at 2–32K
• 64K: 10/12 (codebook), 12/12 (token-ID)
• Val KL improves: −0.021 [−0.026, −0.015]

Capture Point Ablation (① vs ③)

• Pair B: ③ ties ① on every metric up to 128K (source and target use same partial RoPE)
• Pair A: ③ close to ① up to 8K training cap but collapses beyond: 0.25 vs 0.95 at 32K, 0.00 vs 0.83 at 64K (rotated key cannot be position-free)

Switch Speed

Pair B's O(1) state copy: switch grows by 4.1 ms/1K tokens vs. 236 ms for cold prefill (asymptotic ratio ×57).

Multi-Turn Results

• Segment-wise transfer stays at re-prefill ceiling: Pair B F1 −0.004 [−0.045, +0.043]; Pair A gold-in-answer +0.000 [−0.048, +0.048]
• Naive re-translation collapses on Pair A: turn-5 gold drops to 26% (vs. 70% re-prefill), F1 0.22
• Refresh after turn 4: costs nothing measurable, recovers turns 6–10 in 10-turn check

Sparse-Attention Indexer Rehearsal (Pair C)

• Predicted keys select 63–84% of native top-k blocks but keep needle block selected in ≥99.6% of (layer, step) pairs
• Needle recall: 0.87 (predicted) vs 0.88 (native keys) vs 0.91 (target native) at 400 steps
• Zero GDN state fails: recall 0.13 despite similar val KL (0.078 vs 0.063)—KL alone would hide the failure
• Longer distillation (1000 steps) lowers KL but not recall

Swap-In Analysis

• Exact recall lives in a few deep MLA layers (layer 19 of Kimi-Linear restores 100% recall)
• Neither attention/MLA alone nor states alone suffice: both lose every needle
• Pair B's residual is half attention, half state

Token Invertibility

Layer-0 MLA latent identifies tokens with top-1 accuracy 1.000 (also for unseen IDs). Every cached layer identifies most tokens with linear read-out (0.5–0.8 typical). Implication: KV caches are not anonymized representations—they should be protected like the prompt itself.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Per-object translation mechanisms: The paper establishes that cache translation is not a single recipe but requires object-specific approaches: token-wise maps for per-token objects, closed-form state correction for same-geometry states, replay for missing states, and token identity for exact recall.

2. Capture point matters: The pre-norm, un-rotated capture (①) generalizes beyond training lengths while the engine-native cache (③) fails when source and target rotate keys differently. This is a fundamental insight about where translation should read.

3. Token identity from cache: Demonstrates that MLA layer-0 latents are invertible to token IDs, enabling exact recall without external information—with privacy implications.

4. Behavioral leakage: Translated states carry source style and safety policy features that KL and accuracy metrics do not detect—a subtle failure mode for quality evaluation.

Practical Implications

• Model routing: Routers can switch between models at a fraction of prefill cost (×18.7–46.8 speed-up at 64K)
• Agent pipelines: Conversations can hand off between models without text summaries or re-prefill
• Serving fleets: Migration to new model generations becomes cheaper
• Flash→flagship direction (hybrid→full-MLA) is practically solved at native quality—this is the direction of GLM-5.3-Flash → GLM-5.2/5.3 and Kimi-K3 → Kimi-K2.6

Scaling Hypothesis

The per-token translation cost depends on cached objects (576×576 per MLA layer pair), not hidden size or active parameters, while prefill grows with active parameters. The authors hypothesize speed-ups grow with target size to 300B+ flagships, with caveats: replay-based state synthesis grows with state size (3.4× for GLM-5.3-Flash, 10.3× for Kimi-K3), and sparse attention makes flagship long-context prefill cheaper.

---

Conclusion

KV-cache translation extends from GQA to all cache objects of hybrid-attention models, but not with one mechanism:

• MLA latents: translate token-wise, best from pre-norm, un-rotated capture
• Exact recall: needs token identity, which the MLA layer-0 latent supplies
• Same-geometry recurrent states: copy + closed-form correction at O(1) cost
• Sparse-attention indexer keys: can be predicted without losing the needle on tested pairs

The hybrid→full-MLA direction was indistinguishable from native quality on every task metric, with speed-ups growing to ×18.7 at 64K.

Key limitations: Scale (pairs fit one machine; flagships are 6–35× larger), single tokenizer per family, sparse attention rehearsed only on QSA (one seed), Pair A forward below native, and behavioral leakage undetected by KL.

Future work: Testing at 300B+ scale is the hypothesis the evidence supports and the next experiment should test. Untested gaps include GLM's wider latent expansion (78/11 = 7.1 target latents per source latent), larger states to replay, Kimi-K3's residual-stream changes, and DSA on MLA caches.