# Shallow Encoding Emits Deep Memories

> SEED exploits write-read asymmetry and state sinks in hybrid LLMs to prefill with 63-89% of layers and compress recurrent states by 2-4x, nearly losslessly improving throughput 1.2-1.4x.

- **Source:** [arXiv](https://arxiv.org/abs/2611.06084)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/IqMc3k
- **Whiteboard:** https://picx.dev/p/IqMc3k/image

## Summary

## Summary (Overview)

- **Key finding 1:** Trained hybrid language models exhibit a **write–read asymmetry**: writing a deep layer's memory of a token requires far less deep computation than predicting the next token. This enables a shallow prefill where prompt tokens (except the last) run only 63–89% of the layers.
- **Key finding 2:** Recurrent states in hybrid models have a **state sink**—a rank-one term along a fixed direction per head that holds 69–99.7% of the state energy pooled over heads. The remaining content is low-rank, enabling compact state representation.
- **SEED method:** Builds on both properties to create a serving method for frozen checkpoints—shallow in depth for the prefill (saving compute) and shallow in representation for the decode (saving HBM bytes).
- **Empirical results:** Nearly lossless on 63 of 66 checkpoint–exam cells across eleven official exams on six checkpoints of five production models (30B–320B parameters); delivers 1.2–1.4× throughput improvement for Qwen3.8-Flash-Next on NVIDIA GB300 GPUs.
- **Key theoretical contribution:** Proof that the Gated DeltaNet, Kimi Delta Attention, and Mamba-2 updates preserve the sink-plus-low-rank state form exactly, with an open-loop error bound independent of the number of compressions.

---

## Introduction and Theoretical Foundation

Hybrid language models interleave a few softmax-attention layers with many linear-attention or state-space layers. Every layer maintains a memory of earlier tokens—either a key–value (KV) cache (attention layers) or a fixed-size recurrent state (linear-attention/state-space layers, one $d_k \times d_v$ matrix per head).

**Serving bottlenecks:** The prefill phase is compute-bound (processes all $N$ prompt tokens together), while decode is HBM-bandwidth-bound (reads weights and memories of every resident request per step). Both phases overspend:

- **Prefill:** Only the last prompt token's prediction is used; the other $N-1$ tokens run all $L$ layers only to write memories.
- **Decode:** Every recurrent head loads/stores its whole state at every step; on Qwen3.8-Flash-Next, states take 113.2 MB per request—as much as the attention cache of ~4.5K tokens.

**Two structural properties discovered:**

1. **Writes are easier than reads:** A layer writes its memory through a write path that reads only the residual stream entering it, whereas predicting the next token (a read) requires attention and feed-forward blocks of every deep layer. Feeding the anchor layer's output $h^{(k)}$ (at 63–89% depth) into deep layers' write paths without training moves the next-token distribution by KL < 0.1 nats/token, while an untrained early exit at the same depth is off by 6.8–11.1 nats/token.

2. **State sink:** In the median Gated DeltaNet head of Qwen3.8-Flash-Next, one direction holds 85% of state energy, close to the head's mean value $\bar{v}$. The linear recurrence accumulates this into a rank-one term $a\bar{v}^\top$—the state sink—whose coefficient follows the head's own recurrence.

---

## Methodology

### Roofline Cost Model

The paper uses a roofline model to count costs:

$$t_{\mathrm{step}} \gtrsim \frac{M_{\mathrm{w}}^{\mathrm{step}} + B Q_{\mathrm{r}}(T)}{\mathrm{HBM\;bandwidth}}, \qquad B \leq \frac{M_{\mathrm{HBM}} - M_{\mathrm{w}}}{M_{\mathrm{r}}(T)}$$

where $B$ is the batch size, $Q_{\mathrm{r}}(T)$ is per-request traffic, and $M_{\mathrm{r}}(T)$ is per-request memory.

### SEED Architecture

**Prefill (shallow in depth):** The first $N-1$ prompt tokens run only layers $0..k-1$. **Emitters**—trained copies of the deep layers' write paths—write every deep attention cache and recurrent state from the anchor layer's output $h^{(k)}$. An optional compact **prompt code** encodes $h^{(k)}$ for prefix caching.

**Decode (shallow in representation):** Each recurrent state is stored as its sink plus rank-$R$ factors:

$$S = a\bar{v}^\top + UV^\top$$

where $\bar{v} \in \mathbb{R}^{d_v}$ is a per-head constant, $a \in \mathbb{R}^{d_k}$ is the sink coefficient, and $U \in \mathbb{R}^{d_k \times R}$, $V \in \mathbb{R}^{d_v \times R}$ are content factors.

### Exact Update (Lemma 2)

The factored form is closed under Gated DeltaNet, Kimi Delta Attention, and Mamba-2 updates:

$$a^{+} = g\,a + \beta\,(\omega - g\,\kappa^\top a)\,\kappa,$$
$$U^{+} = [gU, \kappa], \qquad V^{+} = [V, \beta(\varepsilon - gVU^\top\kappa)]$$

where $\omega = v^\top\bar{v}/\|\bar{v}\|^2$ and $\varepsilon = v - \omega\bar{v}$. Each step appends one column to each factor—an exact rank-one update.

### Open-Loop Error Bound (Corollary 4)

$$\|\tilde{S}_t - S_t\|_F \leq \frac{\bar{\epsilon}}{1 - \bar{g}^W} \leq \frac{\bar{\epsilon}}{1 - \bar{g}}$$

where $\bar{g} < 1$ is the largest decay and $\bar{\epsilon}$ is the largest single compression error. Repeated compressions do not compound in the open loop.

### Training

One recipe trains all component sets against the frozen model (the only teacher), using KL divergence to the frozen checkpoint's next-token distribution. Stage 1 distills on fixed text; Stage 2 is on-policy using the teacher's and current set's replies.

---

## Empirical Validation / Results

### Exam Performance

On eleven official exams across six checkpoints, **63 of 66 checkpoint–exam cells lie within two standard errors of the original model or above it**. Example results for Qwen3.8-Flash-Next (bf16):

| Exam | Original | SEED | Difference |
|-------|----------|------|------------|
| GPQA-Diamond | 91.9 | 91.5 | -0.3±0.4 |
| AIME 2024+25 | 99.4 | 98.8 | -0.6±0.5 |
| MATH-500 | 99.3 | 98.8 | -0.5±0.1 |
| MMLU | 92.5 | 92.6 | +0.1±0.1 |
| IFEval | 91.7 | 91.1 | -0.6±0.3 |

### State Compression Savings

| Model | State at R (MB) | Orig. State (MB) | Ratio |
|-------|-----------------|------------------|-------|
| Qwen3.8-Flash-Next | 29.2 | 113.2 | 3.9× |
| Nemotron-3.5-Lightning | 18.8 | 48.2 | 2.6× |
| Kimi-Linear | 21.3 | 41.9 | 2.0× |
| GLM-5.3-Flash | 72.4 | 142.6 | 2.0× |
| Nemotron-3.5-Super | 65.5 | 167.8 | 2.6× |

### Distributional Drift

- Shallow state drifts **0.002–0.007 nats/token** (fp32, exact prefill)
- Official NVFP4 checkpoints drift **1.9–3.5× more** (0.0105–0.0209 nats/token)
- Perplexity change of original model's replies: **<0.6%** for SEED vs. **1.2–2.1%** for NVFP4 checkpoints
- Closed-loop drift grows only 1.2–1.4× from first to last 4K of 16K decode steps while compressions grow 4×

### Serving Throughput

On Qwen3.8-Flash-Next with SGLang on NVIDIA GB300 GPUs (agentic-coding trace):

| Configuration | Peak Ratio | At 40 tok/s Ratio |
|---------------|-----------|-------------------|
| Aggregated NVFP4 TP1 | 1.20× | 1.23× |
| Aggregated bf16 TP2 | 1.16× | 1.26× |
| Disaggregated bf16 1P:1D | 1.42× | 1.42× |

---

## Theoretical and Practical Implications

**Theoretical contributions:**

- **Lemma 1 (sink decomposition):** Splitting each value along a fixed direction $\bar{v}$ decomposes the state into sink coefficient $a_t$ and content $C_t$, each following the head's own recurrence:
  $$a_t = \Phi_t a_{t-1} + \beta_t \omega_t \kappa_t, \qquad C_t = \Phi_t C_{t-1} + \beta_t \kappa_t \varepsilon_t^\top$$
- **Closed-form update:** The sink-plus-low-rank form is invariant under the model's own updates—the split is never recomputed.
- **Open-loop error bound:** Errors do not compound with repeated compressions; the bound depends only on the largest decay horizon.

**Practical implications:**

- SEED works on **frozen checkpoints**—no retraining of the original model needed
- Attention caches remain uncompressed (composes with existing KV-cache techniques)
- The state sink acts as a bias and gain through the head's normalization, analogous to the attention sink in softmax layers
- In disaggregated serving, shallow prefill relieves the prefill instance; shallow states relieve the recurrent-state pool limiting the batch in aggregated serving
- fp16 factors further halve state bytes with negligible drift increase

---

## Conclusion

SEED (Shallow Encoding Emits Deep Memories) exploits two emergent properties of trained hybrid language models to reduce serving costs:

1. **Write–read asymmetry:** Deep memories can be written from a shallow anchor layer's residual stream, so prompt tokens run as few as 63% of layers.
2. **State sink:** Recurrent states are a rank-one sink plus low-rank working memory, a form preserved exactly by the model's own updates.

The method is nearly lossless on official exams across five production models (30B–320B parameters) and delivers 1.2–1.4× throughput gains. Neither property was designed into the models—both emerge from training.

**Future directions:** fp16 shallow states, prefix caches storing prompt codes in place of deep caches (up to 22% fewer bytes), and training with periodic compression in the loop. The open question of how shallow a residual stream can determine deep memories as a function of architecture and training remains.

---

_Markdown view of https://picx.dev/p/IqMc3k, served by PicX — AI-generated visual whiteboard summaries of research papers._