Summary (Overview)

• Key finding 1: Trained hybrid language models exhibit a write–read asymmetry: writing a deep layer's memory of a token requires far less deep computation than predicting the next token. This enables a shallow prefill where prompt tokens (except the last) run only 63–89% of the layers.
• Key finding 2: Recurrent states in hybrid models have a state sink—a rank-one term along a fixed direction per head that holds 69–99.7% of the state energy pooled over heads. The remaining content is low-rank, enabling compact state representation.
• SEED method: Builds on both properties to create a serving method for frozen checkpoints—shallow in depth for the prefill (saving compute) and shallow in representation for the decode (saving HBM bytes).
• Empirical results: Nearly lossless on 63 of 66 checkpoint–exam cells across eleven official exams on six checkpoints of five production models (30B–320B parameters); delivers 1.2–1.4× throughput improvement for Qwen3.8-Flash-Next on NVIDIA GB300 GPUs.
• Key theoretical contribution: Proof that the Gated DeltaNet, Kimi Delta Attention, and Mamba-2 updates preserve the sink-plus-low-rank state form exactly, with an open-loop error bound independent of the number of compressions.

---

Introduction and Theoretical Foundation

Hybrid language models interleave a few softmax-attention layers with many linear-attention or state-space layers. Every layer maintains a memory of earlier tokens—either a key–value (KV) cache (attention layers) or a fixed-size recurrent state (linear-attention/state-space layers, one $d_k \times d_v$ matrix per head).

Serving bottlenecks: The prefill phase is compute-bound (processes all $N$ prompt tokens together), while decode is HBM-bandwidth-bound (reads weights and memories of every resident request per step). Both phases overspend:

• Prefill: Only the last prompt token's prediction is used; the other $N-1$ tokens run all $L$ layers only to write memories.
• Decode: Every recurrent head loads/stores its whole state at every step; on Qwen3.8-Flash-Next, states take 113.2 MB per request—as much as the attention cache of ~4.5K tokens.

Two structural properties discovered:

1. Writes are easier than reads: A layer writes its memory through a write path that reads only the residual stream entering it, whereas predicting the next token (a read) requires attention and feed-forward blocks of every deep layer. Feeding the anchor layer's output $h^{(k)}$ (at 63–89% depth) into deep layers' write paths without training moves the next-token distribution by KL < 0.1 nats/token, while an untrained early exit at the same depth is off by 6.8–11.1 nats/token.

2. State sink: In the median Gated DeltaNet head of Qwen3.8-Flash-Next, one direction holds 85% of state energy, close to the head's mean value $\bar{v}$. The linear recurrence accumulates this into a rank-one term $a\bar{v}^\top$—the state sink—whose coefficient follows the head's own recurrence.

---

Methodology

Roofline Cost Model

The paper uses a roofline model to count costs:

$$t_{\mathrm{step}} \gtrsim \frac{M_{\mathrm{w}}^{\mathrm{step}} + B Q_{\mathrm{r}}(T)}{\mathrm{HBM\;bandwidth}}, \qquad B \leq \frac{M_{\mathrm{HBM}} - M_{\mathrm{w}}}{M_{\mathrm{r}}(T)}$$

where $B$ is the batch size, $Q_{\mathrm{r}}(T)$ is per-request traffic, and $M_{\mathrm{r}}(T)$ is per-request memory.

SEED Architecture

Prefill (shallow in depth): The first $N-1$ prompt tokens run only layers $0..k-1$. Emitters—trained copies of the deep layers' write paths—write every deep attention cache and recurrent state from the anchor layer's output $h^{(k)}$. An optional compact prompt code encodes $h^{(k)}$ for prefix caching.

Decode (shallow in representation): Each recurrent state is stored as its sink plus rank-$R$ factors:

$$S = a\bar{v}^\top + UV^\top$$

where $\bar{v} \in \mathbb{R}^{d_v}$ is a per-head constant, $a \in \mathbb{R}^{d_k}$ is the sink coefficient, and $U \in \mathbb{R}^{d_k \times R}$, $V \in \mathbb{R}^{d_v \times R}$ are content factors.

Exact Update (Lemma 2)

The factored form is closed under Gated DeltaNet, Kimi Delta Attention, and Mamba-2 updates:

$$a^{+} = g\,a + \beta\,(\omega - g\,\kappa^\top a)\,\kappa,$$ $$U^{+} = [gU, \kappa], \qquad V^{+} = [V, \beta(\varepsilon - gVU^\top\kappa)]$$

where $\omega = v^\top\bar{v}/\|\bar{v}\|^2$ and $\varepsilon = v - \omega\bar{v}$. Each step appends one column to each factor—an exact rank-one update.

Open-Loop Error Bound (Corollary 4)

$$\|\tilde{S}_t - S_t\|_F \leq \frac{\bar{\epsilon}}{1 - \bar{g}^W} \leq \frac{\bar{\epsilon}}{1 - \bar{g}}$$

where $\bar{g} < 1$ is the largest decay and $\bar{\epsilon}$ is the largest single compression error. Repeated compressions do not compound in the open loop.

Training

One recipe trains all component sets against the frozen model (the only teacher), using KL divergence to the frozen checkpoint's next-token distribution. Stage 1 distills on fixed text; Stage 2 is on-policy using the teacher's and current set's replies.

---

Empirical Validation / Results

Exam Performance

On eleven official exams across six checkpoints, 63 of 66 checkpoint–exam cells lie within two standard errors of the original model or above it. Example results for Qwen3.8-Flash-Next (bf16):

State Compression Savings

Distributional Drift

• Shallow state drifts 0.002–0.007 nats/token (fp32, exact prefill)
• Official NVFP4 checkpoints drift 1.9–3.5× more (0.0105–0.0209 nats/token)
• Perplexity change of original model's replies: <0.6% for SEED vs. 1.2–2.1% for NVFP4 checkpoints
• Closed-loop drift grows only 1.2–1.4× from first to last 4K of 16K decode steps while compressions grow 4×

Serving Throughput

On Qwen3.8-Flash-Next with SGLang on NVIDIA GB300 GPUs (agentic-coding trace):

---

Theoretical and Practical Implications

Theoretical contributions:

• Lemma 1 (sink decomposition): Splitting each value along a fixed direction $\bar{v}$ decomposes the state into sink coefficient $a_t$ and content $C_t$, each following the head's own recurrence: $$a_t = \Phi_t a_{t-1} + \beta_t \omega_t \kappa_t, \qquad C_t = \Phi_t C_{t-1} + \beta_t \kappa_t \varepsilon_t^\top$$
• Closed-form update: The sink-plus-low-rank form is invariant under the model's own updates—the split is never recomputed.
• Open-loop error bound: Errors do not compound with repeated compressions; the bound depends only on the largest decay horizon.

Practical implications:

• SEED works on frozen checkpoints—no retraining of the original model needed
• Attention caches remain uncompressed (composes with existing KV-cache techniques)
• The state sink acts as a bias and gain through the head's normalization, analogous to the attention sink in softmax layers
• In disaggregated serving, shallow prefill relieves the prefill instance; shallow states relieve the recurrent-state pool limiting the batch in aggregated serving
• fp16 factors further halve state bytes with negligible drift increase

---

Conclusion

SEED (Shallow Encoding Emits Deep Memories) exploits two emergent properties of trained hybrid language models to reduce serving costs:

1. Write–read asymmetry: Deep memories can be written from a shallow anchor layer's residual stream, so prompt tokens run as few as 63% of layers.
2. State sink: Recurrent states are a rank-one sink plus low-rank working memory, a form preserved exactly by the model's own updates.

The method is nearly lossless on official exams across five production models (30B–320B parameters) and delivers 1.2–1.4× throughput gains. Neither property was designed into the models—both emerge from training.

Future directions: fp16 shallow states, prefix caches storing prompt codes in place of deep caches (up to 22% fewer bytes), and training with periodic compression in the loop. The open question of how shallow a residual stream can determine deep memories as a function of architecture and training remains.