# Looping Deeper Without Growing the KV Cache: Recurrent Depth for Causal Encoder-Decoders

> Looped CED places recurrence in the encoder and uses stochastic-depth training to achieve lossless cross-round KV sharing, enabling 36.8x smaller memory and 3.6x higher throughput.

- **Source:** [arXiv](https://arxiv.org/abs/2611.13378)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/lTqmrT
- **Whiteboard:** https://picx.dev/p/lTqmrT/image

## Summary

# Looping Deeper Without Growing the KV Cache: Recurrent Depth for Causal Encoder-Decoders

## Summary (Overview)

- **Core contribution**: The paper introduces **Looped CED**, a novel architecture that combines recurrent-depth computation (looped Transformers) with causal encoder-decoder (CED) KV consolidation to achieve deeper computation without increasing memory footprint.
- **Key finding on loop placement**: Recurrence must be placed in the **encoder** (not decoder) to iteratively refine global KV representations, matching looped dense Transformer quality while preserving long-context scaling.
- **Lossless KV sharing**: Training with **stochastic recurrence depth** induces a contractive fixed-point equilibrium, enabling lossless cross-round KV sharing where only the last round's KV cache is retained.
- **Memory savings**: Looped CED retains only **1/6 of the global KV** of a fixed-point looped dense model and **1/20** of a per-round cached one, achieving zero KV overhead over unlooped CED.
- **Production-scale validation**: At 552B parameters (DeepSeek-V4.1-Flash), Looped CED fits 38 concurrent 128K-token requests on a 4×GB200 node (vs. OOM for baseline), serving **3.6× higher throughput** and cutting multi-turn agent TTFT by up to **29.7×**.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Looped language models scale compute without adding parameters by repeatedly applying a weight-tied layer block across recurrent rounds. However, in production serving, memory capacity and decode throughput are bottlenecked by the **resident KV cache per request**, not parameter count alone.

Recent system-co-designed architectures replace homogeneous full-context layers with asymmetric KV routing. The **causal encoder-decoder (CED)** in DeepSeek-V4.1-Flash equips every layer with bounded sliding-window attention (SWA) while restricting full-context (global) KV generation to a few encoder layers; all decoder layers share a single global KV projected once from the final encoder output.

### Key Design Questions

1. **Where** in the asymmetric CED stack should recurrence be placed to maximize modeling capability?
2. **How** can a looped CED be trained and cached so that multi-round recurrence adds **zero resident KV overhead** without sacrificing quality?

### Theoretical Foundation

**Looped language models** follow a "sandwich" topology with non-recurrent prelude $\mathcal{P}$, weight-tied recurrent core $\mathcal{R}$, and non-recurrent coda $\mathcal{C}$:

$$
e = \mathcal{P}(x), \qquad h_r = \mathcal{R}(h_{r-1}, e), \qquad y = \mathcal{C}(h_T) \tag{1}
$$

where $h_r \in \mathbb{R}^d$ is the per-token state after round $r$ of $T$ (hidden size $d$).

The recurrent core uses **gated input injection** (from Parcae):

$$
\mathcal{R}(h_{r-1}, e) = f\left(\bar{A} \odot h_{r-1} + \Delta \odot (B\tilde{e})\right) \tag{2}
$$

where $f$ is one core pass, $\odot$ is element-wise product, learned $\Delta \in \mathbb{R}^d_{>0}$, $A \in \mathbb{R}^d_{<0}$, $B \in \mathbb{R}^{d \times d}$ give step size, decay $\bar{A} = \exp(\Delta \odot A)$, and input projection.

---

## Methodology

### Architectural Configurations

The 12-layer experimental configuration splits evenly into a 6-layer encoder (layers 0–5) and 6-layer decoder (layers 6–11):

| Component | Configuration |
|-----------|--------------|
| All layers | SWA cache over $w=128$ token window |
| Encoder layers 0–1 | SWA only |
| Encoder layer 2 | Sole encoder global KV source (read by layers 3–5) |
| Boundary layer 6 | Projects shared decoder global KV from encoder output |
| Decoder layers 7–11 | Read shared decoder global KV |

### Two Loop Placements

**Encoder loop (Iterative Global KV Refinement)**: Recurrent core resides in encoder, encompassing at least one global KV source layer. All $T$ rounds refine global KV representations — prefill runs all $T$ encoder rounds.

**Decoder loop (Static-Context Iterative Readout)**: Recurrent core in decoder, downstream of boundary projection. All global KV entries finalized before loop; rounds repeatedly query static global KV. Prefill compute unchanged.

### Compute Asymmetry (12-layer config, $T=8$)

- **Unlooped CED prefill**: 6 layer passes/token (vs. 12 for dense)
- **Decoder loop prefill**: 6 passes (unchanged)
- **Encoder loop prefill**: 34 layer passes/token
- **Decode (both placements)**: 40 passes/token (vs. 12 unlooped)

### Stochastic Depth Training for Fixed-Point Convergence

To enable lossless last-round KV caching, training samples **stochastic recurrence depth** per sequence from a Poisson distribution with mean 8 (clipped to ≥1, backpropagating through final 4 rounds), while executing deterministic 8 rounds at inference. This pressures the core to settle into a depth-invariant equilibrium.

### Training Setup

- Models trained from scratch on FineWeb-Edu using Muon and AdamW
- Scales: 90M, 150M, 340M parameters ($d \in \{768, 1024, 1536\}$)
- Standard budget: ~40 tokens per non-embedding parameter
- Extended budget: 29.6B tokens for 150M models
- Evaluation: FineWeb-Edu BPB, DCLM CORE/CORE-Extended, plus position-wise BPB on long documents (PG19, FinePDFs, GovReport)

---

## Empirical Validation / Results

### Where to Loop: Encoder vs. Decoder

**Key finding**: Looping the CED encoder yields substantial gains, while decoder loops recover only 16–40% of the encoder loop's gain across scales (even after 5× longer training, only 38%).

**Table 1: Validation BPB of 150M models (6.0B tokens, T=8 rounds)**

| Model | Params (M) | KV (KB) | BPB | Gain |
|-------|-----------|---------|-----|------|
| Dense, no loop | 151.0 | 48 | 0.8747 | - |
| Dense, loop L2–5 | 153.1 | 48 | 0.8461 | 0.0287 |
| Dense, loop L4–7 (Parcae) | 153.1 | 48 | 0.8514 | 0.0233 |
| CED, no loop | 155.2 | 8 | 0.8716 | - |
| **CED, encoder loop L2–5** | **157.3** | **8** | **0.8468** | **0.0248** |
| CED, decoder loop L7–10 | 157.3 | 8 | 0.8648 | 0.0068 |
| CED, encoder loop, fixed rounds | 157.3 | 8 | 0.8487 | 0.0229 |
| YOCO-U style, no loop | 153.1 | 4 | 0.8745 | - |
| YOCO-U style, loop L2–5 | 155.2 | 4 | 0.8506 | 0.0239 |

### Quality Parity with Looped Dense Transformers

Looped CED matches looped dense Transformer validation BPB **within 0.001** at all scales while retaining only **1/6 of the resident global KV**, capturing 89%, 86%, and 93% of the dense loop gain at 90M, 150M, and 340M.

### Long-Context Evaluation

- **Looped CED preserves full-context scaling**: Quality gap between CED encoder loop and dense loop remains invariant to context length across all benchmarks
- **YOCO-U-style ablation fails**: Looping only SWA layers (no global KV source) forfeits 30% of long-range gain beyond the SWA window
- **Copy test**: Encoder loop removes recall deficit (−0.084 to −0.160 BPB vs. unlooped CED); decoder loop barely changes recall; YOCO-U loop recalls worst

### Lossless Cross-Round KV Sharing

**Table 3: Per-round vs. last-round KV caching (validation BPB)**

| Model | Scale | Per | Last | Last−Per |
|-------|-------|-----|------|----------|
| CED, encoder loop | 90M, 3.4B | 0.8921 | 0.8919 | −0.0002 |
| CED, encoder loop | 150M, 6.0B | 0.8465 | 0.8463 | −0.0002 |
| CED, encoder loop | 340M, 13.6B | 0.7965 | 0.7965 | 0.0000 |
| CED, encoder loop | 150M, 29.6B | 0.8124 | 0.8123 | −0.0001 |
| Dense, loop L2–5 | 150M, 6.0B | 0.8457 | 0.8460 | +0.0003 |
| **Fixed-depth CED encoder loop** | **150M, 6.0B** | **0.8484** | **0.8801** | **+0.0317** |

**Critical finding**: Deterministic fixed-depth training causes last-round KV sharing to **fail catastrophically** (+0.0317 BPB degradation), entering a limit cycle of period 4. Stochastic-depth training induces geometric contraction toward a fixed point, making cross-round sharing lossless (<0.001 BPB change).

### Extended Training and Downstream Performance

**Table 2: Extended training (29.6B tokens) and downstream accuracy**

| Model | $BPB_{train}$ | BPB | CORE | CORE-Ext. |
|-------|-------------|-----|------|-----------|
| CED, no loop | 0.8386 | 0.8401 | 19.09 | 12.32 |
| CED, encoder loop | 0.8106 | 0.8124 | **21.63** | **13.54** |
| CED, decoder loop | 0.8278 | 0.8295 | 21.56 | 13.65 |
| Dense, no loop | 0.8389 | 0.8406 | 18.50 | 11.57 |
| Dense, loop L4–7 | 0.8157 | 0.8178 | 22.60 | 14.84 |

Encoder loop improves DCLM CORE by **+2.54 points** (95% CI [+1.4, +3.6]) and CORE-Extended by +1.22, matching the magnitude of 370M Parcae's gains at identical token budget.

### Production Serving Analysis (552B DeepSeek-V4.1-Flash)

**Table 4: Production serving summary at T=8 on 4×GB200**

| Workload / Metric | Per-layer KV, per-round KV | CED, per-round KV | **Looped CED (Ours)** | Advantage |
|---|---|---|---|---|
| Global KV per Token (KB) | 60.0 | 8.5 | **1.6** | 36.8× smaller |
| Usable KV Pool per Node (M tok) | 0.13 | 0.98 | **5.11** | 39× larger |
| 8K In / 1K Out (tok/s/GPU) | 50.9 | 142.5 | **185.1** | 3.6× higher |
| 32K HBM Decode (tok/s/GPU) | — | 289.3 | **818.5** | 2.8× higher |
| 128K Long-Context (tok/s/GPU) | OOM (0 req) | 7 reqs max | **199.2 (38 reqs)** | 38 reqs vs. OOM |
| Multi-Turn Agent TTFT (c=512) | — | 345.1 s | **11.6 s** | 29.7× lower |

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Loop placement principle**: Effective recurrence in a CED requires placing the loop **where global representations are constructed** rather than where they are merely read. The encoder loop's superiority stems from actively refining the global KV source itself across rounds.

2. **Fixed-point training**: Stochastic recurrence depth induces a **contractive fixed point** in the recurrent encoder, making the core depth-invariant in late rounds. This is fundamentally different from deterministic fixed-depth training, which produces limit cycles incompatible with last-round KV sharing.

3. **KV sharing compoundability**: Cross-layer KV sharing (CED) and cross-round KV sharing (last-round caching) **compound losslessly and multiplicatively**, establishing that architectural KV consolidation and temporal KV reuse are orthogonal and complementary.

### Practical Implications

1. **Production serving**: The 36.8× reduction in per-token KV footprint transforms an out-of-memory loop into a servable one, enabling:
   - 38 concurrent 128K-token requests on a single GB200 node
   - 3.6× higher throughput on standard 8K prompts
   - 29.7× lower multi-turn agent TTFT via prevention of prefix-cache eviction collapse

2. **Compute-memory tradeoff**: The cost of looping is compute, not memory — each round re-runs only the 18-layer recurrent core. Encoder loops pay a prefill premium but match decoder loops in decode cost.

3. **Asymmetric prefill–decode recurrence**: Stochastic-depth training enables $T_p < T_d$ (fewer prefill rounds), retaining 99% of quality improvement while halving prefill cost multiplier from 5.1× to 2.7×.

---

## Conclusion

This work establishes **Looped CED** as a principled approach to combining recurrent-depth computation with asymmetric KV consolidation:

1. **Where to loop**: Recurrence must be in the CED encoder to actively refine global KV sources, matching looped dense Transformer quality while preserving long-context scaling.

2. **How to loop**: Stochastic recurrence depth training induces a contractive fixed point, enabling **lossless cross-round KV sharing** with zero additional memory over unlooped models.

3. **Production impact**: Dual KV savings (cross-layer + cross-round) shrink resident KV by 36.8×, delivering 3.1×–16.5× higher throughput and 13.6×–29.7× lower multi-turn latency.

### Future Directions

- Pretraining frontier-scale looped CED MoE end-to-end
- Incorporating lightweight post-training alignment to guarantee fixed-point convergence at extreme scale
- Evaluation on broader long-context reasoning benchmarks
- Extending to disaggregated or large expert-parallel serving deployments

### Limitations

- Controlled pretraining ablations limited to dense models ≤340M parameters (2048-token context)
- Serving results measure cost, not quality (dummy/original weights not trained to loop)
- Per-round KV emulated by capping KV pool
- Encoder loop costs more per request on prompt-heavy workloads

---

_Markdown view of https://picx.dev/p/lTqmrT, served by PicX — AI-generated visual whiteboard summaries of research papers._