Looping Deeper Without Growing the KV Cache: Recurrent Depth for Causal Encoder-Decoders

Summary (Overview)

• Core contribution: The paper introduces Looped CED, a novel architecture that combines recurrent-depth computation (looped Transformers) with causal encoder-decoder (CED) KV consolidation to achieve deeper computation without increasing memory footprint.
• Key finding on loop placement: Recurrence must be placed in the encoder (not decoder) to iteratively refine global KV representations, matching looped dense Transformer quality while preserving long-context scaling.
• Lossless KV sharing: Training with stochastic recurrence depth induces a contractive fixed-point equilibrium, enabling lossless cross-round KV sharing where only the last round's KV cache is retained.
• Memory savings: Looped CED retains only 1/6 of the global KV of a fixed-point looped dense model and 1/20 of a per-round cached one, achieving zero KV overhead over unlooped CED.
• Production-scale validation: At 552B parameters (DeepSeek-V4.1-Flash), Looped CED fits 38 concurrent 128K-token requests on a 4×GB200 node (vs. OOM for baseline), serving 3.6× higher throughput and cutting multi-turn agent TTFT by up to 29.7×.

---

Introduction and Theoretical Foundation

Background and Motivation

Looped language models scale compute without adding parameters by repeatedly applying a weight-tied layer block across recurrent rounds. However, in production serving, memory capacity and decode throughput are bottlenecked by the resident KV cache per request, not parameter count alone.

Recent system-co-designed architectures replace homogeneous full-context layers with asymmetric KV routing. The causal encoder-decoder (CED) in DeepSeek-V4.1-Flash equips every layer with bounded sliding-window attention (SWA) while restricting full-context (global) KV generation to a few encoder layers; all decoder layers share a single global KV projected once from the final encoder output.

Key Design Questions

1. Where in the asymmetric CED stack should recurrence be placed to maximize modeling capability?
2. How can a looped CED be trained and cached so that multi-round recurrence adds zero resident KV overhead without sacrificing quality?

Theoretical Foundation

Looped language models follow a "sandwich" topology with non-recurrent prelude $\mathcal{P}$, weight-tied recurrent core $\mathcal{R}$, and non-recurrent coda $\mathcal{C}$:

$$e = \mathcal{P}(x), \qquad h_r = \mathcal{R}(h_{r-1}, e), \qquad y = \mathcal{C}(h_T) \tag{1}$$

where $h_r \in \mathbb{R}^d$ is the per-token state after round $r$ of $T$ (hidden size $d$).

The recurrent core uses gated input injection (from Parcae):

$$\mathcal{R}(h_{r-1}, e) = f\left(\bar{A} \odot h_{r-1} + \Delta \odot (B\tilde{e})\right) \tag{2}$$

where $f$ is one core pass, $\odot$ is element-wise product, learned $\Delta \in \mathbb{R}^d_{>0}$, $A \in \mathbb{R}^d_{<0}$, $B \in \mathbb{R}^{d \times d}$ give step size, decay $\bar{A} = \exp(\Delta \odot A)$, and input projection.

---

Methodology

Architectural Configurations

The 12-layer experimental configuration splits evenly into a 6-layer encoder (layers 0–5) and 6-layer decoder (layers 6–11):

Two Loop Placements

Encoder loop (Iterative Global KV Refinement): Recurrent core resides in encoder, encompassing at least one global KV source layer. All $T$ rounds refine global KV representations — prefill runs all $T$ encoder rounds.

Decoder loop (Static-Context Iterative Readout): Recurrent core in decoder, downstream of boundary projection. All global KV entries finalized before loop; rounds repeatedly query static global KV. Prefill compute unchanged.

Compute Asymmetry (12-layer config, $T=8$)

• Unlooped CED prefill: 6 layer passes/token (vs. 12 for dense)
• Decoder loop prefill: 6 passes (unchanged)
• Encoder loop prefill: 34 layer passes/token
• Decode (both placements): 40 passes/token (vs. 12 unlooped)

Stochastic Depth Training for Fixed-Point Convergence

To enable lossless last-round KV caching, training samples stochastic recurrence depth per sequence from a Poisson distribution with mean 8 (clipped to ≥1, backpropagating through final 4 rounds), while executing deterministic 8 rounds at inference. This pressures the core to settle into a depth-invariant equilibrium.

Training Setup

• Models trained from scratch on FineWeb-Edu using Muon and AdamW
• Scales: 90M, 150M, 340M parameters ($d \in \{768, 1024, 1536\}$)
• Standard budget: ~40 tokens per non-embedding parameter
• Extended budget: 29.6B tokens for 150M models
• Evaluation: FineWeb-Edu BPB, DCLM CORE/CORE-Extended, plus position-wise BPB on long documents (PG19, FinePDFs, GovReport)

---

Empirical Validation / Results

Where to Loop: Encoder vs. Decoder

Key finding: Looping the CED encoder yields substantial gains, while decoder loops recover only 16–40% of the encoder loop's gain across scales (even after 5× longer training, only 38%).

Table 1: Validation BPB of 150M models (6.0B tokens, T=8 rounds)

Quality Parity with Looped Dense Transformers

Looped CED matches looped dense Transformer validation BPB within 0.001 at all scales while retaining only 1/6 of the resident global KV, capturing 89%, 86%, and 93% of the dense loop gain at 90M, 150M, and 340M.

Long-Context Evaluation

• Looped CED preserves full-context scaling: Quality gap between CED encoder loop and dense loop remains invariant to context length across all benchmarks
• YOCO-U-style ablation fails: Looping only SWA layers (no global KV source) forfeits 30% of long-range gain beyond the SWA window
• Copy test: Encoder loop removes recall deficit (−0.084 to −0.160 BPB vs. unlooped CED); decoder loop barely changes recall; YOCO-U loop recalls worst

Lossless Cross-Round KV Sharing

Table 3: Per-round vs. last-round KV caching (validation BPB)

Critical finding: Deterministic fixed-depth training causes last-round KV sharing to fail catastrophically (+0.0317 BPB degradation), entering a limit cycle of period 4. Stochastic-depth training induces geometric contraction toward a fixed point, making cross-round sharing lossless (<0.001 BPB change).

Extended Training and Downstream Performance

Table 2: Extended training (29.6B tokens) and downstream accuracy

Encoder loop improves DCLM CORE by +2.54 points (95% CI [+1.4, +3.6]) and CORE-Extended by +1.22, matching the magnitude of 370M Parcae's gains at identical token budget.

Production Serving Analysis (552B DeepSeek-V4.1-Flash)

Table 4: Production serving summary at T=8 on 4×GB200

---

Theoretical and Practical Implications

Theoretical Implications

1. Loop placement principle: Effective recurrence in a CED requires placing the loop where global representations are constructed rather than where they are merely read. The encoder loop's superiority stems from actively refining the global KV source itself across rounds.

2. Fixed-point training: Stochastic recurrence depth induces a contractive fixed point in the recurrent encoder, making the core depth-invariant in late rounds. This is fundamentally different from deterministic fixed-depth training, which produces limit cycles incompatible with last-round KV sharing.

3. KV sharing compoundability: Cross-layer KV sharing (CED) and cross-round KV sharing (last-round caching) compound losslessly and multiplicatively, establishing that architectural KV consolidation and temporal KV reuse are orthogonal and complementary.

Practical Implications

1. Production serving: The 36.8× reduction in per-token KV footprint transforms an out-of-memory loop into a servable one, enabling:

   • 38 concurrent 128K-token requests on a single GB200 node
   • 3.6× higher throughput on standard 8K prompts
   • 29.7× lower multi-turn agent TTFT via prevention of prefix-cache eviction collapse

2. Compute-memory tradeoff: The cost of looping is compute, not memory — each round re-runs only the 18-layer recurrent core. Encoder loops pay a prefill premium but match decoder loops in decode cost.

3. Asymmetric prefill–decode recurrence: Stochastic-depth training enables $T_p < T_d$ (fewer prefill rounds), retaining 99% of quality improvement while halving prefill cost multiplier from 5.1× to 2.7×.

---

Conclusion

This work establishes Looped CED as a principled approach to combining recurrent-depth computation with asymmetric KV consolidation:

1. Where to loop: Recurrence must be in the CED encoder to actively refine global KV sources, matching looped dense Transformer quality while preserving long-context scaling.

2. How to loop: Stochastic recurrence depth training induces a contractive fixed point, enabling lossless cross-round KV sharing with zero additional memory over unlooped models.

3. Production impact: Dual KV savings (cross-layer + cross-round) shrink resident KV by 36.8×, delivering 3.1×–16.5× higher throughput and 13.6×–29.7× lower multi-turn latency.

Future Directions

• Pretraining frontier-scale looped CED MoE end-to-end
• Incorporating lightweight post-training alignment to guarantee fixed-point convergence at extreme scale
• Evaluation on broader long-context reasoning benchmarks
• Extending to disaggregated or large expert-parallel serving deployments

Limitations

• Controlled pretraining ablations limited to dense models ≤340M parameters (2048-token context)
• Serving results measure cost, not quality (dummy/original weights not trained to loop)
• Per-round KV emulated by capping KV pool
• Encoder loop costs more per request on prompt-heavy workloads