# One Model to Loop Them All: Elastic Latent Recurrence for Language Models

> Elastic Latent Recurrence eliminates the one-loop tax in looped language models, matching non-recurrent baselines at one loop while gaining up to +7.25 accuracy points with additional loops.

- **Source:** [arXiv](https://arxiv.org/abs/2610.12582)
- **Published:** 2026-10-12
- **Permalink:** https://picx.dev/p/JMPoxM
- **Whiteboard:** https://picx.dev/p/JMPoxM/image

## Summary

## Summary (Overview)

- **Elastic Latent Recurrence (ELR)** is a training recipe for looped (latent recurrent) language models that eliminates the "one-loop tax"—the accuracy gap at a single loop compared to a non-recurrent model of equal size—while enabling inference-time compute scaling via additional loops.
- Three core design choices achieve this: (1) a **gated adaptive-delta state update** with learned per-token/channel step sizes, (2) **within-step mixing of loop counts** via one-loop rank overrides, and (3) **loop-conditioned modulation** that conditions each pass on its loop index and budget.
- ELR matches or exceeds non-recurrent baselines at one loop (within 0.52 points at 4B dense; +1.04 points at 30B-A3B MoE) and gains significant accuracy with additional loops (+7.25 and +4.75 points at four loops).
- Techniques to reduce cost: **delayed mixed-depth training** (only final quarter of pretraining), **cross-loop route replay** for MoE models (11.7% faster training), and **last-loop KV caching** (keeps attention memory at one-loop level).
- The full recipe yields **Koi-4B**, a post-trained 4B model scoring an estimated 18 on the Artificial Analysis Intelligence Index at one loop (matching Qwen3.5-4B) and 29 at four loops.

## Introduction and Theoretical Foundation

Latent recurrent (looped) language models apply a shared block of layers repeatedly, allowing a single checkpoint to trade inference compute for accuracy by varying the loop count $k$. This property, called **elastic depth**, enables control of the quality vs. test-time-compute tradeoff without generating extra tokens (unlike chain-of-thought reasoning).

**The one-loop tax problem:** Existing elastic-depth models fall short of non-recurrent baselines at $k=1$. For example, Huginn LM trails its fixed-depth baseline by 11.5 points on ARC-Easy (34.9 vs. 46.4) even after 800B tokens. This tax means a separate non-recurrent model remains preferable for latency-sensitive deployment.

**Architecture:** A latent recurrent model partitions a backbone into three blocks:
- Input block $\mathcal{I}$ with $L_{\text{in}}$ layers
- Recurrent block $\mathcal{R}_\theta$ with $L_{\text{rec}}$ layers (shared parameters $\theta$)
- Output block $\mathcal{O}$ with $L_{\text{out}}$ layers

The forward pass with loop count $k$ traverses:

$$D_{\text{eff}}(k) = L_{\text{in}} + k L_{\text{rec}} + L_{\text{out}}\tag{1}$$

At $k=1$, every unique layer executes exactly once, matching a non-recurrent model in both parameters and compute.

## Methodology

### 3.1 Adaptive-Delta State Update

The recurrent state is initialized as $s_0 = h^{\text{in}}$ (input block output). At each loop $i$, the state update uses a **gated adaptive-delta rule**:

$$
\begin{array}{c} s_i = \text{Update}(s_{i-1}, r_i, \tilde{s}_i) \\ = s_{i-1} + \beta_i \odot (r_i - s_{i-1}) \end{array}\tag{2}

where $\tilde{s}_i = s_{i-1} + h^{\text{in}}$, $r_i = \mathcal{R}_\theta(\tilde{s}_i)$, and the gate is:

$$\beta_i = \sigma(W_\beta \tilde{s}_i + b_\beta) \odot \text{softplus}(\gamma)$$

The parameters $W_\beta$, $b_\beta$, $\gamma$ are shared across loops, adding only $d^2 + 2d$ parameters for hidden dimension $d$. Initialized to zero (with $\gamma=0$), the initial step size is ~0.35.

### 3.2 Mixed-Depth Training with One-Loop Rank Overrides

Loop counts are sampled uniformly ($p(k) = 1/K$ for $k \in \{1,\ldots,K\}$), but a rotating fraction $f$ of data-parallel ranks **force** $k=1$ on every microbatch. This ensures one-loop and deeper gradients are averaged in the same optimizer update:

$$p_1 = \frac{1}{K} + \left(1 - \frac{1}{K}\right) f, \quad \bar{k} = f + (1-f) \frac{K+1}{2}\tag{3}$$

The override keeps $p_1 \approx 1/3$, balancing one-loop accuracy against gains at deeper loops.

### 3.3 Loop-Conditioned Modulation

Each loop $i$ of $k$ is conditioned on progress $\tau_i = i/k$ and budget $\kappa = k/K$:

$$c_i = \text{MLP}([\phi(\tau_i); \phi(\kappa)])$$

The conditioning vector modulates each recurrent/output layer via FiLM-style operations:

$$
\begin{array}{l} \tilde{x} = (1 + \lambda(c_i)) \odot \text{RMSNorm}(x) + \mu(c_i) \\ \hat{x} = \tilde{x} + A(g(c_i) \odot B\tilde{x}) \\ y = x + (1 + \omega(c_i)) \odot f(\hat{x}) \end{array}\tag{4}

All conditioning heads are **zero-initialized** at the switch to mixed-depth training, so the model initially preserves the one-loop checkpoint's function exactly.

### 3.4 Cost Reduction Techniques

- **Delayed mixed-depth training:** Train at $k=1$ for the first 75% of tokens, then apply mixed depths. Total training compute relative to baseline:

$$(1-\rho) + \rho \frac{D_{\text{eff}}(\bar{k})}{D_{\text{eff}}(1)}\tag{5}$$

For the 4B configuration, this costs ~1.16× baseline FLOPs vs. 1.62× for mixed-depth from scratch.

- **Cross-loop route replay (MoE):** Compute expert assignments once on the first loop, reuse on later loops. Cuts mixed-depth update time by 11.7%.

- **Last-loop KV caching:** Keep a single attention KV cache that each loop overwrites, holding memory at the one-loop level.

## Empirical Validation / Results

### 4.2 Main Results (Pretraining)

Models: GDP-Hybrid-4B (dense, 4B params, 1T tokens) and GDP-Hybrid-30B-A3B (MoE, 30B total/3B active, 300B tokens).

| Model | Ten-Task Avg at $k=1$ | Ten-Task Avg at $k=4$ | Gain ($k=1 \to 4$) |
|---|---|---|---|
| GDP-Hybrid-4B ELR | 65.13 | 72.38 | +7.25 |
| GDP-Hybrid-4B Baseline | 65.65 | — | — |
| GDP-Hybrid-30B-A3B ELR | 66.08 | 70.83 | +4.75 |
| GDP-Hybrid-30B-A3B Baseline | 65.04 | — | — |

**Diminishing returns:** Two and three loops capture 79% and 96% of the gain (4B), and 87% and 91% (30B-A3B).

### 4.3 Cross-Loop Route Replay Results

| Routing | Δ Val Loss at $k=4$ | Δ Training Update Time (Mixed-depth) | Δ Training Update Time (Fixed $k=4$) | Δ Inference Forward Time |
|---|---|---|---|---|
| (A) Dynamic routing | reference | reference | reference | reference |
| (B) Inference-only replay | +0.0055 | — | — | -1.2% |
| (C) Replay training | +0.0038 | -11.7% | -19.6% | -1.2% |

### 5.2 Koi-4B (Post-Trained)

- **AA-Index v4.1 estimates:** 18 at $k=1$, 26 at $k=2$, 28 at $k=3$, 29 at $k=4$.
- At $k=1$, matches Qwen3.5-4B; at $k=4$, exceeds Gemma4-26B-A4B (6.5× parameters) and trails only Qwen3.6-35B-A3B (9× parameters).
- Higher loop counts produce **shorter responses** (32.7% fewer output tokens at $k=4$) while scoring higher—latent depth substitutes for explicit chain-of-thought tokens.

### 5.3 Last-Loop KV Cache Results

At $k=4$, score reductions vs. per-loop caching range from 0–3% across benchmarks, with attention KV memory held at 1× (vs. 2.5× for per-loop caching, saving 4.5 GiB per 128K-token request).

### 6. Ablations (GPT-0.5B, 300B tokens)

- **State update rules:** Adaptive delta is the only rule matching baseline at $k=1$ and achieves lowest loss at all loop counts. Replace and residual rules perform worst.
- **Sampling schemes:** Independent-rank sampling (each rank draws its own $k$) gives lowest loss but 1.36× step time; one-loop overrides recover most of the benefit at synchronized step time.
- **One-loop overrides:** Without overrides ($f=0$), one-loop loss is +0.0225 above baseline; $f=0.15$ ($p_1=0.32$) is the smallest fraction that falls below baseline.
- **Delayed mixed-depth:** Starting at 75% of training tokens retains 84% of five-loop improvement at 1.20× training compute (vs. 1.79× from scratch).
- **Loop conditioning:** Leaves one-loop loss nearly unchanged (1.6748 → 1.6739), lowers loss by 0.009 at $k=2$ and 0.022 at $k=5$.

## Theoretical and Practical Implications

- **Architecture design:** The gated adaptive-delta update provides a principled way to make recurrent depth learnable without explicit stability constraints (unlike Parcae's linearized stability conditions or STARS' spectral radius regularization). The learned step size starts small (~0.35) and is trained end-to-end.
- **Training dynamics:** Within-step mixing of loop counts is crucial—synchronized sampling (all ranks share a depth per step) fails to preserve the one-loop endpoint even at the same marginal distribution. This suggests that gradient averaging across depths within a single update is essential for balanced optimization.
- **Practical deployment:** ELR enables a single checkpoint to serve multiple latency/quality tiers, eliminating the need for separate models. The late acquisition of elastic depth (final quarter of pretraining) makes this practical for large-scale training runs.
- **MoE efficiency:** Cross-loop route replay demonstrates that routing decisions can be reused across loops with minimal quality loss, offering a template for other recurrent MoE designs.
- **Memory efficiency:** Last-loop KV caching decouples memory footprint from inference depth, making deeper inference feasible on memory-constrained hardware.

## Conclusion

ELR demonstrates that looped language models can match non-recurrent baselines at one loop while gaining accuracy at additional loops, eliminating the one-loop tax. Three key choices enable this: the gated adaptive-delta update, within-step loop-count mixing, and loop-conditioned modulation. The recipe is cost-effective (delayed mixed-depth training at ~1.16× baseline compute), scalable to MoE models, and produces Koi-4B—a single checkpoint spanning 18 to 29 on the AA-Index purely by loop count.

**Open questions:**
- Gains diminish after the second loop; reweighting the loop-count distribution may recover them.
- Conditioning closes most (but not all) of the gap to fixed-depth specialists at endpoints.
- Combining ELR with per-request or per-token adaptive depth (complicated by budget conditioning) and post-training the MoE variant remain future work.

---

_Markdown view of https://picx.dev/p/JMPoxM, served by PicX — AI-generated visual whiteboard summaries of research papers._