Summary (Overview)

• Elastic Latent Recurrence (ELR) is a training recipe for looped (latent recurrent) language models that eliminates the "one-loop tax"—the accuracy gap at a single loop compared to a non-recurrent model of equal size—while enabling inference-time compute scaling via additional loops.
• Three core design choices achieve this: (1) a gated adaptive-delta state update with learned per-token/channel step sizes, (2) within-step mixing of loop counts via one-loop rank overrides, and (3) loop-conditioned modulation that conditions each pass on its loop index and budget.
• ELR matches or exceeds non-recurrent baselines at one loop (within 0.52 points at 4B dense; +1.04 points at 30B-A3B MoE) and gains significant accuracy with additional loops (+7.25 and +4.75 points at four loops).
• Techniques to reduce cost: delayed mixed-depth training (only final quarter of pretraining), cross-loop route replay for MoE models (11.7% faster training), and last-loop KV caching (keeps attention memory at one-loop level).
• The full recipe yields Koi-4B, a post-trained 4B model scoring an estimated 18 on the Artificial Analysis Intelligence Index at one loop (matching Qwen3.5-4B) and 29 at four loops.

Introduction and Theoretical Foundation

Latent recurrent (looped) language models apply a shared block of layers repeatedly, allowing a single checkpoint to trade inference compute for accuracy by varying the loop count $k$. This property, called elastic depth, enables control of the quality vs. test-time-compute tradeoff without generating extra tokens (unlike chain-of-thought reasoning).

The one-loop tax problem: Existing elastic-depth models fall short of non-recurrent baselines at $k=1$. For example, Huginn LM trails its fixed-depth baseline by 11.5 points on ARC-Easy (34.9 vs. 46.4) even after 800B tokens. This tax means a separate non-recurrent model remains preferable for latency-sensitive deployment.

Architecture: A latent recurrent model partitions a backbone into three blocks:

• Input block $\mathcal{I}$ with $L_{\text{in}}$ layers
• Recurrent block $\mathcal{R}_\theta$ with $L_{\text{rec}}$ layers (shared parameters $\theta$)
• Output block $\mathcal{O}$ with $L_{\text{out}}$ layers

The forward pass with loop count $k$ traverses:

$$D_{\text{eff}}(k) = L_{\text{in}} + k L_{\text{rec}} + L_{\text{out}}\tag{1}$$

At $k=1$, every unique layer executes exactly once, matching a non-recurrent model in both parameters and compute.

Methodology

3.1 Adaptive-Delta State Update

The recurrent state is initialized as $s_0 = h^{\text{in}}$ (input block output). At each loop $i$, the state update uses a gated adaptive-delta rule:

\begin{array}{c} s_i = \text{Update}(s_{i-1}, r_i, \tilde{s}_i) \\ = s_{i-1} + \beta_i \odot (r_i - s_{i-1}) \end{array}\tag{2} where $\tilde{s}_i = s_{i-1} + h^{\text{in}}$, $r_i = \mathcal{R}_\theta(\tilde{s}_i)$, and the gate is:

\beta_i = \sigma(W_\beta \tilde{s}i + b\beta) \odot \text{softplus}(\gamma)

The parameters $W_\beta$, $b_\beta$, $\gamma$ are shared across loops, adding only $d^2 + 2d$ parameters for hidden dimension $d$. Initialized to zero (with $\gamma=0$), the initial step size is ~0.35. ### 3.2 Mixed-Depth Training with One-Loop Rank Overrides Loop counts are sampled uniformly ($p(k) = 1/K$ for $k \in \{1,\ldots,K\}$), but a rotating fraction $f$ of data-parallel ranks **force** $k=1$ on every microbatch. This ensures one-loop and deeper gradients are averaged in the same optimizer update:

p_1 = \frac{1}{K} + \left(1 - \frac{1}{K}\right) f, \quad \bar{k} = f + (1-f) \frac{K+1}{2}\tag{3}

The override keeps $p_1 \approx 1/3$, balancing one-loop accuracy against gains at deeper loops. ### 3.3 Loop-Conditioned Modulation Each loop $i$ of $k$ is conditioned on progress $\tau_i = i/k$ and budget $\kappa = k/K$:

c_i = \text{MLP}([\phi(\tau_i); \phi(\kappa)])

$$The conditioning vector modulates each recurrent/output layer via FiLM-style operations:$$

\begin{array}{l} \tilde{x} = (1 + \lambda(c_i)) \odot \text{RMSNorm}(x) + \mu(c_i) \ \hat{x} = \tilde{x} + A(g(c_i) \odot B\tilde{x}) \ y = x + (1 + \omega(c_i)) \odot f(\hat{x}) \end{array}\tag{4}

All conditioning heads are zero-initialized at the switch to mixed-depth training, so the model initially preserves the one-loop checkpoint's function exactly.

3.4 Cost Reduction Techniques

• Delayed mixed-depth training: Train at $k=1$ for the first 75% of tokens, then apply mixed depths. Total training compute relative to baseline:

$$(1-\rho) + \rho \frac{D_{\text{eff}}(\bar{k})}{D_{\text{eff}}(1)}\tag{5}$$

For the 4B configuration, this costs ~1.16× baseline FLOPs vs. 1.62× for mixed-depth from scratch.

• Cross-loop route replay (MoE): Compute expert assignments once on the first loop, reuse on later loops. Cuts mixed-depth update time by 11.7%.

• Last-loop KV caching: Keep a single attention KV cache that each loop overwrites, holding memory at the one-loop level.

Empirical Validation / Results

4.2 Main Results (Pretraining)

Models: GDP-Hybrid-4B (dense, 4B params, 1T tokens) and GDP-Hybrid-30B-A3B (MoE, 30B total/3B active, 300B tokens).

Diminishing returns: Two and three loops capture 79% and 96% of the gain (4B), and 87% and 91% (30B-A3B).

4.3 Cross-Loop Route Replay Results

5.2 Koi-4B (Post-Trained)

• AA-Index v4.1 estimates: 18 at $k=1$, 26 at $k=2$, 28 at $k=3$, 29 at $k=4$.
• At $k=1$, matches Qwen3.5-4B; at $k=4$, exceeds Gemma4-26B-A4B (6.5× parameters) and trails only Qwen3.6-35B-A3B (9× parameters).
• Higher loop counts produce shorter responses (32.7% fewer output tokens at $k=4$) while scoring higher—latent depth substitutes for explicit chain-of-thought tokens.

5.3 Last-Loop KV Cache Results

At $k=4$, score reductions vs. per-loop caching range from 0–3% across benchmarks, with attention KV memory held at 1× (vs. 2.5× for per-loop caching, saving 4.5 GiB per 128K-token request).

6. Ablations (GPT-0.5B, 300B tokens)

• State update rules: Adaptive delta is the only rule matching baseline at $k=1$ and achieves lowest loss at all loop counts. Replace and residual rules perform worst.
• Sampling schemes: Independent-rank sampling (each rank draws its own $k$) gives lowest loss but 1.36× step time; one-loop overrides recover most of the benefit at synchronized step time.
• One-loop overrides: Without overrides ($f=0$), one-loop loss is +0.0225 above baseline; $f=0.15$ ($p_1=0.32$) is the smallest fraction that falls below baseline.
• Delayed mixed-depth: Starting at 75% of training tokens retains 84% of five-loop improvement at 1.20× training compute (vs. 1.79× from scratch).
• Loop conditioning: Leaves one-loop loss nearly unchanged (1.6748 → 1.6739), lowers loss by 0.009 at $k=2$ and 0.022 at $k=5$.

Theoretical and Practical Implications

• Architecture design: The gated adaptive-delta update provides a principled way to make recurrent depth learnable without explicit stability constraints (unlike Parcae's linearized stability conditions or STARS' spectral radius regularization). The learned step size starts small (~0.35) and is trained end-to-end.
• Training dynamics: Within-step mixing of loop counts is crucial—synchronized sampling (all ranks share a depth per step) fails to preserve the one-loop endpoint even at the same marginal distribution. This suggests that gradient averaging across depths within a single update is essential for balanced optimization.
• Practical deployment: ELR enables a single checkpoint to serve multiple latency/quality tiers, eliminating the need for separate models. The late acquisition of elastic depth (final quarter of pretraining) makes this practical for large-scale training runs.
• MoE efficiency: Cross-loop route replay demonstrates that routing decisions can be reused across loops with minimal quality loss, offering a template for other recurrent MoE designs.
• Memory efficiency: Last-loop KV caching decouples memory footprint from inference depth, making deeper inference feasible on memory-constrained hardware.

Conclusion

ELR demonstrates that looped language models can match non-recurrent baselines at one loop while gaining accuracy at additional loops, eliminating the one-loop tax. Three key choices enable this: the gated adaptive-delta update, within-step loop-count mixing, and loop-conditioned modulation. The recipe is cost-effective (delayed mixed-depth training at ~1.16× baseline compute), scalable to MoE models, and produces Koi-4B—a single checkpoint spanning 18 to 29 on the AA-Index purely by loop count.

Open questions:

• Gains diminish after the second loop; reweighting the loop-count distribution may recover them.
• Conditioning closes most (but not all) of the gap to fixed-depth specialists at endpoints.
• Combining ELR with per-request or per-token adaptive depth (complicated by budget conditioning) and post-training the MoE variant remain future work.