# Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets

> A pretrained instruction-tuned model can be surgically converted into a recurrent-depth architecture that learns a persistent iterative procedure, outperforming dense baselines on depth-rewarding reasoning tasks.

- **Source:** [arXiv](https://arxiv.org/abs/2608.11233)
- **Published:** 2026-10-12
- **Permalink:** https://picx.dev/p/SKmbyJ
- **Whiteboard:** https://picx.dev/p/SKmbyJ/image

## Summary

## Summary

- This paper demonstrates that a pretrained, instruction-tuned dense language model (Qwen2.5-0.5B-Instruct) can be surgically converted into a recurrent-depth architecture that learns a persistent iterative latent transition, achieving superior performance on depth-rewarding reasoning tasks compared to dense baselines.
- The retrofit installs at two parameter budgets: a full-block adaptation (180.6M forward-active parameters) and a rank-16 LoRA adapter (6.01M forward-active parameters with frozen base weights), with the adapter matching the full-block's overall performance (83.8% vs 84.0%).
- The installed mechanism extrapolates to roughly 1.5 times its supervised depth (frontier/support ratio of 1.44–1.50), demonstrating a reusable procedure rather than terminal-answer lookup.
- The recurrent system outperforms a same-size scratchpad model overall (84% vs 72%), retains accuracy far better beyond depth 10 (53% vs 2.5%), and answers 7.6× faster with fewer output tokens.
- A critical boundary was identified: neither budget could acquire a second, inverse operation while preserving the installed mechanism, revealing a catastrophic-interference boundary.

## Introduction and Theoretical Foundation

The paper addresses a fundamental question: can a pretrained, instruction-tuned dense language model be surgically converted into a recurrent-depth system that recovers a stable iterative mechanism and outperforms dense recipes on tasks rewarding depth—without damaging general capability?

**Motivation**: Transformers typically obtain additional computation by adding layers (design) or tokens (generation). Recurrent-depth models offer a third axis: repeatedly applying a shared transformation to hidden states, increasing effective depth without increasing distinct block parameters. This creates a natural place for adaptive computation and latent iterative computation.

The work is motivated by small from-scratch recursive reasoners (HRM, TRM) showing that iterated latent computation enables networks with millions of parameters to perform deterministic reasoning tasks that defeat much larger models.

**Key theoretical challenges**:
1. **Distribution shift**: The repeated block receives states from a distribution different from pretraining, driving activations off-manifold.
2. **Re-entry requirement**: Input context must be re-injected on later iterations.
3. **Gradient path**: The pathway combining context with carried state must demonstrably receive training signal.
4. **Correct evaluation**: Predictions must be read at the correct point in the loop.
5. **Consolidation erasure**: A consolidated mechanism can be erased by additional training.

## Methodology

### Architecture: The Split-Bridge Retrofit

The pretrained transformer (24 decoder layers, hidden width 896) is split at indices (a, b) = (6, 18):

$$ \text{tokens} \rightarrow \text{Prelude[0:a]} \rightarrow \text{Recurrent Block[a:b]} \rightarrow \text{Coda[b:L]} \rightarrow \text{LM head} $$

The recurrent state update is:

$$ u_t = \text{Reentry}(p, h_t), \quad h_{t+1} = \text{RecurrentBlock}(u_t) $$

where $p$ is the Prelude output and $h_t$ is the block output at iteration $t$. The bridge learns separate projections $W_p, W_s \in \mathbb{R}^{896 \times 896}$ with an identity-biased gate:

```python
translated = self.prelude_proj(prelude) + self.state_proj(work)
gate = self.bridge_gate
return (work + gate * (translated - work))
```

### Training Protocol

**Exact intermediate-state supervision**: Each loop is graded against the correct intermediate state (e.g., after loop 1 the state should encode "Max", after loop 2 "Eve", etc.). Staged training first installs the transition with dense intermediate labels, then removes the scaffold, grading only final answers to test persistence.

**Frozen same-reader evaluation**: A single validated reader decodes all iterations, calibrated against a simulated perfect executor. The primary synthetic family has 128 rows at each depth 1–14 (1,792 rows total).

### Two Training Budgets

| Budget | Forward-active params | Steps to gate | D1–D4 at finish |
|--------|----------------------|---------------|-----------------|
| Full-block | 180.6M | 6,000 | 64, 63, 63, 59 (of 64) |
| Adapter (rank 16) | 6.01M | 4,000 | 64, 64, 60, 53 (of 64) |

## Empirical Validation / Results

### 1. Mechanism Persistence After Scaffold Removal

After outcome-only annealing, the stepwise mechanism persisted:
- **Active-label diagonal (depths 1–4)**: 625/640 = 97.7% correct intermediates
- **Above-diagonal states that continued iterating**: 357/384 = 93.0%
- **Above-diagonal states that held (parked)**: 1/384 = 0.3%

This rules out terminal-answer lookup: the model computes one task step per loop and keeps stepping past the target depth.

### 2. Support-Depth Scaling

The threshold-crossing frontier (where per-step accuracy crosses $\sqrt[4]{0.25} \approx 0.71$) tracked support at a near-constant ratio:

| Support | Alphabet | Frontier | Frontier/Support |
|---------|----------|----------|------------------|
| 4 | N16 | 5.75 | 1.44 |
| 6 | N16 | 9.01 | 1.50 |
| 8 (consolidated) | N16 | 11.61 | 1.45 |
| 12 | N24 | 17.93 | 1.49 |

The N24 support-12 checkpoint showed a ceiling: 70.3% accuracy at depth 18, falling to 46.1% at depth 20 and 10.9% at depth 22.

### 3. Registered Comparison Against Dense Baselines

**Five Phase A systems** on the frozen 1,792-row family:

| Arm | System | Correct/1,792 | Accuracy | Tail (depths 11–14) |
|-----|--------|---------------|----------|---------------------|
| A | Recurrent 0.5B (full block) | 1,506 | 84.04% | 53.13% |
| E | Recurrent 0.5B (adapter, 6.0M) | 1,501 | 83.76% | 48.24% |
| B | Dense 0.5B direct | 496 | 27.68% | 11.72% |
| C | Dense 0.5B scratchpad | 1,292 | 72.10% | 2.54% |
| D | Dense 1.5B direct | 656 | 36.61% | 11.33% |

The scratchpad matched the recurrent arms inside its learned horizon (through depth 10) but collapsed beyond it (2.5% tail accuracy vs. 53.1% for the recurrent system).

### 4. Wall-Clock Latency

At depth 14, the full-block recurrent arm answered in 301.8 ms versus 2,283.5 ms for the scratchpad—**7.6× faster** with 1 median output token versus 96.

### 5. General-Capability Preservation

Using a preregistered battery (ARC-Easy and ARC-Challenge, full evaluation), all promoted checkpoints were non-inferior at loop 1 within the 3-point margin. The single nominal dip did not survive Bonferroni correction (corrected p = 0.308).

### 6. Optimizer Interaction Finding

Gradient-slice scaling within a combined projection **fails silently** under both Adam-family and Muon optimizers. Under AdamW, per-element moment normalization cancels a constant gradient scale exactly; under Muon, whole-matrix orthogonalization discards global scale. The working solution: a separate tensor in a dedicated optimizer parameter group with a true 10× learning rate, verified at the realized-update level.

### 7. Adapter-Budget Depth Crossover

The 6M adapter (3.3% of full-block budget) was **better on trained support** (1,021/1,024 vs 1,005/1,024, p = 0.000855) and through depth 11, while the full-block won at depths 12–14. The interpolated crossover falls at depth 11.54.

### 8. Verbal Transfer

With verbal fine-tuning, the full-block variant reached 79–86% on controlled verbal renderings. Zero-shot transfer was minimal at both budgets (16–20%), but adapter verbal training begun from the installed mechanism outpaced matched fresh training by **18.6 percentage points**, including on a held-out surface.

### 9. Acquisition-Retention Boundary

The inverse operation was learnable in isolation (63/64) and with rehearsal (64/64), but no continuation acquired it while preserving the installed mechanism and general capability at either budget. The adapter retention probe hard-stopped at step 100 with forward mechanism collapse (minimum stratum 0.09375 vs 0.93 floor).

## Theoretical and Practical Implications

**What the model learned**: The evidence collectively rules out final-label lookup. The model predicts registered intermediate states, continues transitions beyond target states, retains the operator after intermediate losses are removed, and separates from dense scratchpads most sharply beyond the scratchpad's horizon—demonstrating that reusable procedures, not just retrievable content, can be trained into pretrained models.

**Why this matters**: A recurring concern is that adaptation installs retrievable content more readily than reusable procedure. This study is a constructive counterpoint: an architecture (one block reused over depth) and a training regimen (per-step supervision followed by scaffold removal) that together instill a procedure applied step-by-step past trained depth and across task surfaces.

**The optimizer interaction finding** has broad implications: any parameter needing its own learning rate in a retrofit must be a separate tensor in its own optimizer group. Scaling a slice of a shared matrix's gradient fails silently—invisible in every standard training signal.

**The retention boundary** shows converted pretrained models are not empty substrates: new training competes with consolidated transitions and retained language-model behavior. Larger substrates, detachable adapters, routing, or explicit modularity may move this boundary.

## Conclusion

A pretrained Qwen2.5-0.5B model can be converted into an identity-preserving recurrent-depth architecture at two measured parameter budgets. The converted model learns a persistent loop-indexed state transition with validity frontier scaling at 1.44–1.50× its supervised support, outperforms registered dense recipes on the frozen synthetic family, and does so most decisively beyond the serialized scratchpad's learned horizon.

The program measured its limits: general natural reasoning remains unproven, learned depth selection is a registered bounded negative for the current controller, and neither budget could retain a newly acquired inverse operation alongside the consolidated mechanism. The deterministic program is complete; the companion study addresses guided stochastic width on the frozen substrate, asking whether target-conditioned stochastic latent trajectories add exact multi-solution coverage beyond output sampling and additional depth.

---

_Markdown view of https://picx.dev/p/SKmbyJ, served by PicX — AI-generated visual whiteboard summaries of research papers._