Training Methods Become the New Battleground for Recurrent Depth: The Truncation Debate and Progress Alignment
Highlights of This Issue
The most prominent signal this issue is that training methods for recurrent models are moving from empirical recipes toward verifiable science. The scaling study from the previous issue focused on the conversion between recurrence count and parameter count; this issue places the two degrees of freedom in the training process—forward recurrence count and backpropagation truncation depth—into the same scaling law. Truncate or Not? uses nearly 600 training runs to fit the first unified scaling law that simultaneously models forward and backward recurrence counts, providing an answer to a question previously conflated: when both model size and compute are freely selectable, full BPTT outperforms truncation, and two fully trained recurrences can approach the non-recurrent compute-optimal model with approximately 27–32% fewer parameters. This directly responds to the previous issue's debate about whether scaling parameters beats scaling recurrence—the conclusion is that the competitiveness of recurrence depends on the training contract, not expressiveness.
The second theme is that depth extrapolation shifts from running more iterations to giving each iteration explicit progress. PAR proposes progress-aligned recurrence: aligning the prediction at the k-th iteration with the state after the first k operations, combined with RMS norm anchoring to keep the step mapping stable, achieving 16x length extrapolation on swap-S5 (compared to 0.8% for explicit CoT). This provides the clearest training objective design to date for recurrence count as a test-time compute scaling axis—depth extrapolation is not achieved through more iterations but by binding each iteration to a verifiable intermediate state.
The third theme is the scaling recipe for recurrent MoE. Loop the Loopies! proposes a layer-loop pattern and a compute-matched scaling recipe, transforming recurrence from a parameter-saving device into a compute-matched scaling mechanism, validated at 0.1B–20B scale; LOOM diagnoses two obstacles preventing recurrent MoE from exceeding two rounds—variance growth from residual accumulation and expert selection collapse—and provides a recipe for stable scaling to 9–12 rounds. Together, they indicate that the bottleneck for recurrent MoE lies in training stability, not expressiveness.
The fourth theme is that adaptive inference moves from FLOP savings to system throughput. Continuous Depth Batching is the first to bring depth-adaptive inference from theoretical simulation to an end-to-end system, achieving nearly 99% conversion of FLOP savings into actual throughput gains; Where to Spend Test-Time Compute uses controlled experiments to prove that in recursive reasoning models, the answer state carries failures, and resampling y recovers more failures than perturbing the reasoning state z—providing concrete guidance for test-time compute allocation.
Community and Updates
- The boundaries of recurrent/recursive reasoning are directly tested: The illusion of thinking finds on frontier LRMs a non-monotonic scaling limit where reasoning effort first rises then falls with compositional complexity, and accuracy collapses beyond a threshold—an important counterexample to the assumption that more depth is always better, creating tension with PAR's optimistic conclusions this issue.
- The attribution of recurrence benefits is systematized: Tiny in Parameters, Not in Compute proposes a meta-analysis framework for decomposing recurrence gains by task—attributing gains to depth on Sudoku and to regularization on ARC—and explicitly lists the matched parameter/compute controls needed for attribution.
- A new survey Looped and Recurrent-Depth Transformers identifies training stability (rather than expressiveness) as the binding constraint for recurrent-depth architectures and points out the lack of compute-matched baselines and unified inference-time scaling laws—corroborating the findings of Truncate or Not and LOOM this issue.
- An equal-compute comparison at language scale appears: Scaling up Test-Time Compute with Latent Reasoning reports a direct equal-compute comparison between implicit test-time scaling of a 3.5B recurrent-depth model and explicit CoT, involving adaptive computation and KV sharing; only the abstract is currently visible, with specific scaling curves pending full text confirmation.
- Skeptical voices remain: Latent Reasoning and Recurrent Depth reminds that running a shared block 20 times costs about 20x compute without parameter growth, calling for compute-matched scans and full economic evaluation—consistent with the field's usual caution standards.
Open Questions
- Truncate or Not prefers full BPTT when both recurrence counts are free, while PAR demonstrates 16x depth extrapolation with progress-aligned objectives—does the truncation conclusion still hold after objective alignment? The interaction between training objectives and truncation strategies remains systematically unstudied.
- The finding that "readable answers do not imply reusable states" (Learning to Compute Is Not Learning to Retain or Reuse) directly challenges the implicit assumption in early-exit/adaptive computation that "readable answers justify stopping"—how does this reconcile with CDB's system-level early exit this issue?
- Loopie's layer-loop and Ouro/Huginn's model-loop differ in training behavior, but their differences in depth extrapolation behavior have not been characterized—does layer-loop change the mechanism where the answer token is determined at the first iteration?
- The non-monotonic limit of "reasoning effort rising then falling with complexity" on frontier LRMs—does it also appear in recurrent language models? If so, is it driven by the residual accumulation/state drift mechanisms discussed this issue?
Papers in this issue
Training looped language models to align each iteration with the sequential task state and anchoring norm scale enables 16x length generalization in state tracking.
Editor's noteDepth extrapolation shifts from running more iterations to giving each iteration explicit progress: the progress-aligned objective makes the k-th iteration predict the state after the first k operations, combined with RMS norm anchoring to keep the step mapping stable, achieving 16x length extrapolation on swap-S5 (compared to 0.8% for explicit CoT), with ablations proving both are indispensable. It also provides label-free prefix self-distillation and a learned halting criterion. The limitation is that it applies to sequential state-tracking tasks with clear intermediate states, but it is the clearest depth extrapolation training objective design to date in this direction.
Full backpropagation through time is optimal for looped transformers when both forward and backward loop counts are free, despite truncation helping when forward loops are fixed.
Editor's noteThe first unified scaling law that simultaneously models forward and backward recurrence counts, using nearly 600 training runs to answer the truncation debate: when both model size and compute are freely selectable, full BPTT outperforms truncation, and two fully trained recurrences can approach the non-recurrent compute-optimal model with approximately 27–32% fewer parameters. Relative to Iso-Depth and Parcae, which fix the backward recurrence or tie it to T, this is a clear increment with direct guidance for training recipe design.
Loopie's compute-matched looped MoE recipe achieves gold medals on IMO and IPhO 2025 using only 3.5T tokens.
Editor's noteProposes the layer-loop pattern—each layer iterates locally before moving to the next—and a compute-matched scaling recipe (halving storage layers, doubling micro-batches, reinvesting memory gains into width/depth), transforming recurrence from a parameter-saving device into a compute-matched scaling mechanism, validated at 0.1B–20B scale with released 20B-A2B and 6B-A0.6B models. Relative to Ouro/Huginn's model-loop, this is a new architectural degree of freedom, directly responding to the previous issue's debate about scaling parameters versus scaling recurrence.
Continuous depth batching realizes up to 99% of theoretical depth-adaptive speedup for looped language models, with fully looped architectures yielding the largest gains.
Editor's noteThe first to bring depth-adaptive inference from theoretical simulation to an end-to-end system, using continuous depth batching (CDB) to solve the problem of tokens with different recurrence depths not being batchable together, achieving nearly 99% conversion of FLOP savings into actual throughput gains, with a lookahead gate and asynchronous scheduling to reduce GPU idle time. Relative to prior theoretical simulations of CDB, this is a concrete increment, important for understanding the deployment implications of depth as a test-time scaling axis.
Gated Answer Resampling, a tuning-free test-time method that resamples only the decoded answer state, boosts TRM accuracy from 85.9% to 99.4% on Sudoku and 68.8% to 85.6% on PPBench at minimal compute cost.
Editor's noteControlled comparison of where to inject randomness in recursive reasoning models—answer state y versus reasoning state z, continue versus restart—finds that the answer state carries failures, and resampling y recovers more failures than perturbing z (including PTRM's stepwise noise), with the gap widening as the compute budget grows. The GAR method is hyperparameter-free and compute-efficient, surpassing PTRM at minimal cost on frozen checkpoints. Relative to the input-conditioned search test-time scaling axis from issue 2, this is a complementary concrete increment.
Controlled ablations isolate the sandwich envelope as the primary driver of looped language model gains, with input injection helping context tasks but hurting math.
Editor's noteFirst to decompose the architectural differences between the Ouro and Huginn recurrent families into three separable axes—iteration envelope, input injection, and hidden state organization—under matched parameter counts, logical depth, and token budgets, extending to MoE and showing that Huginn-style design advantages transfer and that per-iteration expert routing is causally important. Relative to the cross-family mechanism comparison in issue 2, the increment lies in isolating individual design choices rather than comparing full recipes.
LOOM enables looped mixture-of-experts models to scale to 9-12 loops by combining residual scaling, embedding re-injection, and per-loop routers, achieving substantial perplexity and accuracy gains without increasing parameters.
Editor's noteDiagnoses two obstacles preventing recurrent MoE from exceeding two rounds—variance growth from residual accumulation (curse of depth) and expert selection collapse from shared routing—and proposes a recipe with residual scaling, embedding re-injection, per-recurrence routing, and Looping Residual, stably scaling to 9–12 rounds on 100M–1.7B. Relative to the recurrent MoE scaling law in issue 1 that only models without providing stabilization recipes, this is a concrete increment.
Looped transformers that compute answers exactly can fail to retain or reuse them, with training supervision and exit variability—not gradient truncation—determining retention and state reusability.
Editor's noteFirst to strictly distinguish the properties of computing, retaining, and reusing in recurrent transformers, using precise symbolic targets and counterfactual interventions to prove: even if the model computes correctly at training depth, answers are lost in subsequent recurrences; even if answers are readable, their hidden states cannot be frozen and reused within several recurrences after the answer appears. This directly challenges the implicit assumption in early-exit/adaptive computation that readable answers justify stopping, providing a new mechanistic perspective on depth extrapolation failures.
Test-time depth in recurrent networks only helps when the trained operator settles toward a fixed point, a regime controllable via a single fixed-point training objective.
Editor's noteIntroduces measurable finite-time dynamical states (settling/marginal/drift) as unified predictors of test-time depth safety and transition capability for recurrent-depth models, and proves that terminal fixed-point objectives can causally switch dynamical states. Relative to the convergence time scaling in issue 1, it adds a cross-architecture failure taxonomy and Lyapunov spectrum anisotropy signatures, extended to Huginn-3.5B validation.
RecurTrace adds loop-time attention memory and oracle-distilled halting to frozen LLMs, improving MathQA accuracy by 2.2 points at matched compute while scaling gains with model size.
Editor's noteIntroduces per-token memory attention along the recurrence time axis (Loop Memory Attention), allowing weight-shared recurrent blocks to read their own early recurrence states, paired with an oracle-distilled halting head based on loss improvement and sequence-level adaptive depth. Relative to input-conditioned search in issue 2 and loss-gain labels in issue 1, it positions intra-recurrence memory as a key mechanism for adaptive depth, with quantitative comparisons against ACT, PonderNet, CALM, LoopUS, and TaH, and scaling curves from 0.6B to 8B.
