Loop Depth Delivers on Compute-Equivalent Promises: From Single-Loop Tax to Joint Scaling Laws
Highlights
The most prominent signal this issue is that loop depth is shifting from "positive examples on toy tasks" to "compute-equivalent evidence on real language models." Multiple works answer the same question from different angles—under matched parameters and compute, is looping worth it? ELR directly eliminates the "single-loop tax," making a single-loop looped model match a same-size non-looped baseline, with each additional loop adding accuracy; TaH2 is the first to show that post-trained looped models can surpass non-looped baselines under matched test-time compute. Meanwhile, the other side of the compute-equivalent comparison also appears: Cracking GPT-6 records on a real 3B model that accuracy peaks exactly at the training depth and then collapses with representation collapse (effective rank 131.6→14.2) in depth extrapolation failure, while Does Recurrence Pay? gives a controlled negative example where looping loses to a standard transformer at ~140M parameters and 20x GPU cost. The coexistence of positive and negative evidence indicates that "whether looping is free" depends on training recipes and architectural details, not looping itself.
The second theme is the establishment of scaling laws. This issue features the first joint scaling law modeling loop depth and sparsity (Scaling Laws for Looped Mixture of Experts), using a bounded, sparsity-dependent effective parameter mapping to characterize the diminishing returns of looping and how sparsity boosts those returns; LOOM diagnoses two specific failure modes of deep looped MoE (curse of depth and expert selection collapse) and provides recipes that scale to 9–12 loops. These works turn the number of loops from a "hyperparameter" into a "predictable scaling axis."
The third theme is the engineering of adaptive depth and test-time compute. Fixed-point shaping (Towards Looped Models Done Right, Part II) brings learned depth priors and orthogonal input injection, yielding distilled prefill and rollout state reuse in RL; InfiLoop pushes loop depth to tens of thousands of iterations with loop-native residuals; at the system level, continuous depth batching (Depth-adaptive Inference via Continuous Depth Batching) is the first end-to-end implementation of per-token variable-depth serving. Mechanistic research is also advancing—Dissecting Hierarchical Reasoning Models uses causal interventions to show that linearly decodable probe directions are not causally relevant, sounding an alarm for interpretability claims in this direction.
Community and Dynamics
- The negative example of depth extrapolation is worth reading in full: Cracking GPT-6 measures on Nanbeige4.2-3B-Base that depth does not extrapolate, representation collapses to a low-rank fixed point, and finds that loop 1 is more like intermediate encoding than a draft answer, which ordinary CoT supervision cannot fill; Does Recurrence Pay? gives a controlled negative example for per-token temporal recurrence (carrying state along the sequence direction, not looping along depth)—note that it belongs to the sequence-direction recurrence explicitly excluded by this direction, serving as a warning signal rather than a direct test of depth looping.
- The ANIRA poster at ICML 2026 shows that dynamic compute allocation aligned with task complexity can emerge without difficulty supervision, but explicitly cautions that such emergence does not guarantee algorithmic generalization.
- Controlled experiments in recurrent-depth-ttc find that what drives length extrapolation is the supervision scheme (iterative targets vs. final-loop only) rather than architecture, and width-matched looping often adds no extra capability; Polymorphic Universal Transformer systematically catalogs failure modes from weight sharing, including oversmoothing, rank collapse, and inference degradation.
Open Questions
- Positive and negative compute-equivalent evidence coexist: ELR and TaH2 report looping wins under matched compute, while Cracking GPT-6 reports depth extrapolation collapse. Is it the training recipe (loop count sampling, truncated backprop, depth prior shaping) or architectural details (residual scaling, input injection) that cause the divergence? No work yet explains both sides simultaneously.
- Is depth extrapolation fundamentally limited? Retrofitting Recurrent Depth reports an extrapolation frontier of about 1.5x training support, while InfiLoop claims stability to tens of thousands of iterations—does the difference come from task type (language vs. synthetic puzzles) or mechanism design?
- After eliminating the "single-loop tax," does the benefit of looping continue to grow with the number of loops, or does it exhibit overthinking pathology (flipping to negative, abandoning correct answers) as observed in explicit CoT by When More Thinking Hurts? Where is the boundary of loop depth as a test-time compute axis?
Papers in this issue
A learned, entropy-regularized depth prior and orthogonal input injection improve fixed-point convergence in looped language models, enabling 2x faster RL training and 1.8x faster prefill.
Editor's noteSubstantive progress in fixed-point shaping: learned depth priors (policy-gradient-style feedback + entropy regularization + budget control) adaptively adjust the training depth distribution, and orthogonal input injection (OrthoInj) projects the carried state out of the input direction to suppress amplification/cancellation. Also derives two fixed-point-inspired efficiency shortcuts—distilled prefill and Neumann gradient updates from saved endpoints in RL. Relative to Huginn's fixed priors and Parcae's injection, the learned prior and orthogonal projection are concrete increments, with matched-parameter and FLOP non-looped baselines throughout.
Loop scaling laws unify recurrence and sparsity in MoE models, showing sparsity yields ~3x active-parameter efficiency and recurrence ~2x total-parameter efficiency on reasoning tasks.
Editor's noteThe first joint scaling law modeling loop depth and sparsity, with a bounded, sparsity-dependent effective parameter mapping that captures diminishing returns of looping and how sparsity raises the return asymptote. It contrasts clearly with Parcae and Iso-Depth's unbounded linear/power-law mappings and SMELT's fixed R=2, and validates on held-out loop counts, expert counts, model sizes, and data. Reports loop extrapolation curves up to R=16 and downstream reasoning benchmarks, directly useful for designing compute/memory-constrained looped MoE.
Elastic Latent Recurrence eliminates the one-loop tax in looped language models, matching non-recurrent baselines at one loop while gaining up to +7.25 accuracy points with additional loops.
Editor's noteDirectly eliminates the 'single-loop tax'—a single-loop ELR has the same serving cost as a same-size non-looped model, achieves similar accuracy on the same data, and each additional loop trades more inference compute for more accuracy. Three key choices: gated state updates with learned step sizes, mixing loop counts within a single optimization step, and grafting loop-conditioned modulation onto a trained single-loop checkpoint (zero-initialized heads). Provides matched-baseline comparisons at 4B and 30B-A3B, and releases Koi-4B demonstrating inference-time depth scaling. Limitation: does not explore extrapolation beyond training loop counts.
A pretrained instruction-tuned model can be surgically converted into a recurrent-depth architecture that learns a persistent iterative procedure, outperforming dense baselines on depth-rewarding reasoning tasks.
Editor's noteRetrofits loop depth into pretrained dense language models via split-bridge reentry design and exact intermediate-state supervision, under both full-block and LoRA adapter parameter budgets. Relative to prior retrofitting by McLeish et al., the increments are the identity-preserving single-loop path, split-bridge reentry with separated optimizer groups, and systematic characterization of depth extrapolation frontier (~1.5x support) and catastrophic interference boundaries under both budgets. Includes scratchpad comparisons and compute/latency trade-offs; the negative result for learned depth selection is a valuable boundary finding.
LoopCD is a training-free contrastive decoding method that uses intermediate recurrent states of Looped Transformers as weak references, boosting AIME pass@1 by up to 11.45 points and halving compute at equal accuracy.
Editor's noteTraining-free contrastive decoding method that uses intermediate loop states of looped transformers as weak references to guide final predictions, without auxiliary models or extra training. Since shared layer blocks execute repeatedly, intermediate and final states reside in the same semantic representation space and can be decoded by the shared output head; LoopCD contrasts the final prediction with the first iteration, guiding token selection along the loop refinement direction. Consistently improves across multiple looped model families, supporting reduced depth inference and significant FLOP savings.
TaH2 post-trains looped transformers with token-adaptive depth supervision, improving accuracy-compute scaling by 53% and beating non-looped baselines at matched test-time compute.
Editor's noteOnline lookahead depth supervision—uses measured per-token loss gains on the current backbone to generate cost-sensitive labels, jointly training the backbone and an iterative decision maker. Relative to TaH's offline mismatched labels and Ouro's frozen-backbone gate training, this is a concrete increment, and is the first to show that post-trained looped models can surpass non-looped baselines under matched test-time compute, with gains growing with depth cap. Provides scaling curves and depth extrapolation on AIME, code, and QA, matching the evidence style preferred in this direction; modest gains and increased training complexity are the main reservations.
InfiLoop replaces carry-last state replacement with learned attention over recurrent outputs, enabling exact O(d) memory streaming and 97.9% Sudoku accuracy at 20,000+ test-time iterations.
Editor's noteLoop-native residual connections replace the carry-last rule, using learned content weighting with exponential time decay, supporting exact streaming loops with O(d) memory, directly addressing the problem of 'deeper loops undoing earlier progress.' Relative to TRM and FPRM's carry-last or fixed-point damping, this is a concrete increment. Shows continued improvement beyond 20,000 iterations on Sudoku, Maze, and ARC, with learned step sizes emerging as self-verification behavior; limitation is reliance on synthetic puzzle tasks.
Looped reasoning in a typed decision model adds 13.5 accuracy points over a same-size single-pass model, matching a 4.2B model with just 1.4B parameters.
Editor's noteTurns a pretrained looped language model into a typed decision model—reads option probabilities after each loop, trains each round with a proper scoring rule (cross-entropy + Brier), enabling a single model to serve under any budget of 1–8 loops. Relative to Jev-LCT, which only loops the top layer and early-exits, the increments are full-stack loop pretraining, per-loop readout, and systematic analysis of calibration/depth generalization. Provides controlled comparisons with same-shape and larger single-pass models and loop extrapolation curves; the gain depending on a loopable pretrained backbone (Ouro) is its limitation.
