Loop Count Is Not a Compute Dial: A Contract View of Depth Reusability
Highlights of This Issue
The most central signal this issue is that "whether loop count is a reliable axis for scaling compute" is further tightened. Issue 5, There Is No Free Loop, already provided scale-related negative results. This issue, Agreement rather than normalisation keeps depth reusable in looped language models, shifts the question from "training recipe" to "what determines depth reusability": by precisely rearranging loop-boundary normalization on released weights (without changing the readout path, r=1 identity check), it proves that what breaks depth reusability is not state norm growth but the mismatch between loop topology and the readout head. From-scratch training shows that training without boundary normalization extrapolates better beyond training depth, and readout refitting recovers only about one-third of the loss; full recovery requires retraining together with the last block layer. This is highly consistent with the community survey A Loop Count Is Not a Compute Dial, which systematically distinguishes four "test-time loop" operations—truncation, continuation, easy-to-hard iteration, and refinement—each requiring different control baselines, and points out that continuation tends to flatten or even reverse beyond training depth on general benchmarks, while only easy-to-hard iteration converts on algorithmic tasks. "Loop count" itself is not a dial; the key lies in the training contract and task structure.
The second theme is rigorous theory for depth extrapolation. Learning to Extrapolate in Depth with Looped Transformers for the first time characterizes the finite extrapolation window of looped transformers trained at fixed depth: training depth K converges to an effective parameter that trades off propagation and residual error, giving a certified window for test depth—too few loops under-propagate, too many create spurious connections, and the certified depth gain grows exponentially with K−H. This complements the mechanistic study in Issue 4, Beyond the Training Horizon, and the PAR training objective in Issue 3, turning the "help-then-hurt" overthinking phenomenon into a provable window boundary.
The third theme is the engineering of recursive reasoning and adaptive depth. What Limits Recursive Reasoning Models for the first time isolates architectural and optimization factors of HRM/TRM/URM in a unified pipeline, finding that optimization (gradient paths, physical batching, state stabilization) matters more than explicit hierarchical structure, improving a 13.6M model on arithmetic OOD from 36.2% to 71.16%. On the serving side, Not All Loops Cost the Same decouples depth allocation by prefill and decode phases, implementing training-free serving optimization in vLLM; Depth as a Candidate Source treats loop depth as a complementary source of candidate solutions, replacing single-depth selection with budget allocation, proving that success sets at different depths are non-nested. On the architecture side, Rectified Looped Transformer uses a streaming residual-time interpolation schedule (η_k=1/(K−k)) instead of direct feedback, reducing the depth dependence of perturbation amplification from exponential to polynomial; Library of Layers makes layer order itself a learnable runtime decision.
Community and Dynamics
- A Loop Count Is Not a Compute Dial is the most systematic skeptical survey in this direction to date: it decomposes "test-time looping" into four operations—truncation, continuation, easy-to-hard iteration, and refinement—each requiring different control baselines, and notes that continuation flattens or even reverses beyond training depth on general benchmarks (Ouro-1.4B drops 26.3 points on AIME with four extra steps), while only easy-to-hard iteration converts on algorithmic tasks—providing necessary controls for evaluating any claim that "more loops = more compute."
- Small-scale negative results continue: Spidory reports that one loop matches three loops (1/3 compute), with extra loops only recovering the single-loop tax—contrary to the intuition that "more loops are better," but lacking scaling curves and non-loop baselines, so it should be treated as anecdotal evidence.
- Community skepticism about looped language models is also fermenting: Recurrent Looped Transformer: What Survives After the Hype cites controlled evaluations claiming that looping is not cost-effective for language models, costing about 20x GPU time, with only tie-level gains on small algorithmic state-tracking tasks—specific conclusions should defer to the original papers.
Open Questions
- Agreement rather than normalisation proves that depth reusability depends on the "contract" between looping and readout rather than normalization itself, while A Loop Count Is Not a Compute Dial notes that continuation flattens or reverses beyond training depth—is "contract matching" the missing variable determining whether continuation can convert? How does this unify with the scale-related negative results in Issue 5, There Is No Free Loop?
- Learning to Extrapolate in Depth provides a certified upper bound that "too many loops create spurious connections"—does this mechanism hold at language model scale? Can it explain the robustness differences of Loopback to extra accesses in Issue 5, and provide a more reliable criterion for adaptive stopping than "stop when the answer is readable"?
- Depth as a Candidate Source proves that success sets at different depths are non-nested—if single-depth serving is inherently limited, how do combined allocation and the phase-level depth allocation of Not All Loops Cost the Same synergize? Does budget allocation conflict with "contract matching"?
- What Limits Recursive Reasoning Models proves on 13.6M that optimization matters more than architecture—does this conclusion hold for larger recursive reasoners? Does optimization dominance imply that "loop count" as an expansion axis is equally limited in recursive reasoning?
Papers in this issue
Loop-boundary normalization in looped language models is a readout contract, not a stability mechanism; removing it breaks inference while state scale stays intact, and training without it extrapolates better.
Editor's noteFor the first time, on released weights, by precisely rearranging loop-boundary normalization (without changing the readout path, r=1 identity check), it proves that what breaks depth reusability is not state norm growth but the mismatch between loop topology and the readout head; and from-scratch training shows that training without boundary normalization extrapolates better beyond training depth, and readout refitting recovers only about one-third of the loss; full recovery requires retraining together with the last block layer. Relative to the negative results in Issue 5, There Is No Free Loop, and the recipe in Issue 4, Closing the Loop, this paper shifts focus from training recipe to the contract of normalization placement and readout calibration, and provides a methodological correction for comparing looped vs. stacked depth under equal overshoot (K/r0).
Recursive reasoning performance is driven primarily by optimization choices—gradient horizon, physical batch size, and stabilized state updates—rather than architectural hierarchy, yielding a 13.6M-parameter model that doubles OOD accuracy on arithmetic.
Editor's noteThe first to isolate architectural and optimization factors of HRM/TRM/URM in a unified pipeline, systematically revealing that optimization (gradient paths, physical batching, state stabilization) matters more than explicit hierarchical structure, and giving a large improvement on a 13.6M model on arithmetic OOD (71.16% vs 36.2%). Relative to the truncation scaling study in Issue 3, Truncate or Not, the increment is simultaneously comparing gradient path placement and physical batching vs. gradient accumulation, and emphasizing state stabilization recipes. Limitations: very small model scale, no strict comparison with equal-compute non-loop models.
Looped transformers trained at depth K extrapolate to harder graph connectivity instances at test depths up to an exponential window, beyond which overthinking causes failures.
Editor's noteFor the first time, provides a theory of the finite extrapolation window for looped transformers trained at fixed depth: training depth K converges to an effective parameter that trades off propagation and residual error; within the certified window, too few loops under-propagate, too many create spurious connections, and the certified depth gain grows exponentially with K−H. Relative to the mechanistic study in Issue 4, Beyond the Training Horizon, and the PAR training objective in Issue 3, this turns "help-then-hurt" into a provable window boundary with rigorous training dynamics. Limitations: restricted to linear attention, no softmax, no residual, graph-connectivity toy tasks.
LoopDuet shows that asymmetric loop depths for prefilling versus decoding—with prefilling shallower—boost throughput by up to 19% and quality by 6.04 EM points without retraining.
Editor's noteFor the first time, decouples depth allocation of looped transformers by serving phase (prefill vs decode), proposing joint selection of (S,P) budgets, compact batch execution, and a queue age controller, implementing training-free serving optimization in vLLM. Relative to Issue 3's Continuous Depth Batching (grouping by token depth) and Issue 2's Enabling Dynamic Computation (early-exit KV aliasing), the increment is phase-level depth allocation and load adaptation. Main contribution is in the serving system layer; increment for training methods or mechanism analysis is limited.
Deeper recurrent computation can lose correct answers found at shallower depths, so allocating inference budgets across multiple depths outperforms any single stopping depth.
Editor's noteFor the first time, treats loop depth as a complementary source of candidate solutions rather than a single stopping point, proposing a Depth-based Portfolio Allocation (DPA) framework that replaces single-depth selection with budget allocation, and proves that success sets at different depths are non-nested. Relative to Issue 5's Best-of-Depth candidate set training and Issue 4's Adaptive-Depth stopping criteria, the increment is treating depth diversity as a resource allocation problem for test-time compute rather than a training objective or stopping rule. Limitations: Huginn backbone, limited scale on four benchmarks.
Rectified Looped Transformer replaces direct feedback with a remaining-time interpolation schedule, achieving bounded state norms, polynomial perturbation growth, and consistent perplexity gains without extra parameters.
Editor's noteIntroduces a streaming residual-time interpolation schedule (η_k=1/(K−k)) instead of direct feedback, reducing the depth dependence of state norm and perturbation amplification from exponential to polynomial, and provides theoretical guarantees of depth-consistent state bounds. Relative to Issue 1's One Model to Loop Them All's learned gating and Fixed-Point Reasoners' damped feedback, the increment is a learning-free analytic schedule and polynomial amplification bounds, with empirical evidence on language modeling perplexity, inference depth robustness, and contrastive decoding.
Library of Layers lets a router dynamically reorder transformer layers per input, matching dense, looped, and sparse MoE baselines while enabling interpretable, controllable routing paths.
Editor's noteFor the first time, makes layer order itself a learnable runtime decision: a globally shared library of transformer blocks, with a router dynamically selecting the next layer per window, unifying a continuous spectrum from parameter reuse (small library) to sparsity (large library). Relative to Issue 3's Towards Looped Models Done Right's fixed loop order and Issue 1's One Model to Loop Them All's fixed loop block, the increment is dynamic layer-level ordering and window-level routing. Limitations: main evidence on synthetic tasks and 1B-scale math, and comparison with looped models only matches parameters, not strictly equal compute.
