The Recurring Compute Ledger: From Savings to Reconciliation
Highlights of This Issue
The main theme of this issue is the centralized audit of whether looping actually saves compute and buys capability. The most direct negative result comes from Do Halting Heads Really Save Compute in Looped Language Models?: halting heads trained with mixed losses predict an average of 3.7 steps under standard exit rules but actually run the full 7.6–7.9 steps, saving almost no computation; batching (running one pass until the slowest token stops) further erases the remaining advantage. The authors also provide a pre-training computable upper bound test for benefits and point out that stepwise loss + strong priors are the combination that makes early stopping truly usable—this sets a reporting standard for any claim of saving compute with halting heads. Complementing this is Late Entry in Looped Language Models: it reverses early exit—new tokens skip the first few passes and directly enter the last K passes over deep cached context, and uses Wake verification to unify greedy exact decoding, margin gating, and lossy late entry into a single engine, providing speedup curves that grow with the number of loops on Ouro-2.6B and Huginn-3.5B.
The second theme is equal-compute training. UpLoop is the first to train looped transformers under strictly equal token and equal training FLOPs budgets: it first uses progressive depth growth (shallow to deep) to save early compute, then later consumes this compute with shared middle-block loops, giving precise FLOP equations to guarantee budget matching, and outperforming dense baselines on language modeling and 22 downstream tasks at 370M–1.3B scale. This directly responds to the core question of whether looping can bring gains under equal compute, and contrasts with the scale-related negative results in Issue 5's There Is No Free Loop—the difference being that looping is placed in the pretraining schedule rather than added post-training.
The third theme is mechanism: what is the loop actually computing? Inside the Loop proves on a permutation traversal task with known intermediate states that looped models trained without intermediate supervision form a "staircase"—each iteration advances one hop, and activation patching proves it is causally used; whereas non-shared models of the same effective depth fail at the same learning rate, redefining looping as iterative program learning rather than depth simulation. Shared Weights, Selected Computations proves that hidden states entering the loop can causally select different computations (one-hop vs. two-hop transitions) executed by the shared frozen block, and traces the control back to attention routing.
Community and Updates
- Independent re-analyses of NVIDIA's looped model Dory1.0-4B are beginning to appear. sysnote's re-analysis points out that looping improves coding/agent scores but knowledge scores remain completely unchanged (AA-Omniscience stays at 10.0% from 1 to 4 loops), and that the average score "exceeding larger models" depends on a single hallucination/refusal metric; the compute cost of four loops is about 2.7x per token.
- HackerNoon's Alice test commentary offers a similar framework: weight-shared looping increases computation but not information, so extra loops cannot fill knowledge gaps, and reports that the third loop actually degrades on a certain causal reasoning benchmark.
- An independent evaluation on LessWrong finds that loop gains concentrate in no-CoT reasoning (NCRI about +18 points) while barely affecting general mini-ECI—gains may be concentrated in specific tasks rather than general capability.
- Safety perspective: LessWrong's depth-conditioned backdoor analysis is the first to mechanistically analyze depth-conditioned backdoors in looped language models—trigger tokens can act as a "run clock" tied to the number of loops, and activation patching localizes depth tracking to hidden states.
Open Questions
- Do Halting Heads Really Save Compute proves that the savings of mixed-loss heads are systematically overestimated, and stepwise loss + strong priors are the only usable combination—does this mean the premises of service-side optimizations in Issue 6's Not All Loops Cost the Same and Issue 3's Continuous Depth Batching (that early exit indeed saves compute) need recalibration? Can the pre-training upper bound test be generalized as a standard audit tool for adaptive computation?
- Late Entry shows that deep cached context can replace the first few loops local to the token—how does this unify with the "loop-readout contract" in Issue 6's Agreement rather than normalisation? Does "skipping the first few passes" equate to changing the matching relationship between loop topology and readout?
- UpLoop gains under equal FLOPs by placing loops in the pretraining schedule, while Issue 5's There Is No Free Loop shows no gain from post-training loops at 4B/9B—does "when looping pays off" depend on whether it appears in pretraining or post-training? Where is the scale upper bound?
- Inside the Loop and Shared Weights, Selected Computations both prove that loops execute "selected different computations" rather than the same algorithmic step—does this support or revise the convergence selection theory in Issue 2's When Does Recurrence Become an Algorithm? Does the premise of loop count as an extension axis (each step equivalent) thereby waver?
Papers in this issue
Halting heads in looped language models save far less compute than reported because mixture-trained heads rarely stop tokens and batching overhead eliminates most remaining gains.
Editor's noteFirst systematic revelation that compute savings reports for halting heads in looped language models are systematically overestimated: mixed-loss heads predict an average of 3.7 steps under standard exit rules but actually run the full 7.6–7.9 steps, and batching erases most of the remaining advantage. Relative to Issue 6's Not All Loops Cost the Same service-stage allocation and Issue 3's Continuous Depth Batching, the increment is providing negative results at both the training loss and reporting metric (expected vs realized vs batched) levels, and proposing a pre-training computable upper bound test. It is a necessary control for any work claiming compute savings via early stopping.
Late entry into deep context caches outperforms early exit in looped language models, yielding up to 8.4x exact and 12.5x lossy decoding speedups.
Editor's noteReverses early exit to propose "late entry"—new tokens skip the first few passes and directly enter the last K passes over deep cached context, proving that deep cached context can replace token-local early loops. Wake verification unifies greedy exact, margin-gated, and lossy late entry into a single engine, providing speedup curves and precision guarantees that grow with loop count (T=8 to 64) on Ouro-2.6B and Huginn-3.5B. Relative to Issue 1's LoopCD and Issue 5's Dory self-speculative decoding, the increment is compressing the critical-path verification itself and unifying lossless/lossy regimes.
UpLoop trains looped Transformers with identical parameters, tokens, and FLOPs to dense baselines, improving reasoning and language modeling across scales via progressive depth growth and shared-weight looping.
Editor's noteFirst to train looped transformers under strictly equal token and equal training FLOPs budgets: progressive depth growth saves early compute, later consumed by shared middle-block loops, with precise FLOP equations guaranteeing budget matching, outperforming dense baselines on language modeling and 22 downstream tasks at 370M–1.3B. Relative to Issue 4's Closing the Loop from-scratch recipe and Issue 3's Loop the Loopies compute-matching scaling, the increment is combining depth growth with looping in the pretraining schedule. Directly responds to the core question of whether looping brings gains under equal compute.
Looped transformers solve sequential algorithmic tasks untied models of equal effective depth cannot, by executing one algorithm step per iteration without intermediate supervision.
Editor's noteFirst to provide evidence of the "staircase" mechanism in looped models on a permutation traversal task with known intermediate states: without intermediate supervision, each iteration advances one hop, and activation patching proves causal use, while non-shared models of the same effective depth fail at the same learning rate. Relative to Issue 2's When Does Recurrence Become an Algorithm convergence selection theory and Issue 2's Monitoring latent reasoning, the increment is using tasks with real intermediate states to directly track progress per iteration and causally verify, redefining looping as iterative program learning rather than depth simulation. Limitation: synthetic tasks and small models.
Looped Transformers perform state-dependent computation, not fixed one-loop-one-step algorithms, and a learned linear layer can steer frozen shared weights via attention routing.
Editor's noteFirst to prove that hidden states entering the loop can causally select different computations (one-hop vs. two-hop transitions) executed by the shared frozen block, and trace control back to attention routing via activation patching. Relative to Issue 2's When Does Recurrence Become an Algorithm convergence selection theory, the increment is providing concrete causal intervention evidence and attention-pattern patching tools, validated on Ouro-2.6B. Limitation: synthetic graph traversal tasks, controller long-range composition accuracy degrades.
Adaptive latent inference that stops reasoning when progress stalls improves accuracy by up to 7.2 points while cutting computation by 29%, as completing fixed latent horizons can degrade performance.
Editor's noteFirst to systematically reveal the phenomenon that fixed-horizon implicit CoT "running the full horizon actually harms reasoning", dividing into frontier advancement/candidate organization/answer revision phases, and proposing to use "remaining constructive progress" as an online signal to adaptively adjust implicit reasoning depth without retraining. Relative to Issue 3's RecurTrace stopping heads and Issue 4's Adaptive-Depth training schedules, the increment is estimating remaining progress from the implicit CoT trajectory itself and allocating computation accordingly, with monotonic allocation theory guarantees. Limitation: arithmetic tasks and small models, no equal-compute non-loop control.
InfuseLoop injects cross-trajectory latent states into parallel looped transformers, eliminating training spikes and enabling quality gains at 8 loops with minimal overhead.
Editor's noteFirst to introduce cross-trajectory hidden state injection in parallel looped transformers: leveraging already-available but unused completed cross-trajectory states in PLT overlapping schedules to eliminate training loss spikes and extend depth to 8 loops. Relative to Issue 4's LoopFold parallel folding, the increment is explicitly using cross-trajectory states with O(Rd) storage and causality proofs, including strict comparisons with vanilla PLT and non-loop baselines. Limitation: 0.6B scale, depth gains still smaller than serial looping.






