Full text not available for this paper

Summary of "Inside the Loop: A Mechanistic Study of Looped Transformers on Permutation Traversal"

Summary (Overview)

  • Key finding: Looped transformers (which reuse a shared block of layers across multiple iterations) solve sequential algorithmic tasks that identically-trained untied transformers of the same effective depth cannot solve, despite having ~8× fewer parameters.

  • Mechanistic discovery: Through linear probing, the authors show that looped models build a "staircase" — each loop iteration advances the computation by approximately one hop of the permutation traversal, with the rate adapting to the iteration budget (T > H, T = H, or T < H).

  • Causal validation: Activation patching experiments confirm the model causally uses the intermediate states it builds, not just represents them linearly — replacing states mid-loop redirects the model's final answer to the donor's walk.

  • Contrast with untied models: Untied 32-layer transformers only solve the task when trained at a lower learning rate (η = 3×10⁻⁴) or with auxiliary supervision on intermediate layers, and even then they expose all hops only in their final layers rather than gradually.

  • Implication: Looped transformers function as iterative program learners — they execute an algorithm across iterations without intermediate supervision, making the iteration budget a design parameter for multi-step reasoning tasks.


Introduction and Theoretical Foundation

Background

Looped transformers (depth-recurrent transformers) reuse one block of LL decoder layers across TT iterations, scaling effective depth without adding parameters or tokens. While the concept dates back to Universal Transformers (Dehghani et al., 2019), recent large-scale successes (Geiping et al., 2025; Zhu et al., 2025) have revived interest.

Key Theoretical Questions

The paper addresses two central questions:

  1. Do looped models outperform untied models at matched effective depth? If weight tying steers training toward an iterative algorithm, looped models should solve tasks that untied models cannot.

  2. Does each loop iteration advance the computation by a fixed number of hops? An iterative algorithm predicts each iteration adds a fixed number of steps; a simulated deep stack need not tie nodes to iterations.

Theoretical Foundation

The work builds on the "physics-of-LMs" lineage (Allen-Zhu, 2025; Ye et al., 2025), using tasks with known ground-truth intermediate states. The theoretical basis includes:

  • Weight tying inductive bias: Sharing weights should favor learning a repeated function (Giannou et al., 2023; Saunshi et al., 2025)
  • Expressivity results: An LL-layer transformer looped TT times nearly matches an LTLT-layer model (Saunshi et al., 2025)
  • Fixed-point dynamics: Weight-tied layers converge to cyclic fixed points (Bai et al., 2019; Blayney et al., 2026)

The key distinction the paper targets: does a looped model run an algorithm (advancing the solution step-by-step) or merely simulate a deeper untied stack with fewer parameters?


Methodology

Task: Directed Permutation Traversal (DEPO)

A random permutation π\pi over N=16N = 16 nodes is sampled uniformly from single NN-cycles. The model receives:

  • The edges of the cycle in shuffled order
  • A query with start node ss and hop count HH
  • The answer is πH(s)\pi^H(s) — the node HH steps along the cycle

Key property: Every intermediate hop hh has a known reference state zh=πh(s)z_h = \pi^h(s) for h∈{1,…,H−1}h \in \{1, \ldots, H-1\}, providing ground truth for probing internal states.

Model Architectures

The two model families compute:

x(t)=fθ(x(t−1)+τt),t=1,…,T(looped)x^{(t)} = f_\theta \left( x^{(t-1)} + \tau_t \right), \quad t = 1, \ldots, T \quad \text{(looped)} h(ℓ)=gϕℓ(h(ℓ−1)),ℓ=1,…,LT(untied)h^{(\ell)} = g_{\phi_\ell} \left( h^{(\ell-1)} \right), \quad \ell = 1, \ldots, LT \quad \text{(untied)}

where:

  • fθf_\theta = LL-layer block with shared parameters θ\theta
  • gϕℓg_{\phi_\ell} = distinct layer ℓ\ell with parameters ϕℓ\phi_\ell
  • τt\tau_t = learnable iteration embedding
  • Deff=LTD_{\text{eff}} = LT = effective depth (layer evaluations per forward pass)

Both families use decoder-only architecture with width d=256d = 256, 8 heads, pre-norm RMSNorm, SwiGLU MLPs, and RoPE. The looped model additionally zero-initializes output projections of attention/MLP sublayers.

Training Protocol

  • 100k steps with AdamW, peak learning rate η=10−3\eta = 10^{-3} (unless stated)
  • 5 seeds per configuration
  • Hop curriculum: H=1H = 1 for first 7k steps, linearly widening to H≤15H \le 15 by step 30k
  • Loss: next-token cross-entropy on every non-pad token, read from final layer only (no intermediate supervision)

Probing Methodology

Linear probes: For every depth ℓ\ell and hop hh, a linear classifier decodes reference state zhz_h from the residual stream at the position just before the answer. Probe selectivity:

Sℓ,h=Accreal(ℓ,h)−Accshuffled(ℓ,h)S_{\ell,h} = \text{Acc}_{\text{real}}(\ell, h) - \text{Acc}_{\text{shuffled}}(\ell, h)

A hop counts as decodable where Sℓ,h≥0.5S_{\ell,h} \geq 0.5; the first such depth is its onset.

Logit lens: Stop the loop after iteration tt and apply the model's own final RMSNorm and unembedding head to see what it would write.

Activation patching: Replace the receiver's states at the readout position after iteration tt with those of a donor prompt (same permutation, different start node), then let remaining iterations finish.


Empirical Validation / Results

1. Looped Models Are Better Learners

Effective depth is necessary but not sufficient. At Deff=32D_{\text{eff}} = 32:

ModelSeeds Solving All 15 Hops
Looped 8×44/5
Looped 4×85/5
Looped 2×164/5
Untied 32-layer0/5
Looped 16×20/5

At learning rate η=10−3\eta = 10^{-3}, no untied model solves the task at any depth from 2 to 32 layers.

Table 1: Looped 4×8 vs. Untied 32-layer

TaskηLooped 4×8Untied 32-layer
N=1610⁻³5/50/5
N=163×10⁻⁴5/54/5
N=1610⁻⁴1/51/5
N=16, untied 16 layers3×10⁻⁴–1/5
N=16, auxiliary loss10⁻³–2/5
N=323×10⁻⁴H⋆ = 6, 0, 1H⋆ = 1, 1, 1

At N=32N = 32 (where all hops require forward steps), the untied model never gets past the first hop, while the looped model (with ⅛ the parameters) solves the first six hops in one seed.

2. Looped Models Build a Staircase

Probing results at H = 4 (Figure 4):

  • Looped 4×8: Successive intermediate nodes become decodable a few layers apart, gradually reaching the answer by layer 16
  • Untied 32-layer (same LR): Does not solve; at most one hop ever becomes decodable
  • Untied 32-layer (lower LR): All hops become decodable only between layers 26–30, in a sudden "flurry"
  • Untied 32-layer (auxiliary loss): Builds a staircase resembling the looped model, but with non-uniform step sizes

Iteration budget adaptation (Figure 5):

  • T > H (2×16): ~1 hop per iteration, finishing by iteration 12 of 16
  • T = H (4×8): Roughly follows the identity line; ~2 iterations for first hop, then 2 hops in one iteration
  • T < H (8×4): Several hops per iteration to reach the answer within 4 iterations

3. Causal Evidence

Logit lens (Figure 6a): The output head writes the node of hop tt after iteration tt, with probability rising from 0.53 (hop 4) to 1.00 (hop 7). The head never puts >0.10 probability on already-passed nodes.

Activation patching (Figure 6b): When the receiver's states are replaced with the donor's after iteration 2, the model finishes the donor's walk and writes the donor's answer. Share of donor answers reaches 1.00 after iteration 2 in two of three seeds, after iteration 6 in the third.


Theoretical and Practical Implications

Theoretical Implications

  1. Looped transformers are iterative program learners: They execute an algorithm across iterations without any supervision of intermediate steps, contradicting the "simulated deeper stack" hypothesis.

  2. Weight tying provides a unique inductive bias: The shared parameters steer training toward repeated-function solutions that untied architectures cannot find under identical training conditions.

  3. Iteration budget as a design parameter: The rate of hops per iteration adapts to the budget (T vs. H), suggesting the model optimizes its internal schedule to fit available iterations.

  4. Distinction from chain-of-thought: Looped models keep intermediate computation in continuous hidden states rather than discrete tokens, offering higher reasoning bandwidth.

Practical Implications

  1. Parameter efficiency: Looped models achieve superior capability with ~8× fewer parameters (3.22M for 4×8 vs. 25.72M for untied 32-layer).

  2. Learning rate robustness: Looped models learn over a wider range of learning rates, extending to faster rates at which untied models fail.

  3. Architecture design guidance: No model looped fewer than three times solved the task, but the budget need not cover every step (8×4 solves with T=4 < H=8).

  4. Potential for early exit: Models with spare iterations could exit once their walk ends (adaptive halting).

Reconciliation with Prior Work

The paper addresses why Lu et al. (2025) found limited evidence of per-iteration chains in Huginn-3.5B:

  • Readout: Logit lens alone is insufficient; linear probes decode every hop in most seeds
  • Training: Huginn samples a new iteration count per step and backpropagates through only the last 8; the staircase adapts to a fixed budget
  • Task properties: DEPO has a correct node at every step and trains every hop length — conditions generic pretraining lacks

Conclusion

The paper answers its two central questions:

  1. Looped models are better learners: Under identical training at matched effective depth, looped models solve DEPO while untied models (with 8× the parameters) do not. Untied models only succeed with a lower learning rate or intermediate-layer supervision.

  2. Looped models build and use a staircase: In most seeds, looped models take ~1 hop per iteration when the budget allows, packing several hops per iteration when it does not. Linear probes decode these states, and activation patching confirms the model causally uses them.

Main takeaway: Looped transformers appear to be iterative program learners that execute an algorithm across their iterations without intermediate supervision, making the iteration budget a design parameter for tasks with many sequential steps.

Future Directions

  • Whether the staircase survives on tasks mixing retrieval with computation, or on natural language
  • Whether the results scale to the multi-billion parameter regime with preludes/codas around the loop
  • Why tied training reaches the staircase while untied stacks need intermediate supervision
  • Whether running a hop-packing model for more iterations than trained extends its walk

Related papers