Full text not available for this paper
Summary of "Inside the Loop: A Mechanistic Study of Looped Transformers on Permutation Traversal"
Summary (Overview)
-
Key finding: Looped transformers (which reuse a shared block of layers across multiple iterations) solve sequential algorithmic tasks that identically-trained untied transformers of the same effective depth cannot solve, despite having ~8× fewer parameters.
-
Mechanistic discovery: Through linear probing, the authors show that looped models build a "staircase" — each loop iteration advances the computation by approximately one hop of the permutation traversal, with the rate adapting to the iteration budget (T > H, T = H, or T < H).
-
Causal validation: Activation patching experiments confirm the model causally uses the intermediate states it builds, not just represents them linearly — replacing states mid-loop redirects the model's final answer to the donor's walk.
-
Contrast with untied models: Untied 32-layer transformers only solve the task when trained at a lower learning rate (η = 3×10⁻⁴) or with auxiliary supervision on intermediate layers, and even then they expose all hops only in their final layers rather than gradually.
-
Implication: Looped transformers function as iterative program learners — they execute an algorithm across iterations without intermediate supervision, making the iteration budget a design parameter for multi-step reasoning tasks.
Introduction and Theoretical Foundation
Background
Looped transformers (depth-recurrent transformers) reuse one block of decoder layers across iterations, scaling effective depth without adding parameters or tokens. While the concept dates back to Universal Transformers (Dehghani et al., 2019), recent large-scale successes (Geiping et al., 2025; Zhu et al., 2025) have revived interest.
Key Theoretical Questions
The paper addresses two central questions:
-
Do looped models outperform untied models at matched effective depth? If weight tying steers training toward an iterative algorithm, looped models should solve tasks that untied models cannot.
-
Does each loop iteration advance the computation by a fixed number of hops? An iterative algorithm predicts each iteration adds a fixed number of steps; a simulated deep stack need not tie nodes to iterations.
Theoretical Foundation
The work builds on the "physics-of-LMs" lineage (Allen-Zhu, 2025; Ye et al., 2025), using tasks with known ground-truth intermediate states. The theoretical basis includes:
- Weight tying inductive bias: Sharing weights should favor learning a repeated function (Giannou et al., 2023; Saunshi et al., 2025)
- Expressivity results: An -layer transformer looped times nearly matches an -layer model (Saunshi et al., 2025)
- Fixed-point dynamics: Weight-tied layers converge to cyclic fixed points (Bai et al., 2019; Blayney et al., 2026)
The key distinction the paper targets: does a looped model run an algorithm (advancing the solution step-by-step) or merely simulate a deeper untied stack with fewer parameters?
Methodology
Task: Directed Permutation Traversal (DEPO)
A random permutation over nodes is sampled uniformly from single -cycles. The model receives:
- The edges of the cycle in shuffled order
- A query with start node and hop count
- The answer is — the node steps along the cycle
Key property: Every intermediate hop has a known reference state for , providing ground truth for probing internal states.
Model Architectures
The two model families compute:
where:
- = -layer block with shared parameters
- = distinct layer with parameters
- = learnable iteration embedding
- = effective depth (layer evaluations per forward pass)
Both families use decoder-only architecture with width , 8 heads, pre-norm RMSNorm, SwiGLU MLPs, and RoPE. The looped model additionally zero-initializes output projections of attention/MLP sublayers.
Training Protocol
- 100k steps with AdamW, peak learning rate (unless stated)
- 5 seeds per configuration
- Hop curriculum: for first 7k steps, linearly widening to by step 30k
- Loss: next-token cross-entropy on every non-pad token, read from final layer only (no intermediate supervision)
Probing Methodology
Linear probes: For every depth and hop , a linear classifier decodes reference state from the residual stream at the position just before the answer. Probe selectivity:
A hop counts as decodable where ; the first such depth is its onset.
Logit lens: Stop the loop after iteration and apply the model's own final RMSNorm and unembedding head to see what it would write.
Activation patching: Replace the receiver's states at the readout position after iteration with those of a donor prompt (same permutation, different start node), then let remaining iterations finish.
Empirical Validation / Results
1. Looped Models Are Better Learners
Effective depth is necessary but not sufficient. At :
| Model | Seeds Solving All 15 Hops |
|---|---|
| Looped 8×4 | 4/5 |
| Looped 4×8 | 5/5 |
| Looped 2×16 | 4/5 |
| Untied 32-layer | 0/5 |
| Looped 16×2 | 0/5 |
At learning rate , no untied model solves the task at any depth from 2 to 32 layers.
Table 1: Looped 4×8 vs. Untied 32-layer
| Task | η | Looped 4×8 | Untied 32-layer |
|---|---|---|---|
| N=16 | 10⁻³ | 5/5 | 0/5 |
| N=16 | 3×10⁻⁴ | 5/5 | 4/5 |
| N=16 | 10⁻⁴ | 1/5 | 1/5 |
| N=16, untied 16 layers | 3×10⁻⁴ | – | 1/5 |
| N=16, auxiliary loss | 10⁻³ | – | 2/5 |
| N=32 | 3×10⁻⁴ | H⋆ = 6, 0, 1 | H⋆ = 1, 1, 1 |
At (where all hops require forward steps), the untied model never gets past the first hop, while the looped model (with ⅛ the parameters) solves the first six hops in one seed.
2. Looped Models Build a Staircase
Probing results at H = 4 (Figure 4):
- Looped 4×8: Successive intermediate nodes become decodable a few layers apart, gradually reaching the answer by layer 16
- Untied 32-layer (same LR): Does not solve; at most one hop ever becomes decodable
- Untied 32-layer (lower LR): All hops become decodable only between layers 26–30, in a sudden "flurry"
- Untied 32-layer (auxiliary loss): Builds a staircase resembling the looped model, but with non-uniform step sizes
Iteration budget adaptation (Figure 5):
- T > H (2×16): ~1 hop per iteration, finishing by iteration 12 of 16
- T = H (4×8): Roughly follows the identity line; ~2 iterations for first hop, then 2 hops in one iteration
- T < H (8×4): Several hops per iteration to reach the answer within 4 iterations
3. Causal Evidence
Logit lens (Figure 6a): The output head writes the node of hop after iteration , with probability rising from 0.53 (hop 4) to 1.00 (hop 7). The head never puts >0.10 probability on already-passed nodes.
Activation patching (Figure 6b): When the receiver's states are replaced with the donor's after iteration 2, the model finishes the donor's walk and writes the donor's answer. Share of donor answers reaches 1.00 after iteration 2 in two of three seeds, after iteration 6 in the third.
Theoretical and Practical Implications
Theoretical Implications
-
Looped transformers are iterative program learners: They execute an algorithm across iterations without any supervision of intermediate steps, contradicting the "simulated deeper stack" hypothesis.
-
Weight tying provides a unique inductive bias: The shared parameters steer training toward repeated-function solutions that untied architectures cannot find under identical training conditions.
-
Iteration budget as a design parameter: The rate of hops per iteration adapts to the budget (T vs. H), suggesting the model optimizes its internal schedule to fit available iterations.
-
Distinction from chain-of-thought: Looped models keep intermediate computation in continuous hidden states rather than discrete tokens, offering higher reasoning bandwidth.
Practical Implications
-
Parameter efficiency: Looped models achieve superior capability with ~8× fewer parameters (3.22M for 4×8 vs. 25.72M for untied 32-layer).
-
Learning rate robustness: Looped models learn over a wider range of learning rates, extending to faster rates at which untied models fail.
-
Architecture design guidance: No model looped fewer than three times solved the task, but the budget need not cover every step (8×4 solves with T=4 < H=8).
-
Potential for early exit: Models with spare iterations could exit once their walk ends (adaptive halting).
Reconciliation with Prior Work
The paper addresses why Lu et al. (2025) found limited evidence of per-iteration chains in Huginn-3.5B:
- Readout: Logit lens alone is insufficient; linear probes decode every hop in most seeds
- Training: Huginn samples a new iteration count per step and backpropagates through only the last 8; the staircase adapts to a fixed budget
- Task properties: DEPO has a correct node at every step and trains every hop length — conditions generic pretraining lacks
Conclusion
The paper answers its two central questions:
-
Looped models are better learners: Under identical training at matched effective depth, looped models solve DEPO while untied models (with 8× the parameters) do not. Untied models only succeed with a lower learning rate or intermediate-layer supervision.
-
Looped models build and use a staircase: In most seeds, looped models take ~1 hop per iteration when the budget allows, packing several hops per iteration when it does not. Linear probes decode these states, and activation patching confirms the model causally uses them.
Main takeaway: Looped transformers appear to be iterative program learners that execute an algorithm across their iterations without intermediate supervision, making the iteration budget a design parameter for tasks with many sequential steps.
Future Directions
- Whether the staircase survives on tasks mixing retrieval with computation, or on natural language
- Whether the results scale to the multi-billion parameter regime with preludes/codas around the loop
- Why tied training reaches the staircase while untied stacks need intermediate supervision
- Whether running a hop-packing model for more iterations than trained extends its walk
Related papers
- SWE-growth: Teacher-Guided Adaptive Self-Play for Software Engineering Agents
SWE-Growth uses a frozen teacher to guide self-play curricula, improving software agents to 31.7% on SWE-bench Pro and 17.2% on TerminalBench, surpassing student-only training.
- Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
Looped Transformers perform state-dependent computation, not fixed one-loop-one-step algorithms, and a learned linear layer can steer frozen shared weights via attention routing.
- Do Halting Heads Really Save Compute in Looped Language Models?
Halting heads in looped language models save far less compute than reported because mixture-trained heads rarely stop tokens and batching overhead eliminates most remaining gains.