Full text not available for this paper
Summary (Overview)
- Core Question: This paper investigates whether each loop in a Looped Transformer performs the same computation, and if not, what routes the shared parameters to different operations.
- Key Finding 1: Looped Transformers do not follow a fixed "one loop = one algorithmic step" rule—task progress varies across loops even with shared weights, as demonstrated in graph-walk tasks.
- Key Finding 2: A learned linear layer applied to the entering hidden state can steer a frozen loop toward different transitions (e.g., one-hop vs. two-hop targets) without modifying the shared Transformer layers.
- Key Finding 3: Attention pattern patching recovers steering effects and can switch predicted targets, identifying attention routing as a causal pathway for state-dependent computation.
- Key Finding 4: The controllability of transitions depends on what the backbone has learned during training—intermediate supervision changes which computations can induce.
Introduction and Theoretical Foundation
Background and Motivation
Looped Transformers reuse the same set of Transformer layers across multiple loops (Dehghani et al., 2019; Yang et al., 2024), providing a recurrent architecture for latent computation. Prior work has shown these models can:
- Learn iterative learning algorithms and multi-step optimization procedures (Yang et al., 2024; Gatmiry et al., 2024)
- Improve length generalization on tasks with iterative solutions (Fan et al., 2025)
- Use additional loops as latent test-time computation at language-model scale (Geiping et al., 2025; Zhu et al., 2025)
Theoretical Gap
The paper identifies a critical gap: weight sharing alone does not imply that every loop performs the same operation. The parameters are shared, but the hidden state entering those parameters changes as computation proceeds. The training objective can leave intermediate trajectories underdetermined when only final answers are supervised.
Key Theoretical Framework
The paper frames the problem as distinguishing between:
- What algorithms an architecture can express vs. what computation is selected by training
- The role of the entering hidden state as a control signal for frozen shared parameters
Methodology
Task Setup: Graph Walks
The primary test case uses graph walks where:
- Each input consists of a permutation on a graph , a start node , and a requested path depth
- The target is (the node reached after 8 steps)
- Models D8L8 and D8L6 repeat a two-layer Transformer block for 8 and 6 loops respectively
State Steering: The Linear Controller
The core control mechanism modifies the loop input:
where is the frozen shared Transformer block and is a learned linear layer with:
- A learned diagonal term
- A rank-48 term
- A bias term
The parameterization is:
This token-wise, linear, low-rank form constrains 's expressive power—it cannot aggregate graph information across token positions on its own; the frozen block must execute that part of the transition.
Attention Patching Interventions
The paper uses activation patching to separate attention routing from retrieved content:
Supervision Comparison
Two training objectives are compared:
where is the cross-entropy loss at loop .
Models Evaluated
- D8L8/D8L6 graph models: Two-layer Transformer blocks with hidden dimension 256
- Ouro-2.6B: A large language model fine-tuned on letter-walk and antonym cancellation tasks
- Qwen3-8B: Task-adapted for numerical/calendar tasks
- Synthetic knowledge graph: 64 entities, 16 relations, relation composition task
Empirical Validation / Results
1. Task Progress Varies Across Loops
Native readout trajectories reveal different patterns:
- Some models advance mainly one node per loop
- Others advance two nodes per loop
- Some stabilize at the final answer even with additional loops
2. State Steering Redirects Next Target
One-loop control results (D8L6 seed 6):
| Condition | One-hop Accuracy | Two-hop Accuracy |
|---|---|---|
| No | 48.2% | — |
| With | 99.6% | — |
| With | — | 98.2% |
Two-loop composition results:
| Sequence | Accuracy |
|---|---|
| One–one | 93.6% |
| One–two | 65.3% |
| Two–one | 74.5% |
| Two–two | 68.1% |
Longer reuse degrades: At the eighth additional controlled loop, exact match is approximately 10% for all sequence families.
3. Attention Routing Mediates State-Dependent Computation
Pattern vs. output patching:
| Model | Pattern patch → E | Output patch → D |
|---|---|---|
| D8L6 | 77.8% | 80.9% |
| Ouro | 71.1% | 93.8% |
Steering transfer via pattern patching (D8L6):
- Patching steered patterns into unsteered run: one-hop accuracy rises from 39.2% to 61.6%
- Reverse patch reduces accuracy from 99.7% to 68.1%
Target switching (patching between one-hop and two-hop runs):
| Patch direction | One-hop answer rate | Two-hop answer rate |
|---|---|---|
| Two→One | 30.3% | 65.9% |
| One→Two | 78.5% | 20.8% |
4. Steering Depends on Backbone's Learned Algorithms
Matched graph supervision results (D8L8):
| Supervision | Stay | One-hop | Two-hop |
|---|---|---|---|
| Final-only + | 100.0% | 66.5% | 50.5% |
| Stepwise + | 100.0% | 100.0% | 13.2% |
The ordering reverses between one-hop and two-hop accuracy depending on supervision regime.
Ouro antonym cancellation (depths 5–8):
- Final-only + : 98.0% accuracy
- Stepwise + : 7.8% accuracy
5. Depth Extrapolation Limitations
- Parity task: Steering improves accuracy from 45.05% to 73.73% over lengths 101–500, but degrades at lengths 505–1000 (36.40% with vs. 47.55% without)
- Graph continuation: Accuracy falls from 97.42% (loops 9–16) to 35.11% (loops 17–32) beyond the map-training range
6. Additional Evidence
Qwen3-8B continuation (long requests, –8):
- + : 89.69% accuracy
- only: 28.59%
- only: 31.25%
- Stop at : 25.62%
Knowledge graph relation composition:
| Relation length | only | With affine |
|---|---|---|
| 1–3 | 99.87% | 100.00% |
| 4–16 | 2.53% | 99.92% |
| 17–24 | 1.54% | 42.87% |
| 25–32 | 1.56% | 1.65% |
Theoretical and Practical Implications
Theoretical Implications
-
Shared parameters ≠ shared computation: Weight sharing does not imply a fixed one-loop-one-step correspondence. The computation performed by a shared block depends on the hidden state with which it is entered.
-
Hidden state as control signal: The evolving hidden state acts as a "program selector" that routes the shared computation toward different algorithms.
-
Attention as routing mechanism: Attention patterns carry the state-dependent selection of which computation to execute, providing a causal pathway for control.
-
Learning-dependent controllability: What can be steered depends on what the backbone has learned— selects computations learned by the backbone rather than creating new algorithms.
Practical Implications
-
Test-time computation control: The ability to steer frozen models through state modifications suggests practical methods for controlling latent reasoning without retraining.
-
Finite operating regime: Controllers degrade under longer composition, highlighting that successful long-horizon recurrent reasoning requires states that remain compatible with repeatedly routing computation.
-
Supervision design: The choice of training supervision (final-only vs. stepwise) fundamentally shapes what computations can be steered later.
Conclusion
Main Takeaways
- Looped Transformers exhibit state-dependent computation: the same frozen parameters perform different operations depending on the entering hidden state.
- A simple linear layer can redirect computation toward different semantic targets, demonstrating that the hidden state can control shared computation.
- Attention routing is a causal pathway for this control—patching attention patterns recovers steering effects and switches predicted targets.
- Backbone training shapes controllability: intermediate supervision changes which transitions can be induced by the controller.
Future Directions
- Understanding why controllers degrade under longer composition and how to maintain state compatibility
- Investigating whether more sophisticated controllers (beyond linear maps) can extend the operating regime
- Scaling these findings to larger models and more complex reasoning tasks
- Exploring how to design training objectives that produce backbones with desired controllability properties
Key Limitations
- The paper does not establish equivalence between pattern patching and full steering
- Results are demonstrated on synthetic tasks and task-adapted models, not general-purpose reasoning
- The broad evaluation sample in graph continuation was not verified against training exclusions
- Controllers show order-dependent accuracy in composition and degrade with longer reuse
Related papers
- Language Models Can Rewrite Their Own Weights
A 27B language model improved its own accuracy by 6.6 points and repaired a damaged copy of itself using only forward passes, with no gradients or training.
- InfuseLoop: Cross-Trajectory Latent Injection for Scalable Parallel Looped Transformers
InfuseLoop injects cross-trajectory latent states into parallel looped transformers, eliminating training spikes and enabling quality gains at 8 loops with minimal overhead.
- Do Halting Heads Really Save Compute in Looped Language Models?
Halting heads in looped language models save far less compute than reported because mixture-trained heads rarely stop tokens and batching overhead eliminates most remaining gains.