Full text not available for this paper

Summary (Overview)

  • Core Question: This paper investigates whether each loop in a Looped Transformer performs the same computation, and if not, what routes the shared parameters to different operations.
  • Key Finding 1: Looped Transformers do not follow a fixed "one loop = one algorithmic step" rule—task progress varies across loops even with shared weights, as demonstrated in graph-walk tasks.
  • Key Finding 2: A learned linear layer JJ applied to the entering hidden state can steer a frozen loop toward different transitions (e.g., one-hop vs. two-hop targets) without modifying the shared Transformer layers.
  • Key Finding 3: Attention pattern patching recovers steering effects and can switch predicted targets, identifying attention routing as a causal pathway for state-dependent computation.
  • Key Finding 4: The controllability of transitions depends on what the backbone has learned during training—intermediate supervision changes which computations JJ can induce.

Introduction and Theoretical Foundation

Background and Motivation

Looped Transformers reuse the same set of Transformer layers across multiple loops (Dehghani et al., 2019; Yang et al., 2024), providing a recurrent architecture for latent computation. Prior work has shown these models can:

  • Learn iterative learning algorithms and multi-step optimization procedures (Yang et al., 2024; Gatmiry et al., 2024)
  • Improve length generalization on tasks with iterative solutions (Fan et al., 2025)
  • Use additional loops as latent test-time computation at language-model scale (Geiping et al., 2025; Zhu et al., 2025)

Theoretical Gap

The paper identifies a critical gap: weight sharing alone does not imply that every loop performs the same operation. The parameters are shared, but the hidden state entering those parameters changes as computation proceeds. The training objective can leave intermediate trajectories underdetermined when only final answers are supervised.

Key Theoretical Framework

The paper frames the problem as distinguishing between:

  1. What algorithms an architecture can express vs. what computation is selected by training
  2. The role of the entering hidden state as a control signal for frozen shared parameters

Methodology

Task Setup: Graph Walks

The primary test case uses graph walks where:

  • Each input consists of a permutation fGf_G on a graph GG, a start node ss, and a requested path depth d=8d = 8
  • The target is fG8(s)f^8_G(s) (the node reached after 8 steps)
  • Models D8L8 and D8L6 repeat a two-layer Transformer block for 8 and 6 loops respectively

State Steering: The Linear Controller JJ

The core control mechanism modifies the loop input:

ht+1=F(J(ht))h_{t+1} = F(J(h_t))

where FF is the frozen shared Transformer block and JJ is a learned linear layer with:

  • A learned diagonal term
  • A rank-48 term
  • A bias term

The parameterization is: J(h)=h(D+AB)+bJ(h) = h(D + AB) + b

This token-wise, linear, low-rank form constrains JJ's expressive power—it cannot aggregate graph information across token positions on its own; the frozen block FF must execute that part of the transition.

Attention Patching Interventions

The paper uses activation patching to separate attention routing from retrieved content:

zraw=α2V2(original)z_{\text{raw}} = \alpha_2 V_2 \quad \text{(original)} zpattern=α1V2(pattern patch)z_{\text{pattern}} = \alpha_1 V_2 \quad \text{(pattern patch)} zoutput=z1=α1V1(output patch)z_{\text{output}} = z_1 = \alpha_1 V_1 \quad \text{(output patch)}

Supervision Comparison

Two training objectives are compared:

Lfinal=ℓ8,Lstepwise=ℓ8+17∑t=17ℓt\mathcal{L}_{\text{final}} = \ell_8, \quad \mathcal{L}_{\text{stepwise}} = \ell_8 + \frac{1}{7}\sum_{t=1}^{7}\ell_t

where ℓt=CE(R(ht),fGt(s))\ell_t = \text{CE}(R(h_t), f^t_G(s)) is the cross-entropy loss at loop tt.

Models Evaluated

  1. D8L8/D8L6 graph models: Two-layer Transformer blocks with hidden dimension 256
  2. Ouro-2.6B: A large language model fine-tuned on letter-walk and antonym cancellation tasks
  3. Qwen3-8B: Task-adapted for numerical/calendar tasks
  4. Synthetic knowledge graph: 64 entities, 16 relations, relation composition task

Empirical Validation / Results

1. Task Progress Varies Across Loops

Native readout trajectories reveal different patterns:

  • Some models advance mainly one node per loop
  • Others advance two nodes per loop
  • Some stabilize at the final answer even with additional loops

2. State Steering Redirects Next Target

One-loop control results (D8L6 seed 6):

ConditionOne-hop AccuracyTwo-hop Accuracy
No JJ48.2%—
With JoneJ_{\text{one}}99.6%—
With JtwoJ_{\text{two}}—98.2%

Two-loop composition results:

SequenceAccuracy
One–one93.6%
One–two65.3%
Two–one74.5%
Two–two68.1%

Longer reuse degrades: At the eighth additional controlled loop, exact match is approximately 10% for all sequence families.

3. Attention Routing Mediates State-Dependent Computation

Pattern vs. output patching:

ModelPattern patch → EOutput patch → D
D8L677.8%80.9%
Ouro71.1%93.8%

Steering transfer via pattern patching (D8L6):

  • Patching steered patterns into unsteered run: one-hop accuracy rises from 39.2% to 61.6%
  • Reverse patch reduces accuracy from 99.7% to 68.1%

Target switching (patching between one-hop and two-hop runs):

Patch directionOne-hop answer rateTwo-hop answer rate
Two→One30.3%65.9%
One→Two78.5%20.8%

4. Steering Depends on Backbone's Learned Algorithms

Matched graph supervision results (D8L8):

SupervisionStayOne-hopTwo-hop
Final-only + JJ100.0%66.5%50.5%
Stepwise + JJ100.0%100.0%13.2%

The ordering reverses between one-hop and two-hop accuracy depending on supervision regime.

Ouro antonym cancellation (depths 5–8):

  • Final-only + JJ: 98.0% accuracy
  • Stepwise + JJ: 7.8% accuracy

5. Depth Extrapolation Limitations

  • Parity task: Steering improves accuracy from 45.05% to 73.73% over lengths 101–500, but degrades at lengths 505–1000 (36.40% with JJ vs. 47.55% without)
  • Graph continuation: Accuracy falls from 97.42% (loops 9–16) to 35.11% (loops 17–32) beyond the map-training range

6. Additional Evidence

Qwen3-8B continuation (long requests, k=5k=5–8):

  • JJ + FF: 89.69% accuracy
  • FF only: 28.59%
  • JJ only: 31.25%
  • Stop at h4h_4: 25.62%

Knowledge graph relation composition:

Relation lengthFF onlyWith affine JJ
1–399.87%100.00%
4–162.53%99.92%
17–241.54%42.87%
25–321.56%1.65%

Theoretical and Practical Implications

Theoretical Implications

  1. Shared parameters ≠ shared computation: Weight sharing does not imply a fixed one-loop-one-step correspondence. The computation performed by a shared block depends on the hidden state with which it is entered.

  2. Hidden state as control signal: The evolving hidden state acts as a "program selector" that routes the shared computation toward different algorithms.

  3. Attention as routing mechanism: Attention patterns carry the state-dependent selection of which computation to execute, providing a causal pathway for control.

  4. Learning-dependent controllability: What can be steered depends on what the backbone has learned—JJ selects computations learned by the backbone rather than creating new algorithms.

Practical Implications

  1. Test-time computation control: The ability to steer frozen models through state modifications suggests practical methods for controlling latent reasoning without retraining.

  2. Finite operating regime: Controllers degrade under longer composition, highlighting that successful long-horizon recurrent reasoning requires states that remain compatible with repeatedly routing computation.

  3. Supervision design: The choice of training supervision (final-only vs. stepwise) fundamentally shapes what computations can be steered later.

Conclusion

Main Takeaways

  1. Looped Transformers exhibit state-dependent computation: the same frozen parameters perform different operations depending on the entering hidden state.
  2. A simple linear layer JJ can redirect computation toward different semantic targets, demonstrating that the hidden state can control shared computation.
  3. Attention routing is a causal pathway for this control—patching attention patterns recovers steering effects and switches predicted targets.
  4. Backbone training shapes controllability: intermediate supervision changes which transitions can be induced by the controller.

Future Directions

  • Understanding why controllers degrade under longer composition and how to maintain state compatibility
  • Investigating whether more sophisticated controllers (beyond linear maps) can extend the operating regime
  • Scaling these findings to larger models and more complex reasoning tasks
  • Exploring how to design training objectives that produce backbones with desired controllability properties

Key Limitations

  • The paper does not establish equivalence between pattern patching and full steering
  • Results are demonstrated on synthetic tasks and task-adapted models, not general-purpose reasoning
  • The broad evaluation sample in graph continuation was not verified against training exclusions
  • Controllers show order-dependent accuracy in composition and degrade with longer reuse

Related papers