# Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does

> Looped Transformers perform state-dependent computation, not fixed one-loop-one-step algorithms, and a learned linear layer can steer frozen shared weights via attention routing.

- **Source:** [arXiv](https://arxiv.org/abs/2609.39892)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/4wbbag
- **Whiteboard:** https://picx.dev/p/4wbbag/image

## Summary

## Summary (Overview)

- **Core Question**: This paper investigates whether each loop in a Looped Transformer performs the same computation, and if not, what routes the shared parameters to different operations.
- **Key Finding 1**: Looped Transformers do not follow a fixed "one loop = one algorithmic step" rule—task progress varies across loops even with shared weights, as demonstrated in graph-walk tasks.
- **Key Finding 2**: A learned linear layer $J$ applied to the entering hidden state can steer a frozen loop toward different transitions (e.g., one-hop vs. two-hop targets) without modifying the shared Transformer layers.
- **Key Finding 3**: Attention pattern patching recovers steering effects and can switch predicted targets, identifying attention routing as a causal pathway for state-dependent computation.
- **Key Finding 4**: The controllability of transitions depends on what the backbone has learned during training—intermediate supervision changes which computations $J$ can induce.

## Introduction and Theoretical Foundation

### Background and Motivation

Looped Transformers reuse the same set of Transformer layers across multiple loops (Dehghani et al., 2019; Yang et al., 2024), providing a recurrent architecture for latent computation. Prior work has shown these models can:
- Learn iterative learning algorithms and multi-step optimization procedures (Yang et al., 2024; Gatmiry et al., 2024)
- Improve length generalization on tasks with iterative solutions (Fan et al., 2025)
- Use additional loops as latent test-time computation at language-model scale (Geiping et al., 2025; Zhu et al., 2025)

### Theoretical Gap

The paper identifies a critical gap: **weight sharing alone does not imply that every loop performs the same operation**. The parameters are shared, but the hidden state entering those parameters changes as computation proceeds. The training objective can leave intermediate trajectories underdetermined when only final answers are supervised.

### Key Theoretical Framework

The paper frames the problem as distinguishing between:
1. **What algorithms an architecture can express** vs. **what computation is selected by training**
2. The role of the **entering hidden state** as a control signal for frozen shared parameters

## Methodology

### Task Setup: Graph Walks

The primary test case uses graph walks where:
- Each input consists of a permutation $f_G$ on a graph $G$, a start node $s$, and a requested path depth $d = 8$
- The target is $f^8_G(s)$ (the node reached after 8 steps)
- Models D8L8 and D8L6 repeat a two-layer Transformer block for 8 and 6 loops respectively

### State Steering: The Linear Controller $J$

The core control mechanism modifies the loop input:

$$h_{t+1} = F(J(h_t))$$

where $F$ is the frozen shared Transformer block and $J$ is a learned linear layer with:
- A learned diagonal term
- A rank-48 term
- A bias term

The parameterization is: $J(h) = h(D + AB) + b$

This token-wise, linear, low-rank form constrains $J$'s expressive power—it **cannot aggregate graph information across token positions** on its own; the frozen block $F$ must execute that part of the transition.

### Attention Patching Interventions

The paper uses activation patching to separate attention routing from retrieved content:

$$z_{\text{raw}} = \alpha_2 V_2 \quad \text{(original)}$$
$$z_{\text{pattern}} = \alpha_1 V_2 \quad \text{(pattern patch)}$$
$$z_{\text{output}} = z_1 = \alpha_1 V_1 \quad \text{(output patch)}$$

### Supervision Comparison

Two training objectives are compared:

$$\mathcal{L}_{\text{final}} = \ell_8, \quad \mathcal{L}_{\text{stepwise}} = \ell_8 + \frac{1}{7}\sum_{t=1}^{7}\ell_t$$

where $\ell_t = \text{CE}(R(h_t), f^t_G(s))$ is the cross-entropy loss at loop $t$.

### Models Evaluated

1. **D8L8/D8L6 graph models**: Two-layer Transformer blocks with hidden dimension 256
2. **Ouro-2.6B**: A large language model fine-tuned on letter-walk and antonym cancellation tasks
3. **Qwen3-8B**: Task-adapted for numerical/calendar tasks
4. **Synthetic knowledge graph**: 64 entities, 16 relations, relation composition task

## Empirical Validation / Results

### 1. Task Progress Varies Across Loops

Native readout trajectories reveal different patterns:
- Some models advance mainly **one node per loop**
- Others advance **two nodes per loop**
- Some stabilize at the final answer even with additional loops

### 2. State Steering Redirects Next Target

**One-loop control results (D8L6 seed 6):**

| Condition | One-hop Accuracy | Two-hop Accuracy |
|-----------|-----------------|------------------|
| No $J$ | 48.2% | — |
| With $J_{\text{one}}$ | 99.6% | — |
| With $J_{\text{two}}$ | — | 98.2% |

**Two-loop composition results:**

| Sequence | Accuracy |
|----------|----------|
| One–one | 93.6% |
| One–two | 65.3% |
| Two–one | 74.5% |
| Two–two | 68.1% |

**Longer reuse degrades**: At the eighth additional controlled loop, exact match is approximately 10% for all sequence families.

### 3. Attention Routing Mediates State-Dependent Computation

**Pattern vs. output patching:**

| Model | Pattern patch → E | Output patch → D |
|-------|-------------------|------------------|
| D8L6 | 77.8% | 80.9% |
| Ouro | 71.1% | 93.8% |

**Steering transfer via pattern patching (D8L6):**
- Patching steered patterns into unsteered run: one-hop accuracy rises from 39.2% to 61.6%
- Reverse patch reduces accuracy from 99.7% to 68.1%

**Target switching (patching between one-hop and two-hop runs):**

| Patch direction | One-hop answer rate | Two-hop answer rate |
|-----------------|--------------------|--------------------|
| Two→One | 30.3% | 65.9% |
| One→Two | 78.5% | 20.8% |

### 4. Steering Depends on Backbone's Learned Algorithms

**Matched graph supervision results (D8L8):**

| Supervision | Stay | One-hop | Two-hop |
|-------------|------|---------|---------|
| Final-only + $J$ | 100.0% | 66.5% | 50.5% |
| Stepwise + $J$ | 100.0% | 100.0% | 13.2% |

The ordering reverses between one-hop and two-hop accuracy depending on supervision regime.

**Ouro antonym cancellation (depths 5–8):**
- Final-only + $J$: 98.0% accuracy
- Stepwise + $J$: 7.8% accuracy

### 5. Depth Extrapolation Limitations

- **Parity task**: Steering improves accuracy from 45.05% to 73.73% over lengths 101–500, but degrades at lengths 505–1000 (36.40% with $J$ vs. 47.55% without)
- **Graph continuation**: Accuracy falls from 97.42% (loops 9–16) to 35.11% (loops 17–32) beyond the map-training range

### 6. Additional Evidence

**Qwen3-8B continuation (long requests, $k=5$–8):**
- $J$ + $F$: 89.69% accuracy
- $F$ only: 28.59%
- $J$ only: 31.25%
- Stop at $h_4$: 25.62%

**Knowledge graph relation composition:**

| Relation length | $F$ only | With affine $J$ |
|-----------------|----------|-----------------|
| 1–3 | 99.87% | 100.00% |
| 4–16 | 2.53% | 99.92% |
| 17–24 | 1.54% | 42.87% |
| 25–32 | 1.56% | 1.65% |

## Theoretical and Practical Implications

### Theoretical Implications

1. **Shared parameters ≠ shared computation**: Weight sharing does not imply a fixed one-loop-one-step correspondence. The computation performed by a shared block depends on the hidden state with which it is entered.

2. **Hidden state as control signal**: The evolving hidden state acts as a "program selector" that routes the shared computation toward different algorithms.

3. **Attention as routing mechanism**: Attention patterns carry the state-dependent selection of which computation to execute, providing a causal pathway for control.

4. **Learning-dependent controllability**: What can be steered depends on what the backbone has learned—$J$ selects computations learned by the backbone rather than creating new algorithms.

### Practical Implications

1. **Test-time computation control**: The ability to steer frozen models through state modifications suggests practical methods for controlling latent reasoning without retraining.

2. **Finite operating regime**: Controllers degrade under longer composition, highlighting that successful long-horizon recurrent reasoning requires states that remain compatible with repeatedly routing computation.

3. **Supervision design**: The choice of training supervision (final-only vs. stepwise) fundamentally shapes what computations can be steered later.

## Conclusion

### Main Takeaways

1. Looped Transformers exhibit **state-dependent computation**: the same frozen parameters perform different operations depending on the entering hidden state.
2. A simple linear layer $J$ can redirect computation toward different semantic targets, demonstrating that **the hidden state can control shared computation**.
3. **Attention routing is a causal pathway** for this control—patching attention patterns recovers steering effects and switches predicted targets.
4. **Backbone training shapes controllability**: intermediate supervision changes which transitions can be induced by the controller.

### Future Directions

- Understanding why controllers degrade under longer composition and how to maintain state compatibility
- Investigating whether more sophisticated controllers (beyond linear maps) can extend the operating regime
- Scaling these findings to larger models and more complex reasoning tasks
- Exploring how to design training objectives that produce backbones with desired controllability properties

### Key Limitations

- The paper does not establish equivalence between pattern patching and full steering
- Results are demonstrated on synthetic tasks and task-adapted models, not general-purpose reasoning
- The broad evaluation sample in graph continuation was not verified against training exclusions
- Controllers show order-dependent accuracy in composition and degrade with longer reuse

---

_Markdown view of https://picx.dev/p/4wbbag, served by PicX — AI-generated visual whiteboard summaries of research papers._
