Summary (Overview)

• Core finding: The reusability of depth in looped language models depends on agreement between the training and inference loop topologies, not on the presence of loop-boundary normalization itself.
• Key result on released weights: Removing the loop-boundary normalization from Ouro-1.4B and Ouro-2.6B (without adding parameters and with the readout path bit-identical) drives accuracy to zero by the trained budget on a balanced binary task, while the state norm stays within a factor of 2.021—contradicting the residual-growth failure account.
• Key result on trained-from-scratch models: In 19M-parameter models trained from scratch (16 seeds per configuration), training without the boundary normalization is never worse than training with it, and extrapolates better beyond the trained budget (Cliff's δ up to 0.867).
• Mechanism localization: The damage sits in the readout together with the last layer of the block—refitting the head over a frozen modified loop recovers ~33% of lost accuracy and plateaus, while refitting the last block layer with it recovers 100%.
• Methodological contribution: Comparing looped vs. stacked depth at equal absolute inference loop count is confounded by unequal overshoot of the trained budget; at matched overshoot $K/r_0$ (also equal compute), the advantage of looping vanishes on one task and shrinks on the other.

---

Introduction and Theoretical Foundation

Looped language models apply one stack of layers to the state $r$ times, with $r$ chosen at inference. The design only pays off if loop steps beyond the trained budget $r_0$ remain usable. The paper asks: what keeps depth reusable?

Two released models differ in a key detail:

• Ouro (Zhu et al., 2025): renormalizes the state at the end of every loop step, using the same module that normalizes the readout input.
• Huginn (Geiping et al., 2025): applies that module once per forward pass.

The paper calls this placement the loop-boundary normalization and makes it an architectural hypothesis testable without training.

Two competing explanations are examined:

1. Normalisation is required: Without boundary normalization, the state diverges and the loop stops being reusable.
2. Agreement is required: The readout is calibrated on the states that training produces, and any change to the loop topology moves its input off that distribution—in either direction.

The nearest prior account (Sharma & Vu, 2026) predicts that an architecture with inter-loop normalization avoids failure by construction, and the residual-growth account expects failure through the scale of the state. Both are contradicted by the measurements here.

---

Methodology

Models

The axis: loop-boundary normalization

The placement is determined by instrumentation: every normalization module is instrumented, and its calls are counted over one forward pass at two loop counts. A module whose count grows with loop count acts inside the loop; one called the same number of times acts on the readout path.

Two operations follow:

• For Ouro: restrict the model-level normalization to the final loop step—readout path stays bit-identical, only the state re-entering the loop changes.
• For Huginn: apply the readout-path normalization to the state before it re-enters the block.

At $r = 1$, every operation is an identity by construction, serving as an implementation check: an arm whose $r=1$ numbers differ from baseline is discarded.

Tasks

Three synthetic task families with an exact per-instance depth $d^\star$ and independent width control $w$:

• circuit: evaluates a Boolean circuit of depth $d^\star$.
• hops: follows $d^\star$ pointers through an in-context map with 26 possible answers.
• paragg: computes the parity of a subset of answers.

Binary families are balanced at 50/50 in every stratum; accuracy floor is 0.500 for binary families.

Metrics and statistics

• Accuracy: arg-max over full vocabulary vs. gold answer token, with explicit floor (chance within answer alphabet or best constant answer).
• Readout entropy (bits), top-1 minus top-2 logit, state norm per loop step, mutual information (Miller-Madow corrected), modal share.
• Released weights: exact McNemar test with cluster bootstrap, Holm correction.
• Trained from scratch: exact Mann-Whitney test over seeds (16 per configuration) and Cliff's δ.

A competence gate is fixed in advance: a pair enters analysis only if peak accuracy reaches 0.30 absolute and exceeds floor by > 0.05.

---

Empirical Validation / Results

4.1 The rearrangement is asymmetric

The identity check passes exactly: at $r = 1$, differences in accuracy and readout entropy between arms are 0.0000 in all nine pairs.

Removing boundary normalization from Ouro breaks the model within a few loop steps:

On hops, the same arm reaches 0.000 by $r=8$ while the unmodified model peaks at 0.166 against a majority floor of 0.072.

4.2 The readout collapses; the state scale does not

The residual-growth prediction is not borne out:

Across every cell of the grid, the ratio of the state norm at $r$ to its value at $r=1$ stays below 2.021.

The readout changes in two stages:

1. Entropy rises: from 0.786 bits at $r=1$ to 12.470 bits at $r=2$—close to uniform over the vocabulary (~15 bits). The head stops answering in format.
2. Entropy falls back: to 5.672 bits at $r=32$, but top-1 minus top-2 logit grows from 0.263 to 1.247, modal share reaches 0.83, and accuracy stays at zero—the head commits to a constant token outside the answer alphabet.

A linear probe fitted to the states at the trained budget reads at chance (0.556 vs. floor 0.519 on circuit), while the model output is zero—so the answer is not recoverable linearly either.

Dose response: normalizing every $k$-th loop step (always including the last):

The response is a threshold, not a gradient.

4.3 Trained from scratch: normalization hurts away from the trained depth

Two configurations differ only in whether the state is normalized at the end of every loop step; both share $r_0 = 4$.

On circuit:

Cliff's δ reaches 0.867 beyond the budget. The normalised arm peaks at its trained depth and degrades on both sides.

On hops: indistinguishable at trained depth (0.999 vs. 0.988, $p = 0.361$), but the normalised arm falls to 0.556 at $K=1$ (vs. 0.979) and 0.946 at $K=8$ (vs. 0.999).

State norm direction: The configuration without normalization lets its state norm grow by a factor of 4.84 (from 366 to 1772) and holds accuracy; the one with normalization holds the norm nearly constant (factor 1.45) and degrades. This reverses the residual-growth prediction.

4.4 The damage sits in the last block layer and the head

Freezing the loop and refitting only the head (unembedding + final normalization, 300 steps):

• Recovers 33% of lost accuracy on circuit (0.639 → 0.756), 31% on hops.
• 3000 steps move it from 33% to 35%—a plateau.

Unfreezing the last layer of the recurrent block (4.2M of 19M parameters):

• Recovers 43% of the loss within 30 steps, 100% by 3000.
• On hops, 98% within 30 steps.

4.5 Looped vs. stacked depth at matched overshoot

Fixed total of 24 layers per token, split three ways:

• 4 loop steps × 6 layers
• 2 loop steps × 12 layers
• 1 loop step × 24 layers (ordinary transformer)

At equal absolute $K = 32$: tidy ordering in favor of looping (0.907, 0.838, 0.490). But this is an artefact: the one-step configuration has $r_0 = 1$, so $K=32$ is a 32-fold overshoot vs. 8-fold for the four-step configuration.

At matched overshoot $K/r_0$ (also equal compute, since layers × $K = 24K/r_0$):

The advantage of looping over a stack is not detectable on circuit at 6 seeds and shrinks without closing on hops.

---

Theoretical and Practical Implications

Theoretical implications

1. Agreement is necessary, not sufficient: Both configurations in Sec. 4.3 agree with their own loop, yet one degrades beyond its budget. The normalization separates two arms that both satisfy agreement.

2. The residual-growth account is wrong: State norm barely moves (factor 2.021) while accuracy collapses to zero. The head leaves the answer alphabet and commits to a constant—a readout-level failure, not a scale-level failure.

3. A contract metaphor: Extra loop steps are usable only while the readout is calibrated for the loop it was trained with. The contract lives in the last block layer and the head that reads it.

4. Norm growth may damp the loop: The block's output scale is set by internal normalizations, so as the state grows, each application moves it by a smaller relative amount and the iteration settles. A boundary normalization rescales the state before every step, keeping updates at full relative size—so the readout sees a state that has not settled.

Practical implications

1. Report extrapolation against overshoot $K/r_0$, not absolute $K$, since the absolute axis confounds comparisons when budgets differ.
2. A modification inside the loop of a released model should be assumed to invalidate its readout until measured, even when the state scale is intact.
3. Loop-boundary normalization is a cost, not a benefit, given agreement: models trained without it extrapolate better in both directions from the trained depth.

---

Conclusion

The paper concludes that normalization placement inside the loop is not a stability hyperparameter but a contract between the loop and the readout. The key takeaways:

1. Removing boundary normalization from released weights breaks the model within a few loop steps while state scale stays within a factor of 2.021: the head leaves the answer alphabet and later commits to a constant that is not an answer.
2. A readout refit over a frozen loop recovers a third of the loss; retraining the last block layer with it recovers all of it.
3. Trained from scratch, the absence of normalization is never worse than its presence and better away from the trained depth—agreement is necessary for depth reusability, and given agreement, the normalization is a cost.
4. At matched overshoot, the advantage of looping over a stack is not detectable at six seeds on one task and reduced on the other.

Future directions implied by the work include: measuring per-step update size to test the damping interpretation; varying the readout normalization axis (the orthogonal axis studied by Sharma & Vu, 2026); and scaling the causal half of the study to larger models where rank tests over seeds become feasible.