Summary (Overview)

• This paper presents the first comprehensive, cross-architectural analysis of looped language models (LMs), examining 8 models from 4 distinct families (Ouro, LoopUS, Parcae, and Raven family including Huginn & Rec-OLMo) using a unified 5-lens inspection protocol rooted in Mechanistic Interpretability.
• Key architectural distinction identified: "carry loops" (Ouro, LoopUS) transmit the prompt only within the looped state, while "re-read loops" (Huginn, Rec-OLMo, Parcae) re-supply the prompt embedding at every iteration—this distinction largely determines model behavior.
• Major finding: Answers are decided early (76-89% of tokens settle on their final top-1 prediction at iteration 1 in most models), and more iterations are not always better—base Ouro peaks at its training depth (iteration 3-4) and degrades thereafter.
• Robustness divergence: Carry loops collapse under noise injection (at amplitude α = 4), while re-read loops remain largely unaffected even at α = 50, due to their training objective of reaching answers from random initial states.
• Attention heads do not multiplex: No model's attention heads change roles across iterations; only a shared, state-correlated drift (correlation r = 0.91–0.99 with hidden state updates) is observed.

Introduction and Theoretical Foundation

Background and Motivation

Looped language models (LMs) reuse a single transformer block multiple times per token generation, promising increased latent computation without additional storage cost. This design is directly motivated by chain-of-thought reasoning (Nye et al., 2021; Wei et al., 2022), where intermediate reasoning steps are performed in latent space rather than decoded as text (Saunshi et al., 2025; Hao et al., 2024).

Despite their rising popularity, analysis of looped LMs remains nascent. Key open questions include:

• Do iterative transformations actually improve capabilities?
• How do models remain stable under prompt perturbation?
• What happens to latent states during repeated application of a shared block?

Theoretical Foundations

The paper builds on several theoretical pillars:

Latent reasoning and fixed points: Re-read loops are trained with a random number of iterations, encouraging convergence to the desired token embedding from any starting point, similar to Deep Equilibrium Models (Bai et al., 2019; Schwarzschild et al., 2021).

Reading iterations without approximation: Unlike standard transformers where intermediate layers must be decoded by output heads they never trained to feed, looped LMs have every iteration ending at the same network position, enabling direct decoding without approximation:

$$h_{t+1} = f(h_t) \quad \text{(carry loop)}$$ $$h_{t+1} = f(h_t, e) \quad \text{(re-read loop)}$$

where $h_t$ is the hidden state at iteration $t$ and $e$ is the prompt embedding.

Architectural taxonomy:

• Carry loops (Ouro, LoopUS): Prompt enters once at iteration 0; subsequent iterations see only $h_t$. Trained at fixed depth (Ouro: 4 iterations) or with halting head (LoopUS: up to 20).
• Re-read loops (Huginn, Rec-OLMo, Parcae): Loop starts from random noise; prompt is re-supplied via embedding $e$ at every iteration. Trained with random iteration counts (mean 8–32).

Methodology

Models Analyzed

The Five Lenses

The investigation protocol comprises five analytical approaches:

1. Lens 1 (Trajectory): Does the hidden state settle? Measures whether late steps are less than a quarter of early steps (settled vs. drifting), counts top-1 flips, and measures attention focus on answer spans.

2. Lens 2 (Commitment): When is the answer decided? Identifies the first iteration from which the top-1 token never changes, and measures the share of total KL-divergence change occurring after the decision.

3. Lens 3 (Readout): Is the last iteration the best? Computes next-token accuracy, log-loss, and Expected Calibration Error (ECE) at each iteration on prompt tokens.

4. Lens 4 (Attention Heads): Do heads change roles? Builds attention fingerprints (8 statistics per head per iteration) and computes a movement ratio comparing within-head movement across iterations to between-head distances.

5. Lens 5 (Noise Injection): Does the loop recover from perturbation? Injects isotropic Gaussian noise scaled to state norm (amplitude $\alpha$) at the loop's entry point and evaluates output similarity, broken/fluent/refusal rates.

Data

• Lenses 1-4: Up to 300 prompts from MATH (5 difficulty levels), GSM8K, synthetic arithmetic, AdvBench/HarmBench harmful requests mixed with XSTest benign prompts, and HotpotQA question-answering.
• Lens 5: AdvBench, English CValues, WildChat subsets, TriviaQA, and MMLU.

Empirical Validation / Results

Lens 1: Hidden State Settling

• Re-read loops settle: Rec-OLMo, Huginn, and Parcae settle after generating 98–100% of tokens.
• Carry loops vary by size: Ouro-2.6B settles 93–96% of tokens; Ouro-1.4B only 10–19% (reasoning tuning improves this to 40–62%).
• Settling ≠ fixed prediction: Huginn's top-1 token changes 1.4–1.7 times per token despite settling.
• Attention focus: Ouro-1.4B raises attention to answer spans 16× within two iterations (to ~2× uniform); Huginn reaches the same level gradually; Parcae and Rec-OLMo remain at uniform level; LoopUS moves attention away (down to 1% of uniform).

Lens 2: Decision Timing

• Early decisions: 76–89% of answer tokens are decided at iteration 1 for Ouro, Parcae, and Rec-OLMo; 90% decided by iteration 2–3.
• LoopUS: 33% decided at iteration 1, but 96% by iteration 2.
• Huginn: The only gradual decider—33% at iteration 1, 90% only by iteration 8–11 of 16.
• Post-decision refinement: 73–88% of distribution change happens after the top-1 token is decided.
• Exit gates are conservative: Ouro's gate fires 3.2–4.1 iterations after the answer is fixed; LoopUS's fires 16.6 iterations after.

Lens 3: Optimal Iteration Depth

• Base Ouro peaks at training depth: Ouro-2.6B best at iteration 4; Ouro-1.4B best at iteration 3–4. Extra iterations lower accuracy (e.g., 0.605 → 0.580 for Ouro-1.4B) and worsen calibration (ECE rises from 0.057 to 0.087).
• Ouro-Thinking degrades monotonically: Log-loss at last iteration is 5% (1.4B) and 22% (2.6B) higher than at first; ECE rises from 0.13–0.14 to 0.18–0.21. The loop breaks more correct first guesses (9.1%) than fixes incorrect ones (7.0%).
• Re-read loops plateau or improve: Huginn improves from 0.454 to 0.620 accuracy (37% log-loss reduction), achieving 95% of gain by iteration 8–10. Parcae and Rec-OLMo gain 2–4 points then remain flat.
• Majority voting never helps: For 3–11% of tokens, an intermediate iteration predicts correctly while the last does not, yet majority vote never beats the best single iteration.

Lens 4: Attention Head Stability

• No multiplexing: Movement ratios range from 0.02 (Rec-OLMo) to 0.44 (Ouro-1.4B). Permutation tests reject "no stable identity" for every model.
• Shared drift dominates: ~50% of head movement is shared across all heads and correlates linearly with hidden state updates (r = 0.91–0.99). Removing it halves the movement ratio for Ouro (0.44 → 0.24).
• Carry loops show consistent, prompt-independent adjustment (cross-prompt correlation r = 0.95–0.98); re-read loops show less consistency (r = 0.29–0.69).

Lens 5: Noise Robustness

Key observations:

• Isotropic noise rarely jailbreaks models (at most ≤4% of safety answers).
• At α = 2, carry models misread requests (24 of 73 flipped answers respond to a harmless misread version).
• At α ≥ 4, carry loops exhibit "prompt amnesia"—language capabilities survive but the prompt is erased from its only storage location.
• Loop depth and smoothing gates do not predict robustness (LoopUS with 20 iterations and smoothing is as fragile as Ouro).

Theoretical and Practical Implications

Theoretical Implications

1. Architecture determines behavior: The carry vs. re-read distinction is the primary determinant of loop dynamics—more so than model size, depth, or training recipe. This provides a principled taxonomy for future looped LM research.

2. Looped LMs learn fixed-depth pipelines, not iterative refinement: Base Ouro's optimal performance at its training depth (4 iterations) suggests the model learns a 4-step computation pipeline rather than a process that keeps improving. This challenges the implicit assumption that more iterations = better reasoning.

3. Latent reasoning is not chain-of-thought: The early commitment to answers (iteration 1–2) and subsequent refinement suggests looped LMs behave more like fixed-depth feature extractors with refinement stages than emergent chain-of-thought reasoners.

4. Attention head interpretability transfers: Since heads do not multiplex, the established taxonomy of standard transformer attention heads (Voita et al., 2019; Clark et al., 2019) can be applied to looped models, simplifying future interpretability work.

Practical Implications

1. Efficiency: Running carry loops beyond training depth wastes computation and can degrade performance. Deployment should match inference iterations to training depth.

2. Safety: Carry loops are vulnerable to activation-space perturbations—noise at α = 4 causes complete loss of alignment behavior. Re-read loops offer inherent robustness due to their training objective, making them preferable for safety-critical applications.

3. Calibration: Reasoning-tuned Ouro models become increasingly overconfident with each iteration (ECE rises from 0.13 to 0.21), suggesting that early-exit strategies could improve both efficiency and calibration.

4. Exit gates are suboptimal: Trained halting heads fire 3–17 iterations after the answer is already decided, suggesting substantial room for improvement in adaptive computation mechanisms.

Conclusion

This study provides the first comprehensive, cross-architectural analysis of looped language models using a unified 5-lens protocol. The central finding is that loop architecture—particularly input re-injection—determines model behavior:

• Carry loops achieve peak performance within their training depth, degrade with additional iterations (especially reasoning-tuned variants), and are fragile under noise injection (collapse at α = 4).
• Re-read loops improve or plateau with additional iterations and remain robust to noise even at α = 50.

The paper demonstrates that answers are typically decided within the first 1–2 iterations, attention heads maintain stable roles across iterations, and majority voting across iterations never improves accuracy.

Future directions identified by the authors include:

• Testing early-stopping mechanisms and their effects on generation quality
• Exploring whether less-refined earlier tokens propagate errors to subsequent tokens
• Developing better adaptive computation mechanisms than current exit gates
• Extending the analysis to larger models and additional looped architectures

The authors emphasize that their findings should guide both efficiency optimization (matching inference iterations to training depth) and safety engineering (preferring re-read architectures for robustness) in the development of future looped language models.