Summary (Overview)

• LoopCD is a training-free contrastive decoding framework for Looped Transformers that leverages intermediate recurrent states as aligned weak prediction references to guide the final, stronger prediction.
• Two variants are proposed: LoopCD-Logits (contrasts in logit space, one extra output pass) and LoopCD-Hidden (contrasts in hidden-state space, zero output overhead).
• At full recurrent depth, LoopCD delivers substantial gains: Ouro-2.6B-Thinking's AIME 2024 pass@1 rises from 61.88% to 73.33%, and Huginn's HumanEval pass@1 from 22.56% to 31.71%.
• LoopCD enables halving the number of recurrent loops while still matching or exceeding full-depth unguided baselines, reducing forward FLOPs by 22.5% to 48.2%.
• The method works across four Looped Transformer families (Ouro, Huginn, Parcae, Looped-Qwen3) spanning mathematical reasoning, code generation, and multiple-choice benchmarks.

---

Introduction and Theoretical Foundation

Background

Looped Transformers decouple depth from parameter count by repeatedly executing a shared block of layers. Each recurrent iteration refines the hidden representation, and standard decoding only uses the final state. However, every intermediate pass produces a hidden state that can be decoded into a valid token distribution — information that is typically discarded.

Theoretical Basis

The core insight is that a looped Transformer's recurrent trajectory inherently produces aligned weak-to-strong prediction pairs for the same prefix, without auxiliary models or modified prompts. This is analogous to:

• Contrastive decoding (Li et al., 2023): guiding a strong predictor away from an aligned weaker reference.
• Autoguidance in diffusion models (Karras et al., 2024): contrasting against a degraded version of the same network.
• Classifier-free guidance (Ho & Salimans, 2022).

Because the same shared parameter block is applied repeatedly, intermediate states lie in the same semantic representation space as the final state and can be decoded by the shared output head.

Key Formulation

For a fixed input prefix, let $h_0$ be the initial state and $h_k$ the state after $k$ iterations. The recurrent block applies an update $f$ for $R$ iterations:

$$\begin{array}{l} h _ {0} \xrightarrow {f} h _ {1} \xrightarrow {f} \dots \xrightarrow {f} \boldsymbol {h} _ {R}, \\ \boldsymbol {z} _ {R} = \operatorname{lmdhead} \bigl (\operatorname{coda} (\boldsymbol {h} _ {R}) \bigr). \end{array}\tag{1}$$

Standard autoregressive decoding samples from $p_R = \mathrm{softmax}(z_R)$.

---

Methodology

Guidance Space

LoopCD applies the contrast either before the coda layers (hidden space) or after the language modeling head (logit space).

LoopCD-Hidden

Combines reference and final hidden states before the coda layers:

$$h ^ {\prime} = \boldsymbol {h} _ {R} + \omega (\boldsymbol {h} _ {R} - h _ {1}), \quad \omega \geq 0.\tag{2}$$

The coefficient $\omega$ sets the strength of the contrast; $\omega = 0$ recovers the unguided state. The combined state passes through coda and lm_head once, preserving the baseline's single pass.

LoopCD-Logits

Contrasts logits from both states after the coda and language modeling head:

$$\begin{array}{l} z _ {1} = \text { lm\_head } \big (\text { coda } (h _ {1}) \big), \qquad z _ {R} = \text { lm\_head } \big (\text { coda } (h _ {R}) \big), \\ z ^ {\prime} = z _ {R} + \omega \left(z _ {R} - z _ {1}\right), \qquad \omega \geq 0. \end{array}\tag{3}$$

This requires one additional pass through the coda layers and lm_head.

Guidance Strength

Fixed Strength

A constant $\omega$ is used for every token within a given evaluation setting.

Adaptive Strength

Adjusts $\omega$ per token based on prediction uncertainty, using the margin between the two largest token probabilities:

$$\omega = \omega_ {\max} \left[ 1 - \left(p _ {R, (1)} - p _ {R, (2)}\right) \right],\tag{4}$$

where $p_{R,(1)}$ and $p_{R,(2)}$ are the two largest probabilities in $p_R = \mathrm{softmax}(z_R)$. The rule approaches the maximum strength as the leading probabilities become equal (high uncertainty) and reduces guidance when one candidate dominates.

---

Empirical Validation / Results

Mathematical Reasoning Gains

Table 1(a): AIME 2024, AIME 2025, OlympiadBench pass rates (Ouro-Thinking models, 16 samples per problem)

Code Generation Gains

Table 1(b): HumanEval and MBPP execution pass rates (base and extended tests)

• Huginn R=32: HumanEval base rises from 23.17% to 28.66% (adaptive logits); LoopCD-Hidden raises it to 31.71%.
• Huginn R=16: HumanEval base rises from 20.12% to 28.05%.
• Ouro-2.6B: HumanEval base rises from 75.61% to 79.88% (adaptive).

Multiple-Choice Benchmarks

Table 2: Seven-benchmark means under LoopCD-Logits (ARC-C, ARC-E, SciQ, MMLU, HellaSwag, WinoGrande, PIQA)

• Every evaluated model improves its benchmark mean.
• Adaptive strength leads in nearly all configurations.
• Largest gains on HellaSwag (e.g., Parcae-1.3B: +4.51 points) and ARC-Challenge (Parcae-1.3B: +4.10 points).

Zero-Overhead Hidden-Space Gains

Table 3: LoopCD-Hidden at fixed strength

LoopCD-Hidden exceeds fixed LoopCD-Logits gains in multiple-choice scoring while adding zero output overhead.

Compute Reduction at Half Depth

• Halving recurrent iterations incurs an unguided accuracy penalty of 0.17–1.29 points.
• Applying LoopCD at halved depth adds 0.75–1.29 points, closing or exceeding the deficit across all six settings.
• Forward FLOPs reduced by 22.5% to 48.2% at equal or superior accuracy.
• Huginn-0125 at 16/32 iterations outperforms its full-depth unguided baseline by +1.02 points (Logits) and +0.51 points (Hidden).

---

Theoretical and Practical Implications

Mechanism: Why LoopCD Works

1. Built-in weak-to-strong trajectory: The first recurrent iteration trails the final prediction by 4.0–23.7 points on the seven-benchmark mean. The first iteration's divergence from the final prediction falls from 0.71 bits (Ouro) to 0.013 after the third iteration.

2. Re-ranking drives gains: The contrast vector $z_R - z_1$ decomposes into:

   • Parallel component: rescales logits like a temperature (preserves token order).
   • Orthogonal component: re-ranks tokens (directly changes top-token selection).

   The orthogonal re-ranking component alone achieves gains matching or exceeding the full update (e.g., +1.72 vs +0.76 for Ouro-1.4B at $\omega = 0.5$).

3. Gains concentrate on uncertain decisions: Guidance adds +6.4 to +13.3 points on the least confident fifth of questions and at most +0.4 on the most confident fifth. Only 5.8%–14.9% of answers flip, and net gains come strictly from resolving close decisions.

Practical Implications

• Training-free: No auxiliary models, modified prompts, or external training required.
• Architecture-agnostic: Works across recurrent training objectives (supervised readouts, sampled depths, frozen-model retrofits).
• Compute-adaptive: Operates at full depth for maximum accuracy or half depth for compute savings.
• Overhead flexibility: Choose between one extra output pass (Logits) or zero overhead (Hidden).

Strength Tuning Insights

• Multiple-choice scoring: Tolerates broad strength band, peaking near $\omega = 0.5$.
• Autoregressive generation: Requires narrower window ($\omega \in [0.2, 0.3]$) as early token shifts compound.
• Adaptive margin gating: Maintains positive gains across a broad range of caps ($\omega_{\max} \in [0.5, 1.0]$) where fixed strength overshoots.
• Moving toward the weaker reference ($\omega < 0$) universally degrades accuracy.

---

Conclusion

LoopCD transforms the inherently available weak-to-strong prediction pairs in Looped Transformers into effective guidance signals for contrastive decoding. Key takeaways:

1. At full depth: Consistent gains across four model families, with mean pass@1 improvements of 5–7 points on mathematical reasoning and positive gains on every multiple-choice and generation suite.

2. At half depth: Matches or exceeds full-depth unguided baselines in six settings, saving 22.5%–48.2% of forward FLOPs.

3. Mechanistic account: Recurrence settles predictions early, the first iteration records the direction of movement, and continuing that movement re-ranks close decisions toward the option recurrence was raising.

Future Directions

• Models trained with intermediate predictions explicitly supervised (like Ouro) may make the weak-strong pair even more useful.
• The interaction between reference depth selection and guidance strength for hidden-space contrasts (particularly for models with stochastic initializations like Huginn) warrants further investigation.
• Extending the adaptive margin-gating rule to hidden-space guidance could further stabilize generation.