Summary (Overview)

• Loopie Series: Two looped Mixture-of-Experts (MoE) language models – Loopie‑20B‑A2B (20B total, 2B active) and Loopie‑6B‑A0.6B (6B total, 0.6B active) – that achieve state-of-the-art performance under a matched pre‑training compute budget.
• Compute‑matched scaling: The Loopie Recipe jointly optimises stored width, stored depth, and recurrent depth so that the looped model’s per‑step wall‑clock time matches a larger vanilla baseline, converting memory savings into additional model capacity.
• Layer‑loop recurrence: Instead of repeating the entire model stack (model‑loop), Loopie applies each layer locally twice before passing to the next layer, yielding better scaling, infrastructure efficiency, and natural parameter‑sharing.
• Post‑training with Supervised Pre‑Training (SPT): A novel large‑scale supervised training stage (2T tokens, batch size 1024, sequence length 131K) that improves both reasoning and general knowledge without catastrophic forgetting, followed by reinforcement learning (GSPO/DAPO) on math and code.
• Gold‑medal performance: Without external tools, Loopie‑20B‑A2B achieves perfect scores on 2025 IMO (35/42) and surpasses the gold threshold on 2025 IPhO (20.3/19.7), using only 3.5T pre‑training tokens – less than one seventh of comparable models.

Introduction and Theoretical Foundation

Looped Transformers (Universal Transformers, Dehghani et al., 2019) reuse the same parameters across multiple computational steps, offering an alternative to stacking distinct layers. While prior work (e.g., Ouro, Huginn) showed parameter efficiency, a fundamental issue remained: looping a model $N$ times multiplies pre‑training compute by $N$, so a looped model must be compared against a vanilla model trained with the same total FLOPs, not just the same stored parameter count. The paper asks:

Can looped Transformers match or exceed vanilla Transformers under the same pre‑training compute budget?

Theoretical motivation comes from scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) and work on recurrence as an inductive bias for iterative refinement, in‑context learning, and algorithmic reasoning. The authors argue that looped models can be competitive only when the recurrent‑depth‑width trade‑off is carefully chosen, leading to the Loopie Recipe.

Methodology

Architecture

Loopie follows the Qwen3‑MoE backbone but replaces the standard model‑loop (whole stack repeated) with layer‑loop:

• Model‑loop: $L_1 \to L_2 \to L_3 \to L_1 \to L_2 \to L_3$
• Layer‑loop: $L_1 \to L_1 \to L_2 \to L_2 \to L_3 \to L_3$

Layer‑loop is chosen for better empirical scaling, infrastructure‑friendly execution (reduced activation memory, simpler pipeline parallelism), and a more local parameter‑sharing pattern.

Loopie Recipe

Given a non‑recurrent reference model (e.g., Qwen3‑30B‑A3B with $D_0=2048$, $L_0=48$, $R_0=1$):

1. Construct a recurrent seed: Halve stored layers, set $R=2$: $L=24$, $D=2048$, $R=2$. This preserves the leading‑order compute $C \propto LD^2 R$ but halves activation memory $M_{\text{act}} \propto DL$.
2. Reinvest memory savings: The freed memory allows doubling the per‑device microbatch size $b$, reducing gradient‑accumulation steps $g$ proportionally, which increases hardware utilisation.
3. Sweep capacity: Search $(D, L)$ pairs around the seed that support the doubled microbatch and match the reference’s measured optimizer‑step time. The chosen configuration is $D_1=2304$, $L_1=27$, $R_1=2$, defining Loopie‑20B‑A2B. The compute proxy is: $$\hat{C} = \frac{LRD^2}{48 \cdot 2048^2} \approx 1.424$$ and the activation memory proxy: $$\hat{M}_{\text{act}} = \frac{DL}{48 \cdot 2048} \approx 0.633$$

Pre‑Training

Two stages:

• Stage 1: 2.28T tokens (4 epochs on Nemotron‑CC‑v2‑HQ) with a warmup‑then‑stable learning rate schedule.
• Stage 2: 1.26T high‑quality tokens (SFT‑style, STEM, code, math, synthetic web) with constant learning rate.

Post‑Training

1. Supervised Pre‑Training (SPT): 2T tokens of instruction‑following, reasoning, code, and math data, with a global batch size of 1024 and sequence length 131K. Only supervised target tokens contribute to the loss, but the scale is that of pre‑training. This avoids catastrophic forgetting and improves both reasoning and general knowledge.
2. Reinforcement Learning: Based on Group Sequence Policy Optimization (GSPO) with asymmetric clipping (DAPO). The policy objective is: $$J_{\text{RL}}^{\text{GSPO}}(\theta) = \mathbb{E}_{q, \{o_i\}} \left[ \frac{1}{G} \sum_{i=1}^G \min\left( s_i(\theta) \hat{A}_i, \; \text{clip}(s_i(\theta), 1-\epsilon_{\text{low}}, 1+\epsilon_{\text{high}}) \hat{A}_i \right) \right]$$ where $s_i(\theta)$ is the length‑normalised sequence‑level importance ratio. Training is done on math and code corpora in two context‑length stages (32K and 64K).

Empirical Validation / Results

Compute‑Matched Comparison

Table 1 (partial): Candidate configurations from the Loopie Recipe.

Figure 3 shows that Loopie‑20B‑A2B overtakes Qwen3‑30B‑A3B after ~600B tokens and maintains a ≈2‑point advantage on downstream average, despite having only 2/3 the active parameters and 1/7 the pre‑training tokens.

Scaling Ladder

Across four model sizes (0.15B to 1B active parameters), Loopie consistently outperforms compute‑matched vanilla baselines (Figure 4), with the gap increasing from +0.6% to +2.2%.

Ablations

Layer‑loop vs. model‑loop: Figure 2 shows layer‑loop surpasses model‑loop after ~1.2T tokens. Loop count: Figure 6 demonstrates that two loop steps ($R=2$) provide the best trade‑off; larger $R$ yields diminishing returns.

Post‑Training Performance

Table 3 (simplified): Comparison of Loopie‑20B‑A2B‑Thinking with similarly sized reasoning models.

Loopie, trained on only 3.5T tokens (vs. 25T for Nemotron models), matches or exceeds them on most benchmarks, demonstrating exceptional token efficiency.

IMO and IPhO 2025

Using a generate‑verify‑refine test‑time scaling pipeline, Loopie‑20B‑A2B achieved:

• IMO 2025: 35/42 points (gold threshold: 35)
• IPhO 2025: 20.3/19.7 points (gold threshold: 19.7)

Theoretical and Practical Implications

• Recurrence as a compute‑matched scaling axis: The Loopie Recipe shows that looped Transformers can be more than parameter‑efficient; they can be compute‑efficient when designed with hardware‑aware trade‑offs. This challenges the conventional view that recurrence is solely a parameter‑saving device.
• Layer‑loop vs. model‑loop: The paper demonstrates that the ordering of recurrent computation matters. Layer‑loop provides a more coherent parameter‑sharing pattern and better infrastructure efficiency, enabling scaling to large MoE models.
• Supervised Pre‑Training (SPT): This novel training regime bridges the gap between SFT and pre‑training, allowing large‑scale supervised learning without catastrophic forgetting. It has practical implications for post‑training of large language models.
• Gold‑medal reasoning without tools: Loopie’s success on IMO and IPhO shows that looped architectures, combined with appropriate post‑training, can achieve frontier‑level reasoning despite much smaller pre‑training budgets.
• Limitations: The paper focuses on math/code reasoning; future work should explore broader capabilities (scientific QA, instruction following, agentic tasks). Systematic studies of inference‑time compute and interactions with other architectural advances are also needed.

Conclusion

Loopie demonstrates that looped MoE Transformers, when designed with the compute‑matched Loopie Recipe and advanced post‑training (SPT + RL), can outperform substantially larger vanilla Transformers under the same pre‑training compute budget. The models achieve gold‑medal performance on 2025 IMO and IPhO problems without external tools, using only a fraction of the training tokens of comparable models. The work establishes recurrent depth as a practical third scaling axis alongside parameter count and token count, and introduces SPT as a promising method for large‑scale supervised training.