Full text not available for this paper

Summary (Overview)

  • UpLoop is a pretraining schedule that trains looped Transformers while exactly matching a dense baseline in final parameter count, total training tokens, and training FLOPs.
  • The method combines progressive depth growth (copying learned layers) with shared-weight middle-block looping, reallocating computation saved early in training to support repeated execution later.
  • Across three model scales (370M, 760M, 1.3B), UpLoop consistently improves held-out language modeling (NLL) and downstream performance—e.g., average score across 22 tasks increases from 22.99 to 27.52 at 1.3B parameters.
  • Gains extend to mathematical reasoning after supervised fine-tuning on GSM8K (zero-shot accuracy from 8.72% to 12.05% at 1.3B).
  • Analyses reveal more aligned gradients across shared calls, functional differentiation among copied layers, and improved contextual retrieval.

Introduction and Theoretical Foundation

Motivation: Looped language models strengthen reasoning through repeated computation with shared weights, increasing execution depth without adding independent parameters. However, each additional pass increases training FLOPs, making these gains expensive.

Core research question: Can we train a looped language model that outperforms a standard dense model with the same model parameters, training tokens, and training FLOPs?

Key insight: A fixed total compute budget constrains accumulated training cost while allowing different stages to use different execution depths. By allocating less computation to early tokens and progressively increasing depth, the computation saved early can support repeated execution in the final stage.

Theoretical foundation: The paper builds on two lines of work:

  1. Looped/recursive language models (Universal Transformers, recent work on iterative computation at fixed parameter counts)
  2. Depth growth/upcycling (initializing larger models by stacking or duplicating layers to save training FLOPs)

Formulation: A standard Transformer applies L layers sequentially:

h(i)=Fθi(h(i−1)),i=1,…,Lh^{(i)} = F_{\theta_i}(h^{(i-1)}), \quad i = 1, \ldots, L

A looped Transformer repeatedly applies a shared block BϕB_\phi:

z(k)=Bϕ(z(k−1)),k=1,…,rz^{(k)} = B_\phi(z^{(k-1)}), \quad k = 1, \ldots, r

The goal is to train a looped model with the same total training tokens D0D_0 and training FLOPs C0C_0 as a standard dense model.


Methodology

Four-Stage Training Schedule

UpLoop follows a four-stage schedule with execution depths:

L4→L2→L→L+(r−1)m\frac{L}{4} \rightarrow \frac{L}{2} \rightarrow L \rightarrow L + (r-1)m

where LL is the dense baseline's depth, mm is the shared middle block's depth, and rr is its repetition count in the final stage.

1. Progressive Depth Growth

  • First transition (L/4→L/2L/4 \rightarrow L/2): Stack the L/4L/4 Transformer layers twice to initialize an L/2L/2-layer model. Each copy inherits both layer weights and Adam states; duplicated layers are then optimized independently.
  • Second transition (L/2→LL/2 \rightarrow L): Partition the model into three blocks (ΘH,ΘB,ΘT)(\Theta_H, \Theta_B, \Theta_T) with depths (L/8,L/4,L/8)(L/8, L/4, L/8), and initialize:
ΘL←(ΘH,ΘB[1],ΘB[2],ΘB[3],ΘT)\Theta_L \leftarrow \left(\Theta_H, \Theta_B^{[1]}, \Theta_B^{[2]}, \Theta_B^{[3]}, \Theta_T\right)

where each ΘB[j]\Theta_B^{[j]} is a separate copy of ΘB\Theta_B.

2. Middle-Block Looping

Select mm consecutive middle layers as a shared block BϕB_\phi. The forward pass becomes:

hout=FT(Bϕ∘r(FH(hin)))h_{\text{out}} = F_T\left(B_\phi^{\circ r}(F_H(h_{\text{in}}))\right)

where Bϕ∘rB_\phi^{\circ r} denotes rr successive calls with shared weights, while head and tail are each executed once.

3. Compute-Matched Schedule

Total token count and training FLOPs:

D=∑j=14Dj,C=∑j=14Djf(dj)D = \sum_{j=1}^{4} D_j, \quad C = \sum_{j=1}^{4} D_j f(d_j)

where f(d)f(d) is the analytical training FLOPs per token at execution depth dd. The stage allocation yields D=D0D = D_0 and C=C0C = C_0, exactly matching the dense baseline.

Stage transitions occur after approximately 9.7%, 23.0%, and 44.3% of total training tokens. The learning-rate schedule follows cumulative tokens without restarting at stage boundaries.


Empirical Validation / Results

Experimental Setup

  • Models: Decoder-only Transformers based on OLMo 2 Ladder, trained on FineWeb-Edu at 100 tokens per parameter (5× Chinchilla budget)
  • Scales: 370M, 760M, 1.3B non-embedding parameters
  • Baselines: Vanilla (dense), Loop from scratch, Loop with dense initialization, Dense-20 (matched depth), LIDAS (growth-only)

Language Modeling Results

UpLoop achieves lower NLL at all three scales:

  • 370M: NLL reduced from 2.78 to 2.74
  • 1.3B: NLL decreased from 2.43 to 2.40

Zero-shot Commonsense Reasoning (Table 1)

ScaleModelOBQAWinoG.ARC-cARC-eHellaSwagSIQAPIQABoolQAverage
370MVanilla20.8052.0128.8462.7535.0238.6967.9057.3748.43
370MUpLoop23.8053.2829.0163.2236.0240.1768.6661.6249.78
760MVanilla26.8055.0132.1768.0639.6739.6670.7858.3551.96
760MUpLoop26.2055.1732.8569.5741.5842.0770.8962.5153.52
1.3BVanilla28.4056.5137.5473.3643.2140.8473.1259.7654.91
1.3BUpLoop30.4057.8537.2073.2344.8442.3773.7266.0656.47

Reasoning and Knowledge Evaluation (22 tasks)

Average scores improve from 18.95 → 20.90 (370M), 21.38 → 24.82 (760M), and 22.99 → 27.52 (1.3B). Largest gains on reasoning primitives:

  • VarAssign-math: 58.10% → 91.10% at 1.3B
  • VarAssign-code: 66.60% → 91.20% at 1.3B
  • CoQA F1: 38.85 → 44.38 at 1.3B

Comparison with Alternative Strategies (370M, Table 3)

ModelParamsTokensFLOPsNLL ↓
Vanilla371M1.00×1.00×2.78
Loop from scratch371M1.00×1.20×2.76
Loop w/ dense init371M1.00×1.11×2.76
Dense-20 (Iso-FLOPs)438M0.83×1.00×2.76
Dense-20 (Iso-Tokens)438M1.00×1.20×2.74
LIDAS (Iso-FLOPs)371M1.20×1.00×2.75
UpLoop371M1.00×1.00×2.74

Supervised Fine-tuning on GSM8K (Table 4)

ModelParamsZero-shot (%)Five-shot (%)
Vanilla1.3B8.725.46
UpLoop1.3B12.058.11

Further Analysis

Gradient alignment: UpLoop achieves mean gradient cosine of 0.586 across shared calls, vs. 0.510 (Loop with dense init) and 0.374 (Loop from scratch).

Perturbation propagation (Table 5):

ModelNormal 2 calls ↓Extended 8 calls ↓
UpLoop0.8671.906
Loop w/ dense init0.9332.965
Loop from scratch0.9232.644

Second-pass reading (Table 6): Blocking target-value attention during the second call reduces accuracy by 8.6 pp (760M) and 14.5 pp (1.3B), confirming the second pass supports answer formation.


Theoretical and Practical Implications

Theoretical implications:

  • Demonstrates that compute reallocation across training stages is an effective alternative to simply adding FLOPs for looped models
  • Shows that staged depth growth creates layers with functional differentiation while retaining shared ancestry (weight cosine similarity of 0.383 for same-ancestor pairs vs. 0.024 for unrelated pairs)
  • Provides evidence that looping strengthens reasoning through improved contextual retrieval and more robust state propagation

Practical implications:

  • Enables training looped models without any additional pretraining compute—a strict improvement over prior approaches
  • The method is scale-agnostic, with consistent gains from 370M to 1.3B parameters
  • Gains transfer to downstream fine-tuning, making it useful for task-specific deployment
  • The schedule is simple and reproducible (fixed transition points, no architecture changes beyond the looped core)

Conclusion

UpLoop demonstrates that looped Transformers can be trained within a standard dense model's parameter, token, and FLOP budgets by combining progressive depth growth with shared-weight looping. The computation saved during early shallow training supports repeated execution later, yielding consistent improvements in language modeling and downstream performance across model scales.

Key takeaways:

  1. Reallocating computation across training stages is an effective strategy for training looped models at no extra cost
  2. Gains are most pronounced in reasoning tasks (variable assignment, math word problems) and contextual retrieval
  3. The method produces more aligned gradients across shared calls and functionally differentiated copied layers

Future directions implied by this work include exploring optimal stage allocation strategies, applying the schedule to larger scales, and investigating the interaction between looping and other efficiency techniques (e.g., sparsity, mixture-of-experts).

Related papers