Full text not available for this paper
Summary (Overview)
- UpLoop is a pretraining schedule that trains looped Transformers while exactly matching a dense baseline in final parameter count, total training tokens, and training FLOPs.
- The method combines progressive depth growth (copying learned layers) with shared-weight middle-block looping, reallocating computation saved early in training to support repeated execution later.
- Across three model scales (370M, 760M, 1.3B), UpLoop consistently improves held-out language modeling (NLL) and downstream performance—e.g., average score across 22 tasks increases from 22.99 to 27.52 at 1.3B parameters.
- Gains extend to mathematical reasoning after supervised fine-tuning on GSM8K (zero-shot accuracy from 8.72% to 12.05% at 1.3B).
- Analyses reveal more aligned gradients across shared calls, functional differentiation among copied layers, and improved contextual retrieval.
Introduction and Theoretical Foundation
Motivation: Looped language models strengthen reasoning through repeated computation with shared weights, increasing execution depth without adding independent parameters. However, each additional pass increases training FLOPs, making these gains expensive.
Core research question: Can we train a looped language model that outperforms a standard dense model with the same model parameters, training tokens, and training FLOPs?
Key insight: A fixed total compute budget constrains accumulated training cost while allowing different stages to use different execution depths. By allocating less computation to early tokens and progressively increasing depth, the computation saved early can support repeated execution in the final stage.
Theoretical foundation: The paper builds on two lines of work:
- Looped/recursive language models (Universal Transformers, recent work on iterative computation at fixed parameter counts)
- Depth growth/upcycling (initializing larger models by stacking or duplicating layers to save training FLOPs)
Formulation: A standard Transformer applies L layers sequentially:
A looped Transformer repeatedly applies a shared block :
The goal is to train a looped model with the same total training tokens and training FLOPs as a standard dense model.
Methodology
Four-Stage Training Schedule
UpLoop follows a four-stage schedule with execution depths:
where is the dense baseline's depth, is the shared middle block's depth, and is its repetition count in the final stage.
1. Progressive Depth Growth
- First transition (): Stack the Transformer layers twice to initialize an -layer model. Each copy inherits both layer weights and Adam states; duplicated layers are then optimized independently.
- Second transition (): Partition the model into three blocks with depths , and initialize:
where each is a separate copy of .
2. Middle-Block Looping
Select consecutive middle layers as a shared block . The forward pass becomes:
where denotes successive calls with shared weights, while head and tail are each executed once.
3. Compute-Matched Schedule
Total token count and training FLOPs:
where is the analytical training FLOPs per token at execution depth . The stage allocation yields and , exactly matching the dense baseline.
Stage transitions occur after approximately 9.7%, 23.0%, and 44.3% of total training tokens. The learning-rate schedule follows cumulative tokens without restarting at stage boundaries.
Empirical Validation / Results
Experimental Setup
- Models: Decoder-only Transformers based on OLMo 2 Ladder, trained on FineWeb-Edu at 100 tokens per parameter (5× Chinchilla budget)
- Scales: 370M, 760M, 1.3B non-embedding parameters
- Baselines: Vanilla (dense), Loop from scratch, Loop with dense initialization, Dense-20 (matched depth), LIDAS (growth-only)
Language Modeling Results
UpLoop achieves lower NLL at all three scales:
- 370M: NLL reduced from 2.78 to 2.74
- 1.3B: NLL decreased from 2.43 to 2.40
Zero-shot Commonsense Reasoning (Table 1)
| Scale | Model | OBQA | WinoG. | ARC-c | ARC-e | HellaSwag | SIQA | PIQA | BoolQ | Average |
|---|---|---|---|---|---|---|---|---|---|---|
| 370M | Vanilla | 20.80 | 52.01 | 28.84 | 62.75 | 35.02 | 38.69 | 67.90 | 57.37 | 48.43 |
| 370M | UpLoop | 23.80 | 53.28 | 29.01 | 63.22 | 36.02 | 40.17 | 68.66 | 61.62 | 49.78 |
| 760M | Vanilla | 26.80 | 55.01 | 32.17 | 68.06 | 39.67 | 39.66 | 70.78 | 58.35 | 51.96 |
| 760M | UpLoop | 26.20 | 55.17 | 32.85 | 69.57 | 41.58 | 42.07 | 70.89 | 62.51 | 53.52 |
| 1.3B | Vanilla | 28.40 | 56.51 | 37.54 | 73.36 | 43.21 | 40.84 | 73.12 | 59.76 | 54.91 |
| 1.3B | UpLoop | 30.40 | 57.85 | 37.20 | 73.23 | 44.84 | 42.37 | 73.72 | 66.06 | 56.47 |
Reasoning and Knowledge Evaluation (22 tasks)
Average scores improve from 18.95 → 20.90 (370M), 21.38 → 24.82 (760M), and 22.99 → 27.52 (1.3B). Largest gains on reasoning primitives:
- VarAssign-math: 58.10% → 91.10% at 1.3B
- VarAssign-code: 66.60% → 91.20% at 1.3B
- CoQA F1: 38.85 → 44.38 at 1.3B
Comparison with Alternative Strategies (370M, Table 3)
| Model | Params | Tokens | FLOPs | NLL ↓ |
|---|---|---|---|---|
| Vanilla | 371M | 1.00× | 1.00× | 2.78 |
| Loop from scratch | 371M | 1.00× | 1.20× | 2.76 |
| Loop w/ dense init | 371M | 1.00× | 1.11× | 2.76 |
| Dense-20 (Iso-FLOPs) | 438M | 0.83× | 1.00× | 2.76 |
| Dense-20 (Iso-Tokens) | 438M | 1.00× | 1.20× | 2.74 |
| LIDAS (Iso-FLOPs) | 371M | 1.20× | 1.00× | 2.75 |
| UpLoop | 371M | 1.00× | 1.00× | 2.74 |
Supervised Fine-tuning on GSM8K (Table 4)
| Model | Params | Zero-shot (%) | Five-shot (%) |
|---|---|---|---|
| Vanilla | 1.3B | 8.72 | 5.46 |
| UpLoop | 1.3B | 12.05 | 8.11 |
Further Analysis
Gradient alignment: UpLoop achieves mean gradient cosine of 0.586 across shared calls, vs. 0.510 (Loop with dense init) and 0.374 (Loop from scratch).
Perturbation propagation (Table 5):
| Model | Normal 2 calls ↓ | Extended 8 calls ↓ |
|---|---|---|
| UpLoop | 0.867 | 1.906 |
| Loop w/ dense init | 0.933 | 2.965 |
| Loop from scratch | 0.923 | 2.644 |
Second-pass reading (Table 6): Blocking target-value attention during the second call reduces accuracy by 8.6 pp (760M) and 14.5 pp (1.3B), confirming the second pass supports answer formation.
Theoretical and Practical Implications
Theoretical implications:
- Demonstrates that compute reallocation across training stages is an effective alternative to simply adding FLOPs for looped models
- Shows that staged depth growth creates layers with functional differentiation while retaining shared ancestry (weight cosine similarity of 0.383 for same-ancestor pairs vs. 0.024 for unrelated pairs)
- Provides evidence that looping strengthens reasoning through improved contextual retrieval and more robust state propagation
Practical implications:
- Enables training looped models without any additional pretraining compute—a strict improvement over prior approaches
- The method is scale-agnostic, with consistent gains from 370M to 1.3B parameters
- Gains transfer to downstream fine-tuning, making it useful for task-specific deployment
- The schedule is simple and reproducible (fixed transition points, no architecture changes beyond the looped core)
Conclusion
UpLoop demonstrates that looped Transformers can be trained within a standard dense model's parameter, token, and FLOP budgets by combining progressive depth growth with shared-weight looping. The computation saved during early shallow training supports repeated execution later, yielding consistent improvements in language modeling and downstream performance across model scales.
Key takeaways:
- Reallocating computation across training stages is an effective strategy for training looped models at no extra cost
- Gains are most pronounced in reasoning tasks (variable assignment, math word problems) and contextual retrieval
- The method produces more aligned gradients across shared calls and functionally differentiated copied layers
Future directions implied by this work include exploring optimal stage allocation strategies, applying the schedule to larger scales, and investigating the interaction between looping and other efficiency techniques (e.g., sparsity, mixture-of-experts).
Related papers
- The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search
A naive LLM critic can drive AutoML search below the no-critic random-search floor on 7 of 8 tabular datasets, with losses up to 0.245 AUC.
- Do Halting Heads Really Save Compute in Looped Language Models?
Halting heads in looped language models save far less compute than reported because mixture-trained heads rarely stop tokens and batching overhead eliminates most remaining gains.
- Rethinking Fixed-Horizon Latent Reasoning
Adaptive latent inference that stops reasoning when progress stalls improves accuracy by up to 7.2 points while cutting computation by 29%, as completing fixed latent horizons can degrade performance.