# UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute

> UpLoop trains looped Transformers with identical parameters, tokens, and FLOPs to dense baselines, improving reasoning and language modeling across scales via progressive depth growth and shared-weight looping.

- **Source:** [arXiv](https://arxiv.org/abs/2610.41720)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/Vv4mO2
- **Whiteboard:** https://picx.dev/p/Vv4mO2/image

## Summary

## Summary (Overview)

- **UpLoop** is a pretraining schedule that trains looped Transformers while exactly matching a dense baseline in final parameter count, total training tokens, and training FLOPs.
- The method combines **progressive depth growth** (copying learned layers) with **shared-weight middle-block looping**, reallocating computation saved early in training to support repeated execution later.
- Across three model scales (370M, 760M, 1.3B), UpLoop consistently improves held-out language modeling (NLL) and downstream performance—e.g., average score across 22 tasks increases from **22.99 to 27.52** at 1.3B parameters.
- Gains extend to mathematical reasoning after supervised fine-tuning on GSM8K (zero-shot accuracy from 8.72% to 12.05% at 1.3B).
- Analyses reveal more aligned gradients across shared calls, functional differentiation among copied layers, and improved contextual retrieval.

---

## Introduction and Theoretical Foundation

**Motivation:** Looped language models strengthen reasoning through repeated computation with shared weights, increasing execution depth without adding independent parameters. However, each additional pass increases training FLOPs, making these gains expensive.

**Core research question:** Can we train a looped language model that outperforms a standard dense model with the **same** model parameters, training tokens, and training FLOPs?

**Key insight:** A fixed total compute budget constrains accumulated training cost while allowing different stages to use different execution depths. By allocating less computation to early tokens and progressively increasing depth, the computation saved early can support repeated execution in the final stage.

**Theoretical foundation:** The paper builds on two lines of work:
1. **Looped/recursive language models** (Universal Transformers, recent work on iterative computation at fixed parameter counts)
2. **Depth growth/upcycling** (initializing larger models by stacking or duplicating layers to save training FLOPs)

**Formulation:** A standard Transformer applies L layers sequentially:

$$h^{(i)} = F_{\theta_i}(h^{(i-1)}), \quad i = 1, \ldots, L$$

A looped Transformer repeatedly applies a shared block $B_\phi$:

$$z^{(k)} = B_\phi(z^{(k-1)}), \quad k = 1, \ldots, r$$

The goal is to train a looped model with the same total training tokens $D_0$ and training FLOPs $C_0$ as a standard dense model.

---

## Methodology

### Four-Stage Training Schedule

UpLoop follows a four-stage schedule with execution depths:

$$\frac{L}{4} \rightarrow \frac{L}{2} \rightarrow L \rightarrow L + (r-1)m$$

where $L$ is the dense baseline's depth, $m$ is the shared middle block's depth, and $r$ is its repetition count in the final stage.

### 1. Progressive Depth Growth

- **First transition ($L/4 \rightarrow L/2$):** Stack the $L/4$ Transformer layers twice to initialize an $L/2$-layer model. Each copy inherits both layer weights and Adam states; duplicated layers are then optimized independently.
- **Second transition ($L/2 \rightarrow L$):** Partition the model into three blocks $(\Theta_H, \Theta_B, \Theta_T)$ with depths $(L/8, L/4, L/8)$, and initialize:

$$\Theta_L \leftarrow \left(\Theta_H, \Theta_B^{[1]}, \Theta_B^{[2]}, \Theta_B^{[3]}, \Theta_T\right)$$

where each $\Theta_B^{[j]}$ is a separate copy of $\Theta_B$.

### 2. Middle-Block Looping

Select $m$ consecutive middle layers as a shared block $B_\phi$. The forward pass becomes:

$$h_{\text{out}} = F_T\left(B_\phi^{\circ r}(F_H(h_{\text{in}}))\right)$$

where $B_\phi^{\circ r}$ denotes $r$ successive calls with shared weights, while head and tail are each executed once.

### 3. Compute-Matched Schedule

Total token count and training FLOPs:

$$D = \sum_{j=1}^{4} D_j, \quad C = \sum_{j=1}^{4} D_j f(d_j)$$

where $f(d)$ is the analytical training FLOPs per token at execution depth $d$. The stage allocation yields $D = D_0$ and $C = C_0$, exactly matching the dense baseline.

**Stage transitions** occur after approximately 9.7%, 23.0%, and 44.3% of total training tokens. The learning-rate schedule follows cumulative tokens without restarting at stage boundaries.

---

## Empirical Validation / Results

### Experimental Setup

- **Models:** Decoder-only Transformers based on OLMo 2 Ladder, trained on FineWeb-Edu at 100 tokens per parameter (5× Chinchilla budget)
- **Scales:** 370M, 760M, 1.3B non-embedding parameters
- **Baselines:** Vanilla (dense), Loop from scratch, Loop with dense initialization, Dense-20 (matched depth), LIDAS (growth-only)

### Language Modeling Results

UpLoop achieves lower NLL at all three scales:
- 370M: NLL reduced from 2.78 to 2.74
- 1.3B: NLL decreased from 2.43 to 2.40

### Zero-shot Commonsense Reasoning (Table 1)

| Scale | Model | OBQA | WinoG. | ARC-c | ARC-e | HellaSwag | SIQA | PIQA | BoolQ | **Average** |
|-------|-------|------|--------|-------|-------|-----------|------|------|-------|-------------|
| 370M | Vanilla | 20.80 | 52.01 | 28.84 | 62.75 | 35.02 | 38.69 | 67.90 | 57.37 | 48.43 |
| 370M | UpLoop | 23.80 | 53.28 | 29.01 | 63.22 | 36.02 | 40.17 | 68.66 | 61.62 | **49.78** |
| 760M | Vanilla | 26.80 | 55.01 | 32.17 | 68.06 | 39.67 | 39.66 | 70.78 | 58.35 | 51.96 |
| 760M | UpLoop | 26.20 | 55.17 | 32.85 | 69.57 | 41.58 | 42.07 | 70.89 | 62.51 | **53.52** |
| 1.3B | Vanilla | 28.40 | 56.51 | 37.54 | 73.36 | 43.21 | 40.84 | 73.12 | 59.76 | 54.91 |
| 1.3B | UpLoop | 30.40 | 57.85 | 37.20 | 73.23 | 44.84 | 42.37 | 73.72 | 66.06 | **56.47** |

### Reasoning and Knowledge Evaluation (22 tasks)

Average scores improve from 18.95 → 20.90 (370M), 21.38 → 24.82 (760M), and 22.99 → 27.52 (1.3B). Largest gains on reasoning primitives:
- **VarAssign-math:** 58.10% → 91.10% at 1.3B
- **VarAssign-code:** 66.60% → 91.20% at 1.3B
- **CoQA F1:** 38.85 → 44.38 at 1.3B

### Comparison with Alternative Strategies (370M, Table 3)

| Model | Params | Tokens | FLOPs | NLL ↓ |
|-------|--------|--------|-------|-------|
| Vanilla | 371M | 1.00× | 1.00× | 2.78 |
| Loop from scratch | 371M | 1.00× | 1.20× | 2.76 |
| Loop w/ dense init | 371M | 1.00× | 1.11× | 2.76 |
| Dense-20 (Iso-FLOPs) | 438M | 0.83× | 1.00× | 2.76 |
| Dense-20 (Iso-Tokens) | 438M | 1.00× | 1.20× | 2.74 |
| LIDAS (Iso-FLOPs) | 371M | 1.20× | 1.00× | 2.75 |
| **UpLoop** | **371M** | **1.00×** | **1.00×** | **2.74** |

### Supervised Fine-tuning on GSM8K (Table 4)

| Model | Params | Zero-shot (%) | Five-shot (%) |
|-------|--------|---------------|---------------|
| Vanilla | 1.3B | 8.72 | 5.46 |
| **UpLoop** | **1.3B** | **12.05** | **8.11** |

### Further Analysis

**Gradient alignment:** UpLoop achieves mean gradient cosine of **0.586** across shared calls, vs. 0.510 (Loop with dense init) and 0.374 (Loop from scratch).

**Perturbation propagation (Table 5):**

| Model | Normal 2 calls ↓ | Extended 8 calls ↓ |
|-------|-------------------|---------------------|
| UpLoop | 0.867 | 1.906 |
| Loop w/ dense init | 0.933 | 2.965 |
| Loop from scratch | 0.923 | 2.644 |

**Second-pass reading (Table 6):** Blocking target-value attention during the second call reduces accuracy by 8.6 pp (760M) and 14.5 pp (1.3B), confirming the second pass supports answer formation.

---

## Theoretical and Practical Implications

**Theoretical implications:**
- Demonstrates that **compute reallocation across training stages** is an effective alternative to simply adding FLOPs for looped models
- Shows that staged depth growth creates layers with **functional differentiation** while retaining shared ancestry (weight cosine similarity of 0.383 for same-ancestor pairs vs. 0.024 for unrelated pairs)
- Provides evidence that **looping strengthens reasoning** through improved contextual retrieval and more robust state propagation

**Practical implications:**
- Enables training looped models **without any additional pretraining compute**—a strict improvement over prior approaches
- The method is **scale-agnostic**, with consistent gains from 370M to 1.3B parameters
- Gains **transfer to downstream fine-tuning**, making it useful for task-specific deployment
- The schedule is **simple and reproducible** (fixed transition points, no architecture changes beyond the looped core)

---

## Conclusion

UpLoop demonstrates that looped Transformers can be trained within a standard dense model's parameter, token, and FLOP budgets by combining progressive depth growth with shared-weight looping. The computation saved during early shallow training supports repeated execution later, yielding consistent improvements in language modeling and downstream performance across model scales.

**Key takeaways:**
1. Reallocating computation across training stages is an effective strategy for training looped models at no extra cost
2. Gains are most pronounced in **reasoning tasks** (variable assignment, math word problems) and **contextual retrieval**
3. The method produces more **aligned gradients** across shared calls and **functionally differentiated** copied layers

**Future directions** implied by this work include exploring optimal stage allocation strategies, applying the schedule to larger scales, and investigating the interaction between looping and other efficiency techniques (e.g., sparsity, mixture-of-experts).

---

_Markdown view of https://picx.dev/p/Vv4mO2, served by PicX — AI-generated visual whiteboard summaries of research papers._
