# InfuseLoop: Cross-Trajectory Latent Injection for Scalable Parallel Looped Transformers

> InfuseLoop injects cross-trajectory latent states into parallel looped transformers, eliminating training spikes and enabling quality gains at 8 loops with minimal overhead.

- **Source:** [arXiv](https://arxiv.org/abs/2611.22426)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/doqKCM
- **Whiteboard:** https://picx.dev/p/doqKCM/image

## Summary

## Summary (Overview)

- **Core Problem**: Parallel Looped Transformers (PLTs) suffer from training-loss spikes and diminishing quality gains as loop depth increases (beyond 2 loops), limiting their scalability for test-time compute scaling.
- **Key Insight**: At each loop boundary, vanilla PLTs pass only the preceding in-trajectory state to the next loop, ignoring causally available cross-trajectory representations from overlapping prediction trajectories.
- **Proposed Solution**: **InfuseLoop**, a cross-trajectory latent injection mechanism that aggregates causally available hidden states from later prediction trajectories via lightweight attention residuals, preserving token-parallel training and concurrent loop execution.
- **Key Results**: At 8 loops, InfuseLoop reduces perplexity by 3.4–17.1% relative to vanilla PLTs and by 10.5–37.3% relative to the non-looped baseline, while improving downstream accuracy by 0.84–1.99 percentage points. Decoding overhead is only 0.3% over vanilla PLTs.
- **Efficiency**: Requires only $O(Rd)$ additional hidden-state storage and maintains the same logical KV-cache footprint as vanilla PLTs.

---

## Introduction and Theoretical Foundation

### Background: Looped Transformers (LTs)

Looped Transformers decouple computational depth from parameter count by repeatedly applying shared Transformer blocks. For a standard full-stack LT with K blocks $f_{\theta_1}, \ldots, f_{\theta_K}$ whose composition is $F_\theta = f_{\theta_K} \circ \cdots \circ f_{\theta_1}$, the model applies this backbone for R loops:

$$h^{(r)}_t = F_\theta\left(h^{(r-1)}_t\right), \quad r = 1, \ldots, R$$
$$x_{t+1} = \text{LM Head}\left(h^{(R)}_t\right)$$

The parameters $\theta = \{\theta_1, \ldots, \theta_K\}$ are shared across loops, so the model performs $K \times R$ Transformer-block applications with only K distinct blocks.

### The Problem: Parallel Looped Transformers (PLTs)

PLTs restructure dependencies across token positions and loop iterations to enable concurrent execution of different loop stages from multiple prediction trajectories. A trajectory predicting $x_{t+1}$ computes:

$$h^{(1)}_t = F_\theta(\text{Embed}(x_{t-R+1}))$$
$$h^{(r)}_t = F_\theta\left(h^{(r-1)}_t + \text{Embed}(x_{t-R+r})\right), \quad r = 2, \ldots, R$$

However, vanilla PLTs exhibit two critical problems:
1. **Training-loss spikes** even with as few as 4 loops
2. **Diminishing refinement gains** beyond 2 loops

### Key Theoretical Insight

The authors identify a **computational dependency gap**: at each loop boundary, a vanilla PLT passes only the previous output of the same prediction trajectory to the next loop. Yet the overlapping execution schedule produces completed states on other trajectories that are causally available but never used as direct recurrent inputs.

---

## Methodology

### 1. Computational Graph Analysis

For a receiving state $h^{(r)}_t$ at execution cycle $\tau(t, r) = t - R + r$, the admissible source set is:

$$\mathcal{A}_{t,r} = \left\{h^{(j)}_s : 1 \leq j < r, \; s + j < t + r\right\}$$

This set satisfies two constraints:
- **Causal availability**: $\tau(s, j) < \tau(t, r)$ — source completed in an earlier cycle
- **Training loop order**: $j < r$ — source from a shallower loop

### 2. InfuseLoop Architecture

**Source Selection**: InfuseLoop selects all shallower-loop outputs from the immediately preceding cycle:

$$\mathcal{S}_{t,r} = \left\{h^{(j)}_{t+r-j-1} : 1 \leq j < r\right\}$$

For example, $\mathcal{S}_{t,4} = \{h^{(3)}_t, h^{(2)}_{t+1}, h^{(1)}_{t+2}\}$. The source with $j = r-1$ is the in-trajectory base state, and the remaining $r-2$ sources are completed outputs of later trajectories.

**Attention-based Aggregation** (inspired by Attention Residuals):

$$k_{t,j \to r} = v_{t,j \to r} = \text{RMSNorm}\left(h^{(j)}_{t+r-j-1}\right)$$
$$e_{t,j \to r} = \frac{w^\top_r k_{t,j \to r}}{\sqrt{d}} + b_{r,j}$$
$$\alpha_{t,j \to r} = \frac{\exp(e_{t,j \to r})}{\sum_{i=1}^{r-1} \exp(e_{t,i \to r})}$$

The aggregated residual replaces the recurrent input:

$$m^{(r)}_t = \rho_r \sum_{j=1}^{r-1} \alpha_{t,j \to r} v_{t,j \to r}$$
$$h^{(r)}_t = F_\theta\left(\text{Embed}(x_{t-R+r}) + m^{(r)}_t\right), \quad r = 2, \ldots, R$$

where $\rho_r$ is a learned aggregation scale and $w_r \in \mathbb{R}^d$ is a learned pseudo-query for each receiving loop.

### 3. Training and Inference

**Training**: Loops are processed sequentially while all sequence positions within a loop are computed in parallel. The training algorithm uses sequence-level tensors ordered by execution cycle, with a right-shift operation to align previous-cycle states.

**Inference**: Each decoding cycle batches all active loops through the backbone (layer-synchronous G-SWA schedule). Only $O(Rd)$ additional hidden-state storage is needed per sequence, independent of sequence length.

---

## Empirical Validation / Results

### Training Stability

**Figure 4** shows that vanilla PLT exhibits pronounced training-loss spikes at both $R=4$ and $R=8$, while InfuseLoop eliminates these spikes and converges more smoothly.

### Main Results (6B tokens training)

**Table 1: Validation loss, perplexity, and downstream accuracy (%)**

| Loops | Model | Val. loss ↓ | Wiki2 | LAMBADA | MMLU | HellaSwag | ARC-E | ARC-C | PIQA | WG | OBQA | Avg. |
|-------|-------|------------|-------|----------|------|-----------|--------|-------|------|-----|------|------|
| R=1 | Single | 2.9941 | 39.72 | 67.02 | 24.58 | 36.44 | 49.92 | 28.58 | 63.38 | 51.70 | 33.00 | 41.09 |
| R=4 | LT | 2.9022 | 34.78 | 41.07 | 24.35 | 40.17 | 54.34 | 30.55 | 66.43 | 54.70 | 33.80 | 43.48 |
| | PLT | 2.9252 | 36.19 | 62.96 | 24.71 | 39.24 | 51.68 | 28.58 | 65.78 | 51.70 | 32.60 | 42.04 |
| | **InfuseLoop** | **2.9112** | **35.70** | **45.08** | 24.49 | 39.49 | 53.75 | 31.40 | 65.89 | 52.96 | 33.00 | **43.00** |
| R=8 | LT | 2.8775 | 33.78 | 35.04 | 23.99 | 41.88 | 55.77 | 31.91 | 65.51 | 54.38 | 32.20 | 43.66 |
| | PLT | 2.9365 | 36.82 | 50.69 | 24.50 | 39.25 | 51.39 | 29.61 | 65.72 | 52.33 | 32.80 | 42.23 |
| | **InfuseLoop** | **2.9058** | **35.55** | **42.04** | **24.96** | 40.00 | 53.20 | 31.43 | 66.16 | 53.35 | 32.40 | **43.07** |

Key findings:
- InfuseLoop closes 52–61% of the validation-loss gap between vanilla PLT and sequential LT
- Perplexity improvements of 1.4–28.4% over vanilla PLT
- Average downstream accuracy improvements of 0.96 pp (R=4) and 0.84 pp (R=8) over vanilla PLT

### Efficiency Results

**Table 2: Logical cache footprint per token (KB) with window size w=64**

| Context length | 512 | 2,048 | 8,192 |
|----------------|-----|-------|-------|
| Single | 114.69 | 114.69 | 114.69 |
| InfuseLoop-4 | 157.72 (+37.5%) | 125.45 (+9.4%) | 117.38 (+2.3%) |
| InfuseLoop-8 | 215.10 (+87.5%) | 139.79 (+21.9%) | 120.96 (+5.5%) |

- Aggregation accounts for **<0.6%** of GPU kernel time
- At 8,192-token context, InfuseLoop retains **86.5%** of Single's throughput at B=32
- Estimated arithmetic intensity: 26.8 FLOPs/byte (far below GB200 ridge point of 312.5 FLOPs/byte), confirming memory-bound decoding

### Depth Scalability

- Vanilla PLT **worsens** validation loss from 4 to 8 loops
- InfuseLoop **reduces** validation loss from 4 to 8 loops
- InfuseLoop-4 dominates PLT-8 on the Pareto frontier (better quality at lower latency)

### Ablations (1B tokens training)

**Table 3: Component ablations**

| Configuration | R=4 Loss ↓ | R=4 Avg. ↑ | R=8 Loss ↓ | R=8 Avg. ↑ |
|---------------|-----------|-----------|-----------|-----------|
| **InfuseLoop** | **3.2482** | **38.26** | **3.2436** | **38.35** |
| Base only | 3.2660 | 38.04 | 3.2799 | 37.59 |
| FullState | 3.3975 | 37.43 | 3.4244 | 37.21 |
| In-trajectory | 3.2735 | 38.19 | 3.2839 | 37.87 |
| Pre | 3.2820 | 37.81 | 3.3707 | 37.41 |
| w/o source RMSNorm | 3.2883 | 38.02 | 3.3550 | 37.59 |
| w/o looping normalization | 3.2488 | 38.33 | 3.2434 | 38.17 |

Key findings:
- Cross-trajectory aggregation is essential (In-trajectory alone doesn't recover the benefit)
- FullState's larger source set ($O(R^2)$ candidates) hurts optimization
- Pre states (≥2 cycles old) are too stale to be useful
- Source RMSNorm matters more than backbone normalization at this scale

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Computational graph analysis**: The paper provides a formal characterization of state availability in PLTs, proving that only in-trajectory states are used across loop boundaries despite causally available cross-trajectory representations.

2. **Causal routing proof**: The appendix formally proves (Lemma 1, Lemma 2, Theorem 1) that:
   - Selected sources are causally available
   - Aggregation preserves autoregressive causality and prefix causality
   - Training and inference evaluate the same dependency graph

3. **Information flow design**: InfuseLoop demonstrates that redesigning information paths across trajectories is an effective lever for depth scaling, complementing prior work on optimization stability.

### Practical Implications

1. **Efficient depth scaling**: InfuseLoop enables quality gains from additional recurrent computation with a fixed backbone and minimal latency overhead (<1% over vanilla PLTs).

2. **Deployment feasibility**: The $O(Rd)$ storage bound and regular execution pattern (static shapes, CUDA Graph support) make InfuseLoop practical for batched inference.

3. **Complementarity with speculative decoding**: The paper notes InfuseLoop is potentially complementary to draft-and-verify decoding, though combined benefits depend on available compute.

---

## Conclusion

InfuseLoop introduces cross-trajectory latent injection as an effective mechanism for scaling parallel recurrent computation in Looped Transformers. By aggregating causally available hidden states from later prediction trajectories through lightweight attention residuals, it:

1. **Eliminates training-loss spikes** that plague vanilla PLTs
2. **Improves depth scaling** up to 8 loops (where vanilla PLT degrades)
3. **Preserves efficiency**: token-parallel training, concurrent loop execution, $O(Rd)$ storage

### Future Directions

- Scaling to larger dense models and mixture-of-experts (MoE) architectures
- Closing the remaining quality gap with sequential LTs
- Combining InfuseLoop with speculative decoding approaches
- Understanding the architectural and optimization factors behind the residual gap with sequential execution

The authors conclude that "cross-trajectory aggregation is important for deep recurrent models: additional computation becomes more effective when representations from later trajectories can reach the loops of earlier ones."

---

_Markdown view of https://picx.dev/p/doqKCM, served by PicX — AI-generated visual whiteboard summaries of research papers._
