# Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

> InfiLoop replaces carry-last state replacement with learned attention over recurrent outputs, enabling exact O(d) memory streaming and 97.9% Sudoku accuracy at 20,000+ test-time iterations.

- **Source:** [arXiv](https://arxiv.org/abs/2610.11570)
- **Published:** 2026-10-12
- **Permalink:** https://picx.dev/p/57qkLe
- **Whiteboard:** https://picx.dev/p/57qkLe/image

## Summary

# Summary of "Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals"

## Summary (Overview)

- **Core Problem Identified**: The paper identifies the "carry-last" rule (feeding the latest block output directly as the next recurrent state) as a key bottleneck in looped Transformers. This rule causes performance degradation at scale—noisy updates overwrite correct intermediate deductions and destabilize solved states.
- **Main Contribution**: Introduces **InfiLoop**, a loop-native residual connection that replaces carry-last with learned attention over recurrent outputs, combining content-based scoring with exponential temporal decay. This admits an exact streaming recurrence requiring only $O(d)$ persistent memory per token, independent of loop count.
- **Key Results**: A 7M-parameter InfiLoop model achieves 97.9% exact accuracy on Sudoku-Extreme, 87.4% on Maze-Hard, 49.1% pass@2 on ARC-AGI-1, and 13.6% pass@2 on ARC-AGI-2, outperforming strong baselines like FPRM, TRM, and HRM.
- **Emergent Behavior**: The learned connection exhibits self-verification: it suppresses outputs during unsolved search (step size ~0.40), surges to 0.91 when the solution appears, and freezes the state once solved, preventing solution loss at extreme depths.
- **Scalability**: InfiLoop continues to improve with test-time looping beyond 20,000 effective steps, translating added depth directly into stronger reasoning.

## Introduction and Theoretical Foundation

### Background and Motivation

Looped Transformers address complex reasoning by repeatedly applying a shared Transformer block, adding computational depth without new parameters. The standard **carry-last rule** connects iterations by feeding the final output of one loop directly into the next:

$$z_t = f_\theta(h_{t-1}; x), \qquad h_t = z_t \tag{1}$$

This makes the unrolled computation equivalent to a deep network with shared weights, reintroducing the "curse of depth" where deeper layers contribute progressively less.

### Key Insight

The authors argue that state replacement is the fundamental bottleneck. Under carry-last:
- Earlier computations are accessible only through information preserved in the latest state
- A noisy or undesirable iteration can degrade useful representations throughout propagation
- The model cannot explicitly retain and select among outputs from earlier loops

### Theoretical Foundation

The paper proposes that the connection between iterations should be **learned** rather than fixed. The key observation is that "a newer output is not necessarily a better output"—yet carry-last passes every new block output to the next iteration in full.

## Methodology

### The InfiLoop Architecture

InfiLoop replaces carry-last with a learned attention-based aggregation over recurrent outputs. For candidate states $z_i$ and recurrent state $h_t$:

$$h_t = \sum_{i=0}^{t} \alpha_{t,i} z_i, \qquad \alpha_{t,i} = \frac{\exp(s_i - \lambda(t-i))}{\sum_{j=0}^{t} \exp(s_j - \lambda(t-j))} \tag{2}$$

where:
- $s_i = q^\top \text{RMSNorm}(z_i)$ is a content score from a shared learned pseudo-query $q$
- $\lambda > 0$ is a learned decay rate
- $\beta = e^{-\lambda} \in (0,1)$ discounts older candidates

### Exact Streaming Recurrence

The aggregation admits an exact constant-memory recurrence. Defining $e_i = \exp(s_i)$ and $\beta = e^{-\lambda}$:

$$N_t = \sum_{i=0}^{t-1} \beta^{t-i} e_i z_i + e_t z_t = \beta N_{t-1} + e_t z_t, \tag{4}$$
$$D_t = \sum_{i=0}^{t-1} \beta^{t-i} e_i + e_t = \beta D_{t-1} + e_t$$

with $h_t = N_t / D_t$. This requires only $O(d)$ memory per token (a vector $N_t$ and scalar $D_t$), independent of loop count.

### Interpretation as Adaptive Residual Update

Substituting $N_{t-1} = D_{t-1} h_{t-1}$ into Equation 4 yields:

$$h_t = h_{t-1} + \gamma_t (z_t - h_{t-1}), \quad \gamma_t = \frac{e_t}{\beta D_{t-1} + e_t} \in (0,1) \tag{5}$$

The step size $\gamma_t$ compares the candidate's content weight $e_t$ with the retained historical mass $\beta D_{t-1}$, allowing the model to suppress low-scoring proposals while accepting high-scoring ones.

### Architecture Details

- **Model**: 7M-parameter nested looped Transformer with two layers per block, hidden dimension 512, 8 attention heads, SwiGLU activations, and depthwise 3×3 convolution
- **Connection**: Applied independently to both inner and outer recurrent axes, adding only $d+1$ learned parameters per axis (513 total per axis)
- **Training**: Deep supervision with H=3 outer loops and L=6 inner updates for Sudoku; AdamW optimizer with peak learning rate $10^{-4}$ (Sudoku) or $3 \times 10^{-4}$ (Maze/ARC)

## Empirical Validation / Results

### Benchmark Performance

**Table 1: Reasoning benchmark results (%)** — Best results in bold; best among 7M-parameter models underlined.

| Method | Params | Sudoku-Ext. | Maze-Hard | ARC-1 | ARC-2 |
|---|---|---|---|---|---|
| **Large language models** | | | | | |
| DeepSeek R1 | 671B | 0.0 | 0.0 | 15.8 | 1.3 |
| Claude 3.7 Sonnet | - | 0.0 | 0.0 | 28.6 | 0.7 |
| o3-mini-high | - | 0.0 | 0.0 | 34.5 | 3.0 |
| Gemini 2.5 Pro (32K) | - | - | - | 37.0 | 4.9 |
| **Recursive reasoning models** | | | | | |
| HRM | 27M | 55.0 | 74.5 | 40.3 | 5.0 |
| Attractor Model | 27M | 91.4 | 93.1 | - | - |
| URM | 14M | 77.6 | - | ≥53.8‡ | ≥16.0‡ |
| TRM | 7M | 74.7 | 85.3 | 44.6 | 7.8 |
| EqR | 7M | 93.0 | - | - | - |
| FPRM | 7M | 94.2 | 87.0 | 47.5 | 6.2 |
| **InfiLoop (ours)** | **7M** | **97.9** | **87.4** | **49.1** | **13.6** |

### Depth Scaling Results

- **At 1,872 layer evaluations**: InfiLoop achieves 90.9% exact accuracy on Sudoku-Extreme, exceeding FPRM's 90.1% at 3,744 layers (half the computation)
- **At 24,960 layers**: InfiLoop reaches 92.74%, vs. 91.63% for FPRM and 83.65% for TRM
- **Key finding**: InfiLoop at 936 layers (86.5%) is never matched by TRM even at 24,960 layers (83.65%)—a 26× depth advantage

### Difficulty Analysis

At 60 empty cells (hardest Sudoku difficulty), InfiLoop maintains 59.5% accuracy vs. 56.5% for FPRM and 36.0% for TRM—a +23.4 percentage point margin over TRM.

### Emergent Self-Verification

The learned step size $\gamma_t$ reveals three distinct phases:
1. **During search**: Suppressed at ~0.40 (below the equal-weight baseline of 0.56)
2. **At solution**: Surges to 0.91, giving dominant weight to the newly found solution
3. **After solving**: Returns to 0.56 baseline; state freezes

The content score $s_t$ correlates with solution correctness without explicit supervision: median score of 7.25 for fully correct boards, dropping to −0.58 at 33 wrong cells.

### Ablation Results

**Table 2: Component and initialization ablations** (Sudoku-Extreme, exact accuracy %)

| Configuration | H3 L6 | H3 L12 | H3 L96 | H32 L6 |
|---|---|---|---|---|
| Inner axis alone | 72.6 | 75.2 | 77.2 | 79.4 |
| Outer axis alone | 75.1 | 76.3 | 76.2 | 80.0 |
| **Both axes (ours)** | **82.1** | **85.3** | **87.4** | **89.7** |

**Temporal decay ablations** (both axes, H3 L6): Reverse decay ($\beta=1.82$): 2.0%; Uniform ($\beta=1$): 38.4%; Forward ($\beta=0.55$): 74.0%; **Learned from $\beta_0=0.55$**: **82.1%**

### Perturbation Stability

Under injected perturbations ($\epsilon \in [10^{-4}, 0.3]$), InfiLoop retains 100% of solved puzzles, while TRM loses 15-20% even without perturbation. TRM trajectories diverge to O(1) relative distance within 12 updates; InfiLoop constrains drift throughout.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Constant-memory aggregation**: The exact streaming recurrence reduces persistent memory from $O(Td)$ to $O(d)$ and total aggregation cost from $O(T^2d)$ to $O(Td)$ compared to Full Attention Residuals
2. **Convex hull guarantee**: Since attention weights are positive and sum to one, $\|h_t\| \leq \max_{0 \leq i \leq t} \|z_i\|$—aggregation never amplifies representation norm
3. **Adaptive residual update**: Equation 5 provides a principled framework for understanding how the model balances historical state vs. new proposals

### Practical Implications

- **Efficient test-time scaling**: InfiLoop achieves higher accuracy with fewer layers, making it practical for deployment where compute is constrained
- **Architecture-agnostic**: The framework applies to both standard looped Transformers and nested recurrent reasoners (TRM, HRM), with separate scorer-decay pairs per recurrent axis
- **Solution retention**: The learned connection prevents catastrophic forgetting of correct solutions, a critical property for long-horizon reasoning

## Conclusion

The paper establishes that **state replacement under carry-last is a primary bottleneck** in deep looped Transformers. The proposed InfiLoop connection:

1. **Preserves solutions indefinitely**: Unlike TRM, which loses 19.7% of solved puzzles within 36 updates, InfiLoop retains all solved states
2. **Scales effectively to extreme depths**: Continues improving beyond 20,000 effective steps
3. **Learns self-verification**: Emergent behavior suppresses bad proposals and accepts good ones without explicit correctness supervision
4. **Achieves state-of-the-art results**: Outperforms all 7M-parameter baselines across four reasoning benchmarks

**Future directions** suggested by the work include:
- Extending the framework to larger model scales and more complex reasoning tasks
- Investigating the interaction between the learned connection and other stabilization techniques (e.g., fixed-point methods)
- Exploring whether the emergent self-verification behavior can be leveraged for interpretability or verification-aware training

The code is publicly available at https://github.com/pixeli99/InfiLoop.

---

_Markdown view of https://picx.dev/p/57qkLe, served by PicX — AI-generated visual whiteboard summaries of research papers._