Summary (Overview)

• ARLOOP is a looped transformer architecture that adapts attention residuals (Kimi Team, 2026) to recurrent depth, replacing PreNorm's fixed unit-coefficient residual aggregation with attention-weighted aggregation over residual sources across depth.
• A key finding distinguishing looped from standard transformers: full residual history is unnecessary. Sliding-window depth attention matches or outperforms full-history attention at larger recurrent depths, because parameter sharing across recurrences produces highly redundant representations.
• Across controlled experiments (140M and 370M parameters; $R=1$ to $5$), ARLOOP consistently achieves the lowest validation perplexity versus PreNorm and prior residual designs (Sandwich-Norm, DeepLoop, Parcae).
• At realistic LLM pretraining scale, ARLOOP-0.6B beats a matched PRENORMLOOP-0.6B on 7 of 8 benchmarks, with large gains on math/coding (+5.2 GSM8K, +3.1 MBPP).
• ARLOOP-1.2B (0.5T tokens, $R=3$) outperforms Huginn-3.5B on all evaluated tasks and is competitive with Qwen3.5-2B despite far less training data. Models and code are released as fully-open artifacts.

Introduction and Theoretical Foundation

Motivation. Looped transformers repeatedly apply a shared set of transformer layers, increasing effective depth without adding distinct parameters (Dehghani et al., 2019; Yang et al., 2024; Saunshi et al., 2025). The number of recurrences $R$ provides an explicit axis for scaling training and inference compute.

Problem. Standard PreNorm residual connections accumulate all residual updates into a single stream with fixed unit coefficients:

$$\boldsymbol{h}_i = \boldsymbol{h}_{i-1} + f_i(\operatorname{Norm}(\boldsymbol{h}_{i-1})), \quad i = 1, \dots, 2L \tag{1}$$

Unrolling gives $\boldsymbol{h}_i = \boldsymbol{h}_0 + \sum_{j=1}^{i} f_j(\operatorname{Norm}(\boldsymbol{h}_{j-1}))$, meaning every earlier residual update contributes with a fixed coefficient. As recurrence increases, later computations have no explicit mechanism to select which earlier-depth representations are useful.

Theoretical basis. Attention residuals (Kimi Team, 2026) replace fixed aggregation with attention-weighted aggregation over depth. The paper asks how this selective access should be adapted to recurrent depth, where the same parameters are applied repeatedly and may produce redundant representations (Zeitoun et al., 2026; Wang et al., 2026a).

Methodology

Looped transformer formulation. An $L$-layer stack $\mathcal{S} = \mathcal{F}_{2L} \circ \cdots \circ \mathcal{F}_1$ is reused for $R$ recurrences:

$$\boldsymbol{y} = \mathrm{LMHead} \circ \mathcal{S}^{(R)} \circ \cdots \circ \mathcal{S}^{(1)}(\boldsymbol{h}_0)$$

with effective depth $2LR$ sublayers but only $2L$ distinct sublayers.

Attention residuals. With $z_0 := h_0$:

$$\boldsymbol{z}_j = f_j(\operatorname{Norm}(\boldsymbol{h}_{j-1})), \quad \alpha_{i,j} = \operatorname{Softmax}_{j \in \{0,\dots,i\}}\left(\boldsymbol{q}_i^\top \operatorname{Norm}(\boldsymbol{z}_j)\right) \tag{2}$$ $$\boldsymbol{h}_i = \alpha_{i,0}\boldsymbol{h}_0 + \sum_{j=1}^{i}\alpha_{i,j}\boldsymbol{z}_j \tag{3}$$

where $\boldsymbol{q}_i$ is a learned query vector for the $i$-th sublayer and $\alpha_{i,j}$ is the attention-based coefficient assigned to residual update $z_j$.

Block attention residuals. To prevent the number of sources from growing linearly with effective depth, consecutive sublayers are grouped into blocks of size $S$:

$$\boldsymbol{b}_k = \sum_{j=(k-1)S+1}^{kS}\boldsymbol{z}_j \tag{4}$$

Attention is performed over the embedding, preceding completed blocks, and the partial sum within the current block.

Sliding-window depth attention. Let $\mathcal{B} = \{b_0, b_1, \dots, b_m\}$ denote available sources ($b_0$ = embedding, $b_m$ = most recent block). With window size $W$, the source set is restricted to:

$$\mathcal{B}_W = \{b_0, b_{\max(1, m-W+1)}, \dots, b_m\}$$

The embedding remains directly accessible at every depth, while attention covers only the $W$ most recent blocks. ARLOOP uses full-history attention for $R \leq 2$ and sliding-window attention for $R > 2$.

Experimental setup.

• Controlled experiments: 140M (6 layers) and 370M (12 layers) decoder-only models trained on 11.2B / 29.6B FineWeb-Edu tokens, context 2048, residual block size of 6/12 sublayers (two blocks per recurrence).
• Scale-up: grouped-query attention + SwiGLU, context 4096, multi-stage data mixture (FineWeb-Edu, FineMath, Stack-Edu), SmolLM-style recipe.
• Baselines: PRENORMLOOP, Sandwich-Norm (Geiping et al., 2025), DeepLoop (Li et al., 2026), Parcae (Prairie et al., 2026), with/without input injection.

Empirical Validation / Results

Table 1: Validation perplexity (↓) at 140M and 370M scales (ARLOOP achieves the lowest PPL in every setting):

Gains over PreNorm grow with recurrence: $-0.11$ PPL at $R=1$ versus $-0.53$ PPL at $R=3$.

Table 2: Full-history vs. sliding-window depth attention (bold = best per row):

Why sliding-window works. At sublayer 47 ($R=4$, 140M), the three most recent residual blocks plus the embedding receive 96% of total attention weight. Pairwise cosine similarities between residual blocks across recurrences are high, indicating redundancy. This differs from standard transformers with matched effective depth, where cross-depth similarity is lower and sliding-window restriction degrades performance.

Table 3: Large-scale comparison (FLOPs in units of $10^{21}$; Data in trillions of tokens):

Key observations: ARLOOP-0.6B > PRENORMLOOP-0.6B on 7/8 tasks (+5.2 GSM8K, +3.1 MBPP); ARLOOP-1.2B beats Huginn-3.5B on all tasks and matches/exceeds Qwen3.5-2B on all benchmarks despite 72× less training data.

Layer-effect analysis (Figure 4). Using the metric of Csordás et al. (2025) on 50 MATH examples, layers in the 2nd and 3rd recurrences of ARLOOP contribute substantially more to subsequent layers than corresponding deeper layers in PRENORMLOOP—evidence that ARLOOP exploits later recurrent computation.

Table 4: Training memory and throughput at $R=4$ (single NVIDIA H100):

Memory overhead is negligible; ARLOOP_full is ~15% slower and ARLOOP_sw ~10% slower than PreNorm, attributable to high-bandwidth memory traffic rather than arithmetic.

Additional findings:

• Fixed vs. dynamic recurrence: fixed recurrence gives better PPL and downstream performance at both 140M and 370M scales.
• Inference-time scaling: performance improves up to the maximum training recurrence but does not extrapolate beyond it, consistent with prior looped LLM observations (Prairie et al., 2026; Zhu et al., 2025).

Theoretical and Practical Implications

• Design principle for looped transformers: selective residual access via depth attention is an effective mechanism for exploiting recurrent computation; the optimal design differs from standard transformers because parameter sharing induces representational redundancy, making local (sliding-window) attention sufficient and cheaper.
• Evaluation insight: PPL gains in controlled settings are modest, but gains on complex generative tasks (math, code) are much larger—highlighting that residual designs must be evaluated beyond perplexity and multiple-choice benchmarks, a gap in prior work (Parcae, DeepLoop, Zeitoun et al.).
• Compute efficiency: looped transformers with attention residuals offer a parameter- and data-efficient route to strong math/coding performance (ARLOOP-1.2B uses 0.5T tokens vs. Qwen3.5-2B's 36T).
• Open science: ARLOOP is fully open (code + checkpoints), addressing the scarcity of fully-open looped LLMs (only Huginn previously released all training artifacts).

Conclusion

ARLOOP adapts attention residuals to recurrent depth, consistently outperforming PreNorm and prior residual designs across parameter scales (140M–1.2B) and recurrence depths ($R=1$–5). The central architectural insight is that looped transformers need only local residual access: sliding-window depth attention matches or beats full-history attention at larger $R$, justified by high cross-recurrence representation similarity and attention concentration on recent sources. At scale, ARLOOP-0.6B substantially improves over matched PreNorm (especially on math/coding), and ARLOOP-1.2B outperforms Huginn-3.5B on all benchmarks and is competitive with Qwen3.5-2B under a far smaller data budget.

Future directions implied: kernel optimization to close the ~10% throughput gap; understanding why selective residual access does not enable recurrence extrapolation (composing information within trained depth vs. generalizing beyond it appear to be distinct problems); and further scaling of looped LLMs with attention residuals.