Summary (Overview)

• LoopCoder-v2 is a 7B-parameter Parallel Loop Transformer (PLT) trained from scratch on 18T tokens, studying how loop count ($R$) affects test-time computation scaling for code tasks.
• The paper introduces a gain–cost view: each additional loop provides latent refinement (gain) but also incurs a positional mismatch cost from the cross-loop position offset (CLP) unique to PLT.
• Empirically, the two-loop variant ($R=2$) strongly outperforms the non-looped baseline ($R=1$) across code generation, reasoning, agentic SWE, and tool-use benchmarks (e.g., SWE-bench Verified: 43.0% → 64.4%, Multi-SWE: 14.0% → 31.0%).
• Three- and four-loop variants regress on many tasks (e.g., SWE-bench Verified drops to 27.6% at $R=3$), revealing a non-monotonic loop-count effect with optimal performance at $R=2$.
• Per-loop diagnostics (hidden-state coherence, attention change, output distribution shift, representational diversity) explain this saturation: loop 2 provides productive refinement; later loops yield diminishing, oscillatory updates and the CLP offset cost dominates.

Introduction and Theoretical Foundation

Looped Transformers scale effective depth by repeatedly applying a shared block $f_\theta$:

$$h^{(0)} = \text{Embed}(x),\quad h^{(r)} = f_\theta\left(h^{(r-1)}\right),\ r=1,\dots,R,\quad \text{logits} = \text{Head}\left(h^{(R)}\right)$$

This increases computational depth without proportionally increasing parameters. However, sequential looping incurs severe inference costs: latency $\propto R$ and KV-cache memory $\propto R \cdot L \cdot S \cdot d$ (Table 1).

The Parallel Loop Transformer (PLT) mitigates this with two mechanisms:

1. Cross-loop position offset (CLP): removes sequential dependency, enabling parallel execution.
2. Shared-KV gated sliding-window attention (G-SWA): reuses the first-loop KV cache, keeping memory nearly constant.

PLT makes loop count a practical design choice, but the optimal $R$ is unclear: too few loops underuse refinement capacity, too many may introduce harmful computation. The authors formulate this as a gain–cost trade-off: each loop provides marginal refinement gain, but CLP introduces a positional mismatch cost at loop boundaries. The key question is identifying the saturation point via internal diagnostics.

Methodology

Model Architecture: 7B dense transformer with PLT, G-SWA (window $w=64$), and first-loop KV sharing. Loop counts $R \in \{1,2,3,4\}$ are compared under matched training, instruction tuning, and evaluation.

PLT Operation:

• Shared-KV G-SWA (Eq. 1):

  $$\tilde{y}^{(r)} = g \odot y^{(r)}_{\text{global}} + (1-g) \odot y^{(r)}_{\text{local}},\quad g = \sigma\left(f_{\text{gate}}(\text{RMSNorm}(h))\right)$$

  where $y^{(r)}_{\text{global}}$ uses frozen KV from loop 1, and $y^{(r)}_{\text{local}}$ uses sliding-window attention over the current loop.

• Cross-loop position offset (CLP) (Eq. 2):

  $$B^{(r)} = \text{Embed}(x) + \text{shift}\left(h^{(r-1)}\right),\quad h^{(r)} = f_\theta\left(B^{(r)}\right)$$

  where $\text{shift}(h^{(r-1)})_i = h^{(r-1)}_{i-1}$ (with $h^{(r-1)}_0 = 0$). This introduces a positional mismatch: token $x_i$ at loop $r$ receives the hidden state of $x_{i-1}$ from the previous loop.

Training: 18T tokens of mixed text and code data, followed by instruction tuning. All variants share architecture except $R$.

Diagnostic Metrics:

• Hidden-state coherence: cosine similarity between consecutive loop representations.
• Attention evolution: change in attention patterns across loops.
• Output distribution shift: KL divergence between loop-wise logits.
• Representational diversity: effective rank of hidden states.
• Intrinsic offset cost $\Omega(r)$: quantifies CLP-induced mismatch from hidden states.

Empirical Validation / Results

The macroscopic loop-count effect is strongly non-monotonic:

• $R=2$ provides broad improvements over $R=1$ across code generation, reasoning, agentic SWE, and tool-use benchmarks.
• $R=3$ and $R=4$ regress on many tasks (e.g., SWE-bench drops to 27.6%), indicating harmful computation.

Per-loop diagnostics explain this:

• Loop 2 (transition $1 \rightarrow 2$): coherent hidden-state updates, significant change in attention routing, increased representation diversity, broad token-level refinement.
• Loop 3 (transition $2 \rightarrow 3$): diminishing updates, oscillatory hidden-state changes, attention patterns become similar to loop 2, effective rank stops increasing.
• The CLP offset cost $\Omega(r)$ remains roughly fixed across loops (about 0.25 relative magnitude), while refinement gain shrinks after loop 2, causing the cost to dominate.

Theoretical and Practical Implications

Theoretical Implications

• Provides a gain–cost framework for loop-count selection in PLT, generalizable to other parallel-loop architectures.
• Explains why PLT saturates at $R=2$: the CLP-induced positional mismatch is an inherent structural cost that does not decrease with more loops, while marginal refinement gains are quickly exhausted.
• Demonstrates that more loops are not always better in recurrent-depth models, challenging naive scaling assumptions.

Practical Implications

• For deployed PLT models, $R=2$ is the recommended operating point, avoiding costly over-looping that degrades performance.
• The diagnostic tools (coherence, attention change, diversity) provide a template for selecting loop count without exhaustive training and evaluation.
• PLT's near-constant latency and memory (Table 1) make $R=2$ deployment practical in resource-constrained settings.

Table 1: Sequential vs. PLT Costs

Conclusion

The paper presents LoopCoder-v2 and a systematic analysis of PLT loop-count selection. Key contributions:

1. Gain–cost view of loop-count selection, balancing refinement gain vs. CLP-induced offset cost.
2. Loop-wise diagnostics showing that loop 2 provides the main productive refinement; later loops yield diminishing, oscillatory updates and reduced diversity.
3. Large-scale empirical evidence with a 7B PLT coder: $R=2$ improves SWE-bench Verified from 43.0% to 64.4%, while $R=3$ regresses, confirming the saturation at two loops.

Future directions include extending the gain–cost framework to other recurrent-depth architectures, exploring adaptive loop count per token/position, and investigating whether the $R=2$ optimum generalizes to other domains and model scales.