# Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers

> LoopDuet shows that asymmetric loop depths for prefilling versus decoding—with prefilling shallower—boost throughput by up to 19% and quality by 6.04 EM points without retraining.

- **Source:** [arXiv](https://arxiv.org/abs/2611.07211)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/IHWPS2
- **Whiteboard:** https://picx.dev/p/IHWPS2/image

## Summary

## Summary (Overview)

- **LoopDuet** is a training-free serving approach for looped transformers that jointly selects prefilling depth ($P$) and decoding depth ($S$) with $P \leq S$, exploiting the asymmetry that decoding is more sensitive to reduced computation than prefilling.
- The method achieves **19% more throughput** than the fastest tested uniform setting meeting the same accuracy criterion on Huginn, and gains **6.04 EM points** over uniform depth eight at 97% of its throughput.
- A **queue-age controller** adapts prefilling depth dynamically during serving, retaining **0.92–1.35 EM points** more quality than fixed-depth controls while maintaining 100% attainment of a one-second time-to-first-token (TTFT) target.
- The execution mechanism gathers tokens requiring further recurrence into compact batches, enabling savings with existing weights and attention kernels in vLLM.
- The work formalizes recurrence as a **third axis of conditional computation** alongside physical layers (MoD) and expert identity (MoE), establishing connections to MoE serving techniques.

---

## Introduction and Theoretical Foundation

### Background and Motivation

**Looped transformers** scale inference-time reasoning by repeatedly applying a shared core, increasing effective depth without increasing parameter count. This weight-sharing property supports different inference budgets, offering quality–cost trade-offs at deployment. However, exploiting this flexibility requires a *depth allocation rule*: deciding how many loops each token receives.

### The Phase Asymmetry

LLM serving processes requests in two phases:
- **Prefilling**: processes prompt tokens in chunks to build the KV cache; the final prompt position produces the first output token
- **Decoding**: generates subsequent tokens one position at a time

A uniform loop budget couples phase costs: lowering prefilling depth also lowers decoding depth. For long prompts with short answers, retaining deep decoding while reducing prefilling depth can dramatically reduce computation.

Prior work exploits this asymmetry:
- **SwiftKV**: distills prompt KV states, skipping later-layer computation during prefilling while decoding runs the full model
- **POP**: applies early exit during prefilling without training, but skips fewer layers to preserve quality
- **SPEED**: processes prefilling tokens through only the first $K$ layers, but requires fine-tuning

**LoopDuet** applies this asymmetry to looped transformers *without training* by reducing prefilling *loops* rather than physical layers, selecting prefilling and decoding depths jointly, and adapting prefilling depth to load dynamically.

### Key Theoretical Insight

The layer-pass model quantifies work concentration. For a prompt of length $L$ followed by $D$ decoding tokens, uniform execution at depth $S$ performs:

$$(L + D) \cdot c_S$$

layer applications, where $c_k = a + bk + c$ (with $a$, $b$, $c$ being prelude, core, and coda layer counts). For OLMo HotpotQA with average 796-token prompts and ~4.7-token answers, prompt positions account for **over 99% of the work**. This concentration motivates reducing prefilling work while retaining decoding depth.

---

## Methodology

### 3.1 Joint Depth Selection ($S$, $P$)

LoopDuet selects prefilling depth $P$ and decoding depth $S$ jointly with $P \leq S$. For an unchunked prompt of length $L$:
- Positions $1, \ldots, L-1$ execute $P$ iterations
- Position $L$ (final prompt position) executes $S$ iterations (producing first output token)
- All subsequent generated positions execute $S$

**Preserving first-token computation** is critical: for short-answer tasks, the first token often determines the answer, so the final prompt position retains full decoding depth.

### 3.2 Compact Execution with Shared Context

To realize savings, the engine removes completed prefilling positions from later core calls:

1. **Full-batch processing**: Prelude and first $P$ core applications operate on the full batch of $N$ tokens
2. **Gather operation**: Let $I$ contain each request's final scheduled query index. Gathering these rows produces a $B \times d$ tensor (where $B$ is number of requests, $d$ is hidden width)
3. **Compact recurrence**: Run the remaining $S - P$ loops on this compact tensor
4. **Scatter**: Update the rows back into the $N \times d$ tensor before the coda

The shared KV cache retains prompt context so continuing positions can attend to full retained context through existing attention kernels.

### 3.3 Queue-Age Controller

The controller adjusts prefilling depth dynamically based on queue age $q$ (age of oldest admitted request awaiting first token):

$$z = \text{clip}\left(\frac{q - 0.25}{1.00 - 0.25}, 0, 1\right)$$

$$P = \text{round}\left(P_{\max} - z(P_{\max} - P_{\min})\right)$$

Rising queue age lowers $P$ toward $P_{\min}$; receding backlog restores it toward $P_{\max}$. **Decoding depth remains fixed.**

### Theoretical Work-Savings Model

The layer-pass savings ratio is:

$$R_{\text{layer}} = \frac{(L + D) \cdot c_S}{(L - 1) \cdot c_P + (D + 1) \cdot c_S}$$

For long prompts with short answers, this approaches $c_S / c_P$: **3.57** for OLMo (32, 8), 2.31 for OLMo (11, 4), and 1.94 for Huginn (32, 16).

Wall-time savings follow Amdahl's law:

$$R_{\text{time}} \approx \left[(1 - f) + f/r + \epsilon\right]^{-1}$$

where $f$ is the baseline wall-time fraction on affected work, $r$ its acceleration, and $\epsilon$ overhead.

---

## Empirical Validation / Results

### 4.2 Joint Depth Selection Results

**Setup**: Full HotpotQA dataset, one A100-SXM4-80GB, $C = 64$ concurrent requests, vLLM 0.14.1, bf16.

**Key Results for Huginn**:
- **(32, 16)** reaches EM 33.69 at 1.82 requests/s, providing **1.19× throughput** over fastest uniform setting meeting the accuracy criterion (20, 20) at 33.52 EM
- **(14, 8)** gains **6.04 EM points** over uniform (8, 8) (30.33 vs 24.29 EM) at 97% of throughput
- Relative to full (32, 32): 1.87× throughput with −0.09-point EM change

**Table 1: LoopDuet vs. Uniform Comparisons (full-set HotpotQA, $C = 64$)**

| Model | LoopDuet (S, P) | Uniform | Throughput ratio | ΔEM | ΔF1 |
|-------|-----------------|---------|-------------------|------|------|
| Huginn | (14, 8) | (8, 8) | 0.973 | +6.04 | +7.13 |
| Huginn | (24, 12) | (12, 12) | 0.979 | −0.45 | −0.47 |
| Huginn | (32, 16) | (16, 16) | 0.960 | +1.62 | +1.90 |
| Huginn | (32, 16) | (18, 18) | 1.073 | +0.97 | +1.10 |
| OLMo r32 | (11, 4) | (4, 4) | 0.918 | +0.32 | +0.49 |

**OLMo**: Uniform (4, 4) reaches 56.46 requests/s with EM 37.95, providing the highest throughput; LoopDuet (11, 4) gives 51.85 requests/s with EM 38.27 (+0.32 points).

### 4.3 Batching Speedups

Removing prefilling loops (holding decoding depth fixed) yields:
- **OLMo r32, $S = 32$**: 2.98× throughput ratio, −0.07 EM change
- **OLMo r32, $S = 11$**: 2.13×, −0.16 EM change  
- **Huginn, $S = 32$**: 1.87×, −0.09 EM change

Measured speedups reach **85–96% of layer-pass ratios**, consistent with compact dense execution realizing most arithmetic savings. Savings persist across concurrency levels ($C = 32, 64, 128$) and token budgets (4,096–16,384).

Latency improvements: OLMo (11, 4) median TTFT falls from 1.26 to 0.58 seconds; share with TTFT ≤ 1 second rises from 44.2% to 99.7%.

### 4.4 Queue Control Under Load

**Table 2: One-second TTFT attainment (%) / EM, OLMo r32, $S = 4$**

| Arrival trace | Fixed (4,4) | Fixed (4,3) | Fixed (4,2) | LoopDuet (queue) |
|---------------|-------------|-------------|-------------|-------------------|
| Poisson 40 req/s | 100 / 37.92 | 100 / 36.29 | 100 / 31.83 | **100 / 37.96** |
| Poisson 60 req/s | 8.6 / 38.04 | 100 / 36.35 | 100 / 31.74 | **100 / 37.27** |
| Burst 10/60 req/s | 46.9 / 38.04 | 100 / 36.42 | 100 / 31.82 | **100 / 37.77** |
| Burst 20/80 req/s | 21.3 / 38.20 | 39.1 / 36.42 | 100 / 31.91 | **87.5 / 35.12** |

Key findings:
- At Poisson 60 req/s: 0.92 EM points gained over best fixed depth at 100% attainment
- At burst 10/60: 1.35 EM points gained, with p99 TTFT of 0.682 seconds
- The trace spends ~87% of wall time at $P = 4$ and 13% at $P = 3$

### 4.5 Token-Role Sensitivity

- **First-token position** requires full depth: reducing final prompt position from 32 to 16 loops lowers EM by 1.74–1.97 points
- **Deeper decoding preserves GSM8K accuracy**: gains of 5.91 (OLMo r32), 3.18 (OLMo r16), 11.68 (Huginn) accuracy points over uniform reduction
- Role sensitivity varies by model: Huginn gains 4.97 EM points from deep decoding on HotpotQA, OLMo only 0.16–0.20

### 4.6 Recurrent-State Analysis

The relative state distance metric:

$$d_i(P; S) = \frac{\|h_i^P - h_i^S\|_2}{\|h_i^S\|_2}$$

Candidate prefilling depths satisfy:

$$q_{0.9}(P; S) \leq \tau, \quad \tau \approx 0.24$$

- At loop four, OLMo r32 GSM8K has prefilling/decoding distances 0.18/0.20, yet prefilling reduction preserves accuracy while uniform reduction loses 5.76 points
- State-distance screening suggests candidate depths of 3–4 for OLMo and 13–16 for Huginn
- Cross-checkpoint validation gives thresholds 0.237–0.241, close to pooled 0.24

---

## Theoretical and Practical Implications

### 3D Conditional Computation Framework

The paper formalizes recurrence as a third axis of conditional computation:
1. **Horizontal axis**: Mixture-of-Experts — token routing among experts within a layer
2. **Vertical axis**: Mixture-of-Depths / layer-skipping — computation adaptation across layers
3. **Recurrent axis (new)**: revisiting shared blocks — states evolve across iterations while parameters remain tied

This connects looped-model serving to mature MoE serving techniques (Lynx, DeepSpeed-MoE, SiDA-MoE, MoE-Infinity), suggesting cross-pollination opportunities.

### Key Design Principles

1. **Phase-aware allocation beats uniform**: Decoding sensitivity exceeds prefilling sensitivity, making asymmetric budgets superior to uniform reduction
2. **Batch-compatible granularity**: Per-token sparsity fails under batching (neuron union coverage 99.80–99.95% at batch 16); a common prefilling-loop boundary removes batched work effectively
3. **Quality retention under load**: Queue-age control enables graceful degradation that retains quality, unlike fixed-depth compromises
4. **State-distance screening**: One reference trajectory can screen candidate depths before task-quality validation, reducing evaluation cost

### Task Sensitivity

The benefit of depth allocation depends on task characteristics:
- GSM8K benefits from deeper decoding (multi-step reasoning)
- HotpotQA shows model-dependent sensitivity
- ARC requires sufficient prefilling depth
- Task relevance of state changes matters alongside their magnitude

---

## Conclusion

**LoopDuet** establishes that phase-dependent depth allocation is a practical way to turn flexible model computation into higher serving capacity and better quality under load:

- **Huginn**: 19% more throughput than fastest uniform setting meeting accuracy criterion; 6.04 more EM points than uniform depth eight at 97% of throughput
- **OLMo**: adaptive prefilling retains 0.92–1.35 more EM points than fixed settings at the same TTFT attainment on evaluated traces

Key insights for future work:
1. Prefilling and decoding depths must be selected **jointly**, as their benefits interact
2. The final prompt position deserves special treatment (full depth for first-token quality)
3. Queue-age control provides effective quality–latency trade-off management without correctness feedback
4. Recurrent-state measurements can screen candidate depths efficiently
5. The 3D MoE abstraction (layers × experts × recurrence) suggests extending MoE serving techniques to recurrent models
6. Joint scheduling of expert residency and per-phase recurrent depth is an open direction for recurrent MoE systems

The work demonstrates that **not all loops cost the same** — the phase in which computation is applied fundamentally determines its quality and latency value, and exploiting this asymmetry yields measurable serving improvements with existing weights and kernels.

---

_Markdown view of https://picx.dev/p/IHWPS2, served by PicX — AI-generated visual whiteboard summaries of research papers._