Summary (Overview)

• LoopDuet is a training-free serving approach for looped transformers that jointly selects prefilling depth ($P$) and decoding depth ($S$) with $P \leq S$, exploiting the asymmetry that decoding is more sensitive to reduced computation than prefilling.
• The method achieves 19% more throughput than the fastest tested uniform setting meeting the same accuracy criterion on Huginn, and gains 6.04 EM points over uniform depth eight at 97% of its throughput.
• A queue-age controller adapts prefilling depth dynamically during serving, retaining 0.92–1.35 EM points more quality than fixed-depth controls while maintaining 100% attainment of a one-second time-to-first-token (TTFT) target.
• The execution mechanism gathers tokens requiring further recurrence into compact batches, enabling savings with existing weights and attention kernels in vLLM.
• The work formalizes recurrence as a third axis of conditional computation alongside physical layers (MoD) and expert identity (MoE), establishing connections to MoE serving techniques.

---

Introduction and Theoretical Foundation

Background and Motivation

Looped transformers scale inference-time reasoning by repeatedly applying a shared core, increasing effective depth without increasing parameter count. This weight-sharing property supports different inference budgets, offering quality–cost trade-offs at deployment. However, exploiting this flexibility requires a depth allocation rule: deciding how many loops each token receives.

The Phase Asymmetry

LLM serving processes requests in two phases:

• Prefilling: processes prompt tokens in chunks to build the KV cache; the final prompt position produces the first output token
• Decoding: generates subsequent tokens one position at a time

A uniform loop budget couples phase costs: lowering prefilling depth also lowers decoding depth. For long prompts with short answers, retaining deep decoding while reducing prefilling depth can dramatically reduce computation.

Prior work exploits this asymmetry:

• SwiftKV: distills prompt KV states, skipping later-layer computation during prefilling while decoding runs the full model
• POP: applies early exit during prefilling without training, but skips fewer layers to preserve quality
• SPEED: processes prefilling tokens through only the first $K$ layers, but requires fine-tuning

LoopDuet applies this asymmetry to looped transformers without training by reducing prefilling loops rather than physical layers, selecting prefilling and decoding depths jointly, and adapting prefilling depth to load dynamically.

Key Theoretical Insight

The layer-pass model quantifies work concentration. For a prompt of length $L$ followed by $D$ decoding tokens, uniform execution at depth $S$ performs:

$$(L + D) \cdot c_S$$

layer applications, where $c_k = a + bk + c$ (with $a$, $b$, $c$ being prelude, core, and coda layer counts). For OLMo HotpotQA with average 796-token prompts and ~4.7-token answers, prompt positions account for over 99% of the work. This concentration motivates reducing prefilling work while retaining decoding depth.

---

Methodology

3.1 Joint Depth Selection ($S$, $P$)

LoopDuet selects prefilling depth $P$ and decoding depth $S$ jointly with $P \leq S$. For an unchunked prompt of length $L$:

• Positions $1, \ldots, L-1$ execute $P$ iterations
• Position $L$ (final prompt position) executes $S$ iterations (producing first output token)
• All subsequent generated positions execute $S$

Preserving first-token computation is critical: for short-answer tasks, the first token often determines the answer, so the final prompt position retains full decoding depth.

3.2 Compact Execution with Shared Context

To realize savings, the engine removes completed prefilling positions from later core calls:

1. Full-batch processing: Prelude and first $P$ core applications operate on the full batch of $N$ tokens
2. Gather operation: Let $I$ contain each request's final scheduled query index. Gathering these rows produces a $B \times d$ tensor (where $B$ is number of requests, $d$ is hidden width)
3. Compact recurrence: Run the remaining $S - P$ loops on this compact tensor
4. Scatter: Update the rows back into the $N \times d$ tensor before the coda

The shared KV cache retains prompt context so continuing positions can attend to full retained context through existing attention kernels.

3.3 Queue-Age Controller

The controller adjusts prefilling depth dynamically based on queue age $q$ (age of oldest admitted request awaiting first token):

$$z = \text{clip}\left(\frac{q - 0.25}{1.00 - 0.25}, 0, 1\right)$$ $$P = \text{round}\left(P_{\max} - z(P_{\max} - P_{\min})\right)$$

Rising queue age lowers $P$ toward $P_{\min}$; receding backlog restores it toward $P_{\max}$. Decoding depth remains fixed.

Theoretical Work-Savings Model

The layer-pass savings ratio is:

$$R_{\text{layer}} = \frac{(L + D) \cdot c_S}{(L - 1) \cdot c_P + (D + 1) \cdot c_S}$$

For long prompts with short answers, this approaches $c_S / c_P$: 3.57 for OLMo (32, 8), 2.31 for OLMo (11, 4), and 1.94 for Huginn (32, 16).

Wall-time savings follow Amdahl's law:

$$R_{\text{time}} \approx \left[(1 - f) + f/r + \epsilon\right]^{-1}$$

where $f$ is the baseline wall-time fraction on affected work, $r$ its acceleration, and $\epsilon$ overhead.

---

Empirical Validation / Results

4.2 Joint Depth Selection Results

Setup: Full HotpotQA dataset, one A100-SXM4-80GB, $C = 64$ concurrent requests, vLLM 0.14.1, bf16.

Key Results for Huginn:

• (32, 16) reaches EM 33.69 at 1.82 requests/s, providing 1.19× throughput over fastest uniform setting meeting the accuracy criterion (20, 20) at 33.52 EM
• (14, 8) gains 6.04 EM points over uniform (8, 8) (30.33 vs 24.29 EM) at 97% of throughput
• Relative to full (32, 32): 1.87× throughput with −0.09-point EM change

Table 1: LoopDuet vs. Uniform Comparisons (full-set HotpotQA, $C = 64$)

OLMo: Uniform (4, 4) reaches 56.46 requests/s with EM 37.95, providing the highest throughput; LoopDuet (11, 4) gives 51.85 requests/s with EM 38.27 (+0.32 points).

4.3 Batching Speedups

Removing prefilling loops (holding decoding depth fixed) yields:

• OLMo r32, $S = 32$: 2.98× throughput ratio, −0.07 EM change
• OLMo r32, $S = 11$: 2.13×, −0.16 EM change
• Huginn, $S = 32$: 1.87×, −0.09 EM change

Measured speedups reach 85–96% of layer-pass ratios, consistent with compact dense execution realizing most arithmetic savings. Savings persist across concurrency levels ($C = 32, 64, 128$) and token budgets (4,096–16,384).

Latency improvements: OLMo (11, 4) median TTFT falls from 1.26 to 0.58 seconds; share with TTFT ≤ 1 second rises from 44.2% to 99.7%.

4.4 Queue Control Under Load

Table 2: One-second TTFT attainment (%) / EM, OLMo r32, $S = 4$

Key findings:

• At Poisson 60 req/s: 0.92 EM points gained over best fixed depth at 100% attainment
• At burst 10/60: 1.35 EM points gained, with p99 TTFT of 0.682 seconds
• The trace spends ~87% of wall time at $P = 4$ and 13% at $P = 3$

4.5 Token-Role Sensitivity

• First-token position requires full depth: reducing final prompt position from 32 to 16 loops lowers EM by 1.74–1.97 points
• Deeper decoding preserves GSM8K accuracy: gains of 5.91 (OLMo r32), 3.18 (OLMo r16), 11.68 (Huginn) accuracy points over uniform reduction
• Role sensitivity varies by model: Huginn gains 4.97 EM points from deep decoding on HotpotQA, OLMo only 0.16–0.20

4.6 Recurrent-State Analysis

The relative state distance metric:

$$d_i(P; S) = \frac{\|h_i^P - h_i^S\|_2}{\|h_i^S\|_2}$$

Candidate prefilling depths satisfy:

$$q_{0.9}(P; S) \leq \tau, \quad \tau \approx 0.24$$

• At loop four, OLMo r32 GSM8K has prefilling/decoding distances 0.18/0.20, yet prefilling reduction preserves accuracy while uniform reduction loses 5.76 points
• State-distance screening suggests candidate depths of 3–4 for OLMo and 13–16 for Huginn
• Cross-checkpoint validation gives thresholds 0.237–0.241, close to pooled 0.24

---

Theoretical and Practical Implications

3D Conditional Computation Framework

The paper formalizes recurrence as a third axis of conditional computation:

1. Horizontal axis: Mixture-of-Experts — token routing among experts within a layer
2. Vertical axis: Mixture-of-Depths / layer-skipping — computation adaptation across layers
3. Recurrent axis (new): revisiting shared blocks — states evolve across iterations while parameters remain tied

This connects looped-model serving to mature MoE serving techniques (Lynx, DeepSpeed-MoE, SiDA-MoE, MoE-Infinity), suggesting cross-pollination opportunities.

Key Design Principles

1. Phase-aware allocation beats uniform: Decoding sensitivity exceeds prefilling sensitivity, making asymmetric budgets superior to uniform reduction
2. Batch-compatible granularity: Per-token sparsity fails under batching (neuron union coverage 99.80–99.95% at batch 16); a common prefilling-loop boundary removes batched work effectively
3. Quality retention under load: Queue-age control enables graceful degradation that retains quality, unlike fixed-depth compromises
4. State-distance screening: One reference trajectory can screen candidate depths before task-quality validation, reducing evaluation cost

Task Sensitivity

The benefit of depth allocation depends on task characteristics:

• GSM8K benefits from deeper decoding (multi-step reasoning)
• HotpotQA shows model-dependent sensitivity
• ARC requires sufficient prefilling depth
• Task relevance of state changes matters alongside their magnitude

---

Conclusion

LoopDuet establishes that phase-dependent depth allocation is a practical way to turn flexible model computation into higher serving capacity and better quality under load:

• Huginn: 19% more throughput than fastest uniform setting meeting accuracy criterion; 6.04 more EM points than uniform depth eight at 97% of throughput
• OLMo: adaptive prefilling retains 0.92–1.35 more EM points than fixed settings at the same TTFT attainment on evaluated traces

Key insights for future work:

1. Prefilling and decoding depths must be selected jointly, as their benefits interact
2. The final prompt position deserves special treatment (full depth for first-token quality)
3. Queue-age control provides effective quality–latency trade-off management without correctness feedback
4. Recurrent-state measurements can screen candidate depths efficiently
5. The 3D MoE abstraction (layers × experts × recurrence) suggests extending MoE serving techniques to recurrent models
6. Joint scheduling of expert residency and per-phase recurrent depth is an open direction for recurrent MoE systems

The work demonstrates that not all loops cost the same — the phase in which computation is applied fundamentally determines its quality and latency value, and exploiting this asymmetry yields measurable serving improvements with existing weights and kernels.