# Late Entry in Looped Language Models

> Late entry into deep context caches outperforms early exit in looped language models, yielding up to 8.4x exact and 12.5x lossy decoding speedups.

- **Source:** [arXiv](https://arxiv.org/abs/2611.24576)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/Uho9II
- **Whiteboard:** https://picx.dev/p/Uho9II/image

## Summary

# Late Entry in Looped Language Models

**Authors:** Ziqi Jin, Jiaxi Li, Weiwen Xu, Aixin Sun

## Summary (Overview)

- **Core idea:** In looped language models (LLMs) that iterate a shared layer stack $T$ times per token, the paper proposes **late entry** — having new tokens skip initial passes and enter directly into the final $K < T$ passes over a deep context cache — as opposed to conventional early exit which halts after initial passes with shallow context caches.

- **Key finding:** Deep cached context substitutes for token-local early passes. One late pass over a four-pass context (.962 agreement) outperforms two early passes over a two-pass context (.954) on Ouro-2.6B, with the advantage holding across model families and recurrence depths up to $T = 64$.

- **Method:** The paper introduces **Wake Verification**, a unified mechanism that completes exact $1 \to T$ pass caches via cross-step pipelining at zero extra serial calls, then projects finished wake states inside the same output GEMM to unify greedy-exact ($\tau = 0$), margin-gated ($\tau > 0$), and lossy ($\tau = \infty$) decoding in a single CUDA graph.

- **Results:** On Ouro-2.6B ($T=4$), $K=2$ Wake reaches **5.2× greedy-exact speedup** at $\tau = 0$ (+26% over $K=4$ block decoding) and **6.8×** at $\tau = \infty$. On Huginn-3.5B, scaling to $T \in \{8, 16, 32, 64\}$ yields **8.4× greedy-exact speedup** at $T=64$ (+153% over $K=64$ block) and up to **12.5× lossy speedup**.

- **Theory:** A formal proposition proves that $\tau = 0$ Wake Verification produces bit-for-bit identical token sequences to sequential $T$-pass greedy decoding with finite-step termination, verified across 258/258 adversarial fp64 tests.

## Introduction and Theoretical Foundation

### Background

Looped language models scale test-time compute without increasing model size by passing each token repeatedly through a shared Transformer stack for $T$ recurrent passes. During autoregressive decoding, running all $T$ passes per generated token turns recurrence depth directly into sequential latency. When each new token is restricted to $K < T$ serial passes, the question arises: **which passes should it execute?**

- **Early exit** (conventional): runs the first $K$ passes and halts, leaving past key-value (KV) caches truncated at shallow depth $K$, so all future tokens attend to immature context states.
- **Late entry** (proposed): skips initial passes and feeds each new token's embedding directly into the final $K$ passes over a deep context KV cache.

### Theoretical Foundation

The looped forward pass is formalized as follows. Given a prefix $x_{\leq i} = (x_1, \ldots, x_i)$, let $e(x_i) \in \mathbb{R}^d$ denote the input embedding of token $x_i$, and let $\mathcal{F}$ denote the shared stack of Transformer layers iterated across passes $k \in \{1, \ldots, T\}$. At pass $k$, $\mathcal{F}$ maps the preceding normalized hidden state $\tilde{h}_i^{(k-1)}$ (initialized at $\tilde{h}_i^{(0)} = e(x_i)$) over the pass-aligned KV cache $\mathcal{C}_{<i}^{(k)}$:

$$
h_i^{(k)} = \mathcal{F}\left(\tilde{h}_i^{(k-1)}; \mathcal{C}_{<i}^{(k)}\right), \qquad \tilde{h}_i^{(k)} = \mathrm{Norm}\left(h_i^{(k)}\right) \tag{1}
$$

where $\mathrm{Norm}(\cdot)$ denotes layer normalization.

**Key insight:** Passes $1$ through $T$ progressively refine prefix representations and store multi-pass contextual reasoning inside the deep KV cache $\mathcal{C}_{<i}^{(T)}$. Once the model is adapted so pass $T$ can form valid attention queries directly from a raw token embedding $e(x_i)$, the new token can read out that distilled context in a single pass rather than re-deriving depth locally.

### Why Raw Late Entry Fails Initially

Applying late entry directly to off-the-shelf models breaks predictions because late passes expect normalized hidden states from earlier passes. On Ouro-1.4B, raw late entry achieves only .552 agreement with full depth versus .938 for simple one-pass truncation. The issue is a **norm mismatch**: pass 1 expects raw token embeddings (median $\ell_2$ norm 0.76), whereas pass 4 expects layer-normalized hidden states emitted by pass 3 (norms 37–51).

**Solution:** A two-stage fine-tuning protocol:
- **Stage 1:** Standard full-depth instruction SFT
- **Stage 2:** Exposes half of the training segments directly to raw token embeddings at pass $T - K + 1$ (with $K \sim \mathrm{Uniform}\{1,2,3\}$) over a full-depth prefix cache

## Methodology

### 1. Late-Pass Block Verification

The paper uses a parameter-free mask-placeholder block decoder as its execution vehicle. For $T=4$, $K=2$, block size $B=2$:

- The last committed anchor token and $B$ candidate proposals each append a branch of $B$ mask placeholders, forming $(1+B)^2$ parallel rows that enter directly at pass $T-K+1$.
- A structured causal mask restricts candidate $\hat{x}_m$ to attend only to $x_0$ and $\hat{x}_{1:m}$ so verification never leaks future tokens.
- The decoder evaluates greedy late-entry predictions, accepts the longest matching prefix plus one bonus token, and reads next-step drafts from placeholder branches at zero separate drafting calls.

### 2. Pipelined Cache Completion

When tentatively accepted tokens advance the frontier after running only passes $T-K+1, \ldots, T$, their KV cache slots for passes $1, \ldots, T-K$ are empty. Rather than pausing generation:

- In-flight tentative token groups and the active $(1+B)^2$-row candidate block execute different pass indices side by side as parallel batch rows inside the same $K$ forward calls.
- Each tentative group advances through true passes $1 \to T$ at $K$ passes per step, finishing within $L = \lceil T/K \rceil$ steps.
- This preserves exact causal KV dependencies because tentative groups enter the wake queue in chronological order.

### 3. Wake Verification

Wake verification projects finished $T$-pass hidden states through the output projection $W_{\mathrm{out}}$ inside the **same single output GEMM** as the active block rows, adding zero Transformer layer calls and zero extra kernel launches.

Formally, let $z_{1:T}^{(i)} \in \mathbb{R}^{|\mathcal{V}|}$ denote the exact $T$-pass logit vector at wake position $i$, let $y_i^* = \arg\max_y z_{1:T}^{(i)}(y)$ be the true $T$-pass greedy next token, and let $\hat{y}_i = x_{i+1}$ be the tentatively accepted token. Parameterized by logit-margin tolerance $\tau \geq 0$, Wake verification triggers a rollback at the earliest index $i^* \in [u, v)$ satisfying:

$$
y_{i^*}^* \neq \hat{y}_{i^*} \quad \text{and} \quad z_{1:T}^{(i^*)}(y_{i^*}^*) - z_{1:T}^{(i^*)}(\hat{y}_{i^*}) > \tau \tag{2}
$$

The single knob $\tau$ unifies three regimes:
- **$\tau = 0$ (Exact Wake):** Guarantees $T$-pass greedy equivalence in exact arithmetic
- **$0 < \tau < \infty$ (Margin-Gated Wake):** Ignores benign near-tie flips while rolling back confident reasoning deviations
- **$\tau = \infty$ (Lossy Late Entry):** Non-rollback decoding

### Proposition 1: Exactness and Termination

> In exact arithmetic, Wake verification at $\tau = 0$ with EOS wake drain produces bit-for-bit identical token sequences to sequential $T$-pass greedy decoding for any $1 \leq K \leq T$ and $B \geq 0$, and terminates in at most $N_{\mathrm{max}} + L$ verified steps.

The proof relies on:
1. **Inductive prefix exactness:** Every position $< u$ equals the sequential $T$-pass greedy prefix with exact $1 \to T$ KV states
2. **Strict progress:** Every rollback advances the permanently verified prefix from $u$ to $i^* + 1 \geq u + 1 > u$, ruling out livelock

## Empirical Validation / Results

### Pass-Allocation Asymmetry (Section 2)

**Table 1: Diagnosing raw late-entry failure on off-the-shelf checkpoints**

| Input state supplied to pass 4 ($K=1$ over four-pass context) | Ouro-1.4B | Ouro-2.6B |
|---|---|---|
| Raw token embedding (unadapted) | .552 | .751 |
| Closed-form linear projection | .816 | .919 |
| True pass-3 hidden state (oracle) | .991 | .992 |
| Reference: 1-pass early exit | .938 | .949 |

**Table 2: First $K$ vs. Last $K$ passes (teacher-forced agreement)**

| Passes/token | Ouro-2.6B GSM8K (First K / Last K) | Ouro-2.6B IFEval (First K / Last K) | Ouro-1.4B GSM8K (First K / Last K) |
|---|---|---|---|
| $K=1$ | .898 / .962 | .757 / .887 | .842 / .948 |
| $K=2$ | .954 / .984 | .866 / .943 | .922 / .976 |
| $K=3$ | .982 / .992 | .949 / .969 | .967 / .990 |

**Key result:** Across all 12 cells, spending passes at the end beats the beginning on identical weights, with margins of +6.4 to +14.1 points at $K=1$.

### End-to-End Results on Ouro-2.6B ($T=4$)

**Table 3: Full-dataset accuracy and throughput (block size $B=8$, 21,403 items)**

| Benchmark | Full depth Acc. | $K=4$ Gap/Spd. | $K=3$ Gap/Spd. (vs. $K=4$) | $K=2$ Gap/Spd. (vs. $K=4$) |
|---|---|---|---|---|
| GSM8K | .873 | -0.6 / 4.05× | -2.2† / 5.09× (+26%) | -3.4† / 6.73× (+66%) |
| IFEval | .732 | +0.7 / 2.16× | +0.2 / 2.82× (+31%) | -3.3† / 3.78× (+75%) |
| MATH500 | .532 | -0.6 / 4.06× | +0.4 / 4.99× (+23%) | -4.0† / 6.49× (+60%) |
| MBPP | .616 | +1.0 / 3.65× | +0.2 / 4.48× (+23%) | -1.4 / 5.80× (+59%) |
| BBH | .577 | 0.0 / 3.30× | -1.3† / 4.30× (+30%) | -2.8† / 6.16× (+87%) |
| MMLU-Pro | .504 | +0.1 / 3.49× | -0.3 / 4.45× (+28%) | -1.6† / 5.91× (+69%) |

### Controlled Comparison: Trained Early Exit vs. Late Entry

**Table 4: GSM8K matched fine-tuning comparison**

| Passes/token | Early exit Acc. (tok/s) | Late entry Acc. (tok/s) | Gap (pts) | Sign-test p |
|---|---|---|---|---|
| $K=1$ | .330 (657) | .629 (544) | +29.9† | $4.4 \times 10^{-77}$ |
| $K=2$ | .669 (431) | .839 (427) | +16.9† | $2.7 \times 10^{-35}$ |
| $K=3$ | .785 (314) | .851 (323) | +6.5† | $8.3 \times 10^{-8}$ |
| $K=4$ | .822 (246) | .867 (257) | +4.5† | $2.2 \times 10^{-5}$ |

### Scaling to Deep Middle-Recurrence: Huginn-3.5B ($T=32$)

**Table 6: Full-dataset results on Huginn-3.5B ($T=32$, 8,871 items)**

| Benchmark | Full depth Acc. (tok/s) | $K=32$ Gap/Spd. | $K=16$ Gap/Spd. (vs. $K=32$) | $K=8$ Gap/Spd. (vs. $K=32$) |
|---|---|---|---|---|
| GSM8K | .393 (21.7) | +0.4 / 3.37× | +1.1 / 5.84× (+74%) | -3.9† / 9.13× (+171%) |
| IFEval | .551 (21.6) | 0.0 / 2.93× | +1.5 / 5.24× (+79%) | -0.9 / 8.96× (+206%) |
| BBH | .283 (22.6) | +0.2 / 3.79× | +0.3 / 6.65× (+75%) | -1.0† / 10.86× (+186%) |
| MBPP | .234 (22.4) | +0.2 / 4.46× | -0.2 / 7.76× (+74%) | -0.6 / 12.41× (+178%) |

### Wake Verification Spectrum

**Table 7 highlights:**

| Benchmark | Budget | $K=T$ Block Spd. | Exact Wake ($\tau=0$) Acc./rb/Spd. | Margin ($\tau>0$) Acc./Spd. | Lossy ($\tau=\infty$) Acc./Spd. |
|---|---|---|---|---|---|
| GSM8K (Ouro) | $K=2$ | 4.13× | .867 / 2.02 / 5.20× (+26%) | .877 / 5.52× | .842† / 6.82× |
| GSM8K (Huginn) | $K=8$ | 3.35× | .392 / 2.22 / 6.09× (+82%) | .397 / 7.15× | .353† / 9.13× |
| MBPP (Huginn) | $K=8$ | 4.47× | .236 / 1.41 / 7.69× (+72%) | .236 / 8.93× | .228 / 12.50× |

### Intra-Model Recurrence Depth Scaling

**Table 8: Huginn-3.5B across $T \in \{8, 16, 32, 64\}$ on GSM8K**

| Depth $T$ | Seq. Acc. (tok/s) | $K=T$ Spd. | $\tau=0$ Acc./rb at $K=T/2$ | $\tau=0$ Spd. at $K=T/2$ | $\tau=0$ Acc./rb at $K=T/4$ | $\tau=0$ Spd. at $K=T/4$ | $\tau=\infty$ Acc./Spd. at $K=T/4$ |
|---|---|---|---|---|---|---|---|
| $T=8$ | .267 (78.6) | 2.86× | .265 / 5.56 | 2.63× | .262 / 12.46 | 1.96× | .030† / 4.82× |
| $T=16$ | .382 (42.1) | 3.32× | .379 / 1.85 | 4.21× | .375 / 6.25 | 3.47× | .146† / 6.56× |
| $T=32$ | .391 (21.9) | 3.35× | .396 / 0.48 | 5.21× | .392 / 2.22 | 6.09× | .353† / 9.13× |
| $T=64$ | .397 (11.1) | 3.34× | .397 / 0.26 | 5.42× | .395 / 0.58 | 8.42× | .392 / 10.57× |

**Key finding:** Rollbacks per 100 tokens fall **21.5×** across $T = 8 \to 64$ at both $K = T/2$ (5.56 → 0.26) and $K = T/4$ (12.46 → 0.58).

### Roofline Model

The paper derives an $L_p$ Roofline law for single-pass GPU latency:

$$
t_{\mathrm{pass}}(M, C) = \left(\left(t_0 + \alpha_{\mathrm{kv}}C\right)^p + \left(\beta_{\mathrm{tc}}M\right)^p\right)^{1/p}, \qquad \Theta(K, B, C, \tau) = \frac{C \cdot \bar{c}_{\mathrm{net}}(K, B, \tau)}{\sum_{j=1}^{K} t_{\mathrm{pass}}\left(M_j(K, B, C), C\right)} \tag{3}
$$

Calibrated on $K=4$ modes only ($t_0 = 3.95$ ms, $\alpha_{\mathrm{kv}} = 0.05$ ms, $\beta_{\mathrm{tc}} = 0.0115$ ms/row, $p = 1.15$), it predicts unseen $K < 4$ multi-request throughput across $C \in [1, 32]$ at **8.3% MAPE** and identifies optimal block sizes on 11/12 (91.7%) operating points.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Pass-allocation asymmetry:** The paper formally demonstrates that in looped language models, deep cached context substitutes for token-local early passes — a structural property of the recurrence, not an artifact of training.

2. **Error localization:** fp32 interventions reveal that residual discrepancies concentrate in the 6% uncertain tail ($p_{\max} < 0.50$), with agreement reaching 1.000 ($K=2$) and .998 ($K=1$) on the 68% of confident tokens ($p_{\max} \geq 0.95$).

3. **Unified decoding framework:** Wake verification unifies greedy-exact ($\tau = 0$), margin-gated ($\tau > 0$), and lossy ($\tau = \infty$) decoding in one engine with provable exactness and termination at $\tau = 0$.

### Practical Implications

- **Speedups without accuracy loss:** Greedy-exact decoding at $\tau = 0$ surpasses full-depth block baselines by +22% to +153% across model families, making exact $T$-pass decoding faster than previously possible.
- **Controlled accuracy-throughput trade-offs:** The $\tau$ knob provides a single continuous control for trading bounded accuracy for throughput, from exact ($\tau = 0$) to fully lossy ($\tau = \infty$).
- **Hardware-aware scaling:** The Roofline law enables zero-shot prediction of multi-request throughput and dynamic block-size selection for serving workloads.
- **Generalization:** Late entry works across architectures (Ouro's stick-breaking readout and Huginn's middle-recurrent Coda), recurrence depths ($T \in \{4, 8, 16, 32, 64\}$), and drafters (trained mask-placeholder and parameter-free n-gram).

## Conclusion

This paper establishes that **saving passes from the two ends of a looped language model is fundamentally asymmetric**: cached context depth substitutes for token-local early passes across Ouro-1.4B, Ouro-2.6B ($T=4$), and middle-recurrent Huginn-3.5B ($T \in \{8, 16, 32, 64\}$). Pairing late-pass block verification with pipelined cache completion and zero-extra-pass Wake verification turns this asymmetry into a unified $\tau \in [0, \infty]$ family that accelerates both greedy-exact $T$-pass decoding (up to 5.2× at $T=4$ and 8.4× at $T=64$) and controlled lossy decoding (up to 6.8× and 12.5×).

### Limitations and Future Directions

1. **Hardware dependence:** Single-request speedups exploit HBM weight-streaming headroom on H100 GPUs; multi-request serving requires dynamic block-size reduction guided by the Roofline law.
2. **Greedy-only verification:** Proposition 1 targets greedy decoding; extending Wake rollback to temperature sampling via speculative rejection sampling is left to future work.
3. **Fine-tuning requirement:** Unlocking late entry on non-adapted checkpoints requires a one-time Stage-2 SFT pass (~1.9B tokens), motivating native late-entry exposure during pretraining and per-token adaptive pass scheduling.

Engine code, training scripts, and checkpoints are available at [github.com/Xalp/LateComer](https://github.com/Xalp/LateComer) and [huggingface.co/collections/Xalphinions/late-entry](https://huggingface.co/collections/Xalphinions/late-entry).

---

_Markdown view of https://picx.dev/p/Uho9II, served by PicX — AI-generated visual whiteboard summaries of research papers._
