Late Entry in Looped Language Models

Authors: Ziqi Jin, Jiaxi Li, Weiwen Xu, Aixin Sun

Summary (Overview)

  • Core idea: In looped language models (LLMs) that iterate a shared layer stack TT times per token, the paper proposes late entry — having new tokens skip initial passes and enter directly into the final K<TK < T passes over a deep context cache — as opposed to conventional early exit which halts after initial passes with shallow context caches.

  • Key finding: Deep cached context substitutes for token-local early passes. One late pass over a four-pass context (.962 agreement) outperforms two early passes over a two-pass context (.954) on Ouro-2.6B, with the advantage holding across model families and recurrence depths up to T=64T = 64.

  • Method: The paper introduces Wake Verification, a unified mechanism that completes exact 1→T1 \to T pass caches via cross-step pipelining at zero extra serial calls, then projects finished wake states inside the same output GEMM to unify greedy-exact (τ=0\tau = 0), margin-gated (τ>0\tau > 0), and lossy (τ=∞\tau = \infty) decoding in a single CUDA graph.

  • Results: On Ouro-2.6B (T=4T=4), K=2K=2 Wake reaches 5.2× greedy-exact speedup at τ=0\tau = 0 (+26% over K=4K=4 block decoding) and 6.8× at τ=∞\tau = \infty. On Huginn-3.5B, scaling to T∈{8,16,32,64}T \in \{8, 16, 32, 64\} yields 8.4× greedy-exact speedup at T=64T=64 (+153% over K=64K=64 block) and up to 12.5× lossy speedup.

  • Theory: A formal proposition proves that τ=0\tau = 0 Wake Verification produces bit-for-bit identical token sequences to sequential TT-pass greedy decoding with finite-step termination, verified across 258/258 adversarial fp64 tests.

Introduction and Theoretical Foundation

Background

Looped language models scale test-time compute without increasing model size by passing each token repeatedly through a shared Transformer stack for TT recurrent passes. During autoregressive decoding, running all TT passes per generated token turns recurrence depth directly into sequential latency. When each new token is restricted to K<TK < T serial passes, the question arises: which passes should it execute?

  • Early exit (conventional): runs the first KK passes and halts, leaving past key-value (KV) caches truncated at shallow depth KK, so all future tokens attend to immature context states.
  • Late entry (proposed): skips initial passes and feeds each new token's embedding directly into the final KK passes over a deep context KV cache.

Theoretical Foundation

The looped forward pass is formalized as follows. Given a prefix x≤i=(x1,…,xi)x_{\leq i} = (x_1, \ldots, x_i), let e(xi)∈Rde(x_i) \in \mathbb{R}^d denote the input embedding of token xix_i, and let F\mathcal{F} denote the shared stack of Transformer layers iterated across passes k∈{1,…,T}k \in \{1, \ldots, T\}. At pass kk, F\mathcal{F} maps the preceding normalized hidden state h~i(k−1)\tilde{h}_i^{(k-1)} (initialized at h~i(0)=e(xi)\tilde{h}_i^{(0)} = e(x_i)) over the pass-aligned KV cache C<i(k)\mathcal{C}_{<i}^{(k)}:

hi(k)=F(h~i(k−1);C<i(k)),h~i(k)=Norm(hi(k))(1)h_i^{(k)} = \mathcal{F}\left(\tilde{h}_i^{(k-1)}; \mathcal{C}_{<i}^{(k)}\right), \qquad \tilde{h}_i^{(k)} = \mathrm{Norm}\left(h_i^{(k)}\right) \tag{1}

where Norm(⋅)\mathrm{Norm}(\cdot) denotes layer normalization.

Key insight: Passes 11 through TT progressively refine prefix representations and store multi-pass contextual reasoning inside the deep KV cache C<i(T)\mathcal{C}_{<i}^{(T)}. Once the model is adapted so pass TT can form valid attention queries directly from a raw token embedding e(xi)e(x_i), the new token can read out that distilled context in a single pass rather than re-deriving depth locally.

Why Raw Late Entry Fails Initially

Applying late entry directly to off-the-shelf models breaks predictions because late passes expect normalized hidden states from earlier passes. On Ouro-1.4B, raw late entry achieves only .552 agreement with full depth versus .938 for simple one-pass truncation. The issue is a norm mismatch: pass 1 expects raw token embeddings (median ℓ2\ell_2 norm 0.76), whereas pass 4 expects layer-normalized hidden states emitted by pass 3 (norms 37–51).

Solution: A two-stage fine-tuning protocol:

  • Stage 1: Standard full-depth instruction SFT
  • Stage 2: Exposes half of the training segments directly to raw token embeddings at pass T−K+1T - K + 1 (with K∼Uniform{1,2,3}K \sim \mathrm{Uniform}\{1,2,3\}) over a full-depth prefix cache

Methodology

1. Late-Pass Block Verification

The paper uses a parameter-free mask-placeholder block decoder as its execution vehicle. For T=4T=4, K=2K=2, block size B=2B=2:

  • The last committed anchor token and BB candidate proposals each append a branch of BB mask placeholders, forming (1+B)2(1+B)^2 parallel rows that enter directly at pass T−K+1T-K+1.
  • A structured causal mask restricts candidate x^m\hat{x}_m to attend only to x0x_0 and x^1:m\hat{x}_{1:m} so verification never leaks future tokens.
  • The decoder evaluates greedy late-entry predictions, accepts the longest matching prefix plus one bonus token, and reads next-step drafts from placeholder branches at zero separate drafting calls.

2. Pipelined Cache Completion

When tentatively accepted tokens advance the frontier after running only passes T−K+1,…,TT-K+1, \ldots, T, their KV cache slots for passes 1,…,T−K1, \ldots, T-K are empty. Rather than pausing generation:

  • In-flight tentative token groups and the active (1+B)2(1+B)^2-row candidate block execute different pass indices side by side as parallel batch rows inside the same KK forward calls.
  • Each tentative group advances through true passes 1→T1 \to T at KK passes per step, finishing within L=⌈T/K⌉L = \lceil T/K \rceil steps.
  • This preserves exact causal KV dependencies because tentative groups enter the wake queue in chronological order.

3. Wake Verification

Wake verification projects finished TT-pass hidden states through the output projection WoutW_{\mathrm{out}} inside the same single output GEMM as the active block rows, adding zero Transformer layer calls and zero extra kernel launches.

Formally, let z1:T(i)∈R∣V∣z_{1:T}^{(i)} \in \mathbb{R}^{|\mathcal{V}|} denote the exact TT-pass logit vector at wake position ii, let yi∗=arg⁡max⁡yz1:T(i)(y)y_i^* = \arg\max_y z_{1:T}^{(i)}(y) be the true TT-pass greedy next token, and let y^i=xi+1\hat{y}_i = x_{i+1} be the tentatively accepted token. Parameterized by logit-margin tolerance τ≥0\tau \geq 0, Wake verification triggers a rollback at the earliest index i∗∈[u,v)i^* \in [u, v) satisfying:

yi∗∗≠y^i∗andz1:T(i∗)(yi∗∗)−z1:T(i∗)(y^i∗)>τ(2)y_{i^*}^* \neq \hat{y}_{i^*} \quad \text{and} \quad z_{1:T}^{(i^*)}(y_{i^*}^*) - z_{1:T}^{(i^*)}(\hat{y}_{i^*}) > \tau \tag{2}

The single knob τ\tau unifies three regimes:

  • τ=0\tau = 0 (Exact Wake): Guarantees TT-pass greedy equivalence in exact arithmetic
  • 0<τ<∞0 < \tau < \infty (Margin-Gated Wake): Ignores benign near-tie flips while rolling back confident reasoning deviations
  • τ=∞\tau = \infty (Lossy Late Entry): Non-rollback decoding

Proposition 1: Exactness and Termination

In exact arithmetic, Wake verification at τ=0\tau = 0 with EOS wake drain produces bit-for-bit identical token sequences to sequential TT-pass greedy decoding for any 1≤K≤T1 \leq K \leq T and B≥0B \geq 0, and terminates in at most Nmax+LN_{\mathrm{max}} + L verified steps.

The proof relies on:

  1. Inductive prefix exactness: Every position <u< u equals the sequential TT-pass greedy prefix with exact 1→T1 \to T KV states
  2. Strict progress: Every rollback advances the permanently verified prefix from uu to i∗+1≥u+1>ui^* + 1 \geq u + 1 > u, ruling out livelock

Empirical Validation / Results

Pass-Allocation Asymmetry (Section 2)

Table 1: Diagnosing raw late-entry failure on off-the-shelf checkpoints

Input state supplied to pass 4 (K=1K=1 over four-pass context)Ouro-1.4BOuro-2.6B
Raw token embedding (unadapted).552.751
Closed-form linear projection.816.919
True pass-3 hidden state (oracle).991.992
Reference: 1-pass early exit.938.949

Table 2: First KK vs. Last KK passes (teacher-forced agreement)

Passes/tokenOuro-2.6B GSM8K (First K / Last K)Ouro-2.6B IFEval (First K / Last K)Ouro-1.4B GSM8K (First K / Last K)
K=1K=1.898 / .962.757 / .887.842 / .948
K=2K=2.954 / .984.866 / .943.922 / .976
K=3K=3.982 / .992.949 / .969.967 / .990

Key result: Across all 12 cells, spending passes at the end beats the beginning on identical weights, with margins of +6.4 to +14.1 points at K=1K=1.

End-to-End Results on Ouro-2.6B (T=4T=4)

Table 3: Full-dataset accuracy and throughput (block size B=8B=8, 21,403 items)

BenchmarkFull depth Acc.K=4K=4 Gap/Spd.K=3K=3 Gap/Spd. (vs. K=4K=4)K=2K=2 Gap/Spd. (vs. K=4K=4)
GSM8K.873-0.6 / 4.05×-2.2† / 5.09× (+26%)-3.4† / 6.73× (+66%)
IFEval.732+0.7 / 2.16×+0.2 / 2.82× (+31%)-3.3† / 3.78× (+75%)
MATH500.532-0.6 / 4.06×+0.4 / 4.99× (+23%)-4.0† / 6.49× (+60%)
MBPP.616+1.0 / 3.65×+0.2 / 4.48× (+23%)-1.4 / 5.80× (+59%)
BBH.5770.0 / 3.30×-1.3† / 4.30× (+30%)-2.8† / 6.16× (+87%)
MMLU-Pro.504+0.1 / 3.49×-0.3 / 4.45× (+28%)-1.6† / 5.91× (+69%)

Controlled Comparison: Trained Early Exit vs. Late Entry

Table 4: GSM8K matched fine-tuning comparison

Passes/tokenEarly exit Acc. (tok/s)Late entry Acc. (tok/s)Gap (pts)Sign-test p
K=1K=1.330 (657).629 (544)+29.9†4.4×10−774.4 \times 10^{-77}
K=2K=2.669 (431).839 (427)+16.9†2.7×10−352.7 \times 10^{-35}
K=3K=3.785 (314).851 (323)+6.5†8.3×10−88.3 \times 10^{-8}
K=4K=4.822 (246).867 (257)+4.5†2.2×10−52.2 \times 10^{-5}

Scaling to Deep Middle-Recurrence: Huginn-3.5B (T=32T=32)

Table 6: Full-dataset results on Huginn-3.5B (T=32T=32, 8,871 items)

BenchmarkFull depth Acc. (tok/s)K=32K=32 Gap/Spd.K=16K=16 Gap/Spd. (vs. K=32K=32)K=8K=8 Gap/Spd. (vs. K=32K=32)
GSM8K.393 (21.7)+0.4 / 3.37×+1.1 / 5.84× (+74%)-3.9† / 9.13× (+171%)
IFEval.551 (21.6)0.0 / 2.93×+1.5 / 5.24× (+79%)-0.9 / 8.96× (+206%)
BBH.283 (22.6)+0.2 / 3.79×+0.3 / 6.65× (+75%)-1.0† / 10.86× (+186%)
MBPP.234 (22.4)+0.2 / 4.46×-0.2 / 7.76× (+74%)-0.6 / 12.41× (+178%)

Wake Verification Spectrum

Table 7 highlights:

BenchmarkBudgetK=TK=T Block Spd.Exact Wake (τ=0\tau=0) Acc./rb/Spd.Margin (τ>0\tau>0) Acc./Spd.Lossy (τ=∞\tau=\infty) Acc./Spd.
GSM8K (Ouro)K=2K=24.13×.867 / 2.02 / 5.20× (+26%).877 / 5.52×.842† / 6.82×
GSM8K (Huginn)K=8K=83.35×.392 / 2.22 / 6.09× (+82%).397 / 7.15×.353† / 9.13×
MBPP (Huginn)K=8K=84.47×.236 / 1.41 / 7.69× (+72%).236 / 8.93×.228 / 12.50×

Intra-Model Recurrence Depth Scaling

Table 8: Huginn-3.5B across T∈{8,16,32,64}T \in \{8, 16, 32, 64\} on GSM8K

Depth TTSeq. Acc. (tok/s)K=TK=T Spd.τ=0\tau=0 Acc./rb at K=T/2K=T/2τ=0\tau=0 Spd. at K=T/2K=T/2τ=0\tau=0 Acc./rb at K=T/4K=T/4τ=0\tau=0 Spd. at K=T/4K=T/4τ=∞\tau=\infty Acc./Spd. at K=T/4K=T/4
T=8T=8.267 (78.6)2.86×.265 / 5.562.63×.262 / 12.461.96×.030† / 4.82×
T=16T=16.382 (42.1)3.32×.379 / 1.854.21×.375 / 6.253.47×.146† / 6.56×
T=32T=32.391 (21.9)3.35×.396 / 0.485.21×.392 / 2.226.09×.353† / 9.13×
T=64T=64.397 (11.1)3.34×.397 / 0.265.42×.395 / 0.588.42×.392 / 10.57×

Key finding: Rollbacks per 100 tokens fall 21.5× across T=8→64T = 8 \to 64 at both K=T/2K = T/2 (5.56 → 0.26) and K=T/4K = T/4 (12.46 → 0.58).

Roofline Model

The paper derives an LpL_p Roofline law for single-pass GPU latency:

tpass(M,C)=((t0+αkvC)p+(βtcM)p)1/p,Θ(K,B,C,τ)=C⋅cˉnet(K,B,τ)∑j=1Ktpass(Mj(K,B,C),C)(3)t_{\mathrm{pass}}(M, C) = \left(\left(t_0 + \alpha_{\mathrm{kv}}C\right)^p + \left(\beta_{\mathrm{tc}}M\right)^p\right)^{1/p}, \qquad \Theta(K, B, C, \tau) = \frac{C \cdot \bar{c}_{\mathrm{net}}(K, B, \tau)}{\sum_{j=1}^{K} t_{\mathrm{pass}}\left(M_j(K, B, C), C\right)} \tag{3}

Calibrated on K=4K=4 modes only (t0=3.95t_0 = 3.95 ms, αkv=0.05\alpha_{\mathrm{kv}} = 0.05 ms, βtc=0.0115\beta_{\mathrm{tc}} = 0.0115 ms/row, p=1.15p = 1.15), it predicts unseen K<4K < 4 multi-request throughput across C∈[1,32]C \in [1, 32] at 8.3% MAPE and identifies optimal block sizes on 11/12 (91.7%) operating points.

Theoretical and Practical Implications

Theoretical Contributions

  1. Pass-allocation asymmetry: The paper formally demonstrates that in looped language models, deep cached context substitutes for token-local early passes — a structural property of the recurrence, not an artifact of training.

  2. Error localization: fp32 interventions reveal that residual discrepancies concentrate in the 6% uncertain tail (pmax⁡<0.50p_{\max} < 0.50), with agreement reaching 1.000 (K=2K=2) and .998 (K=1K=1) on the 68% of confident tokens (pmax⁡≥0.95p_{\max} \geq 0.95).

  3. Unified decoding framework: Wake verification unifies greedy-exact (τ=0\tau = 0), margin-gated (τ>0\tau > 0), and lossy (τ=∞\tau = \infty) decoding in one engine with provable exactness and termination at τ=0\tau = 0.

Practical Implications

  • Speedups without accuracy loss: Greedy-exact decoding at τ=0\tau = 0 surpasses full-depth block baselines by +22% to +153% across model families, making exact TT-pass decoding faster than previously possible.
  • Controlled accuracy-throughput trade-offs: The τ\tau knob provides a single continuous control for trading bounded accuracy for throughput, from exact (τ=0\tau = 0) to fully lossy (τ=∞\tau = \infty).
  • Hardware-aware scaling: The Roofline law enables zero-shot prediction of multi-request throughput and dynamic block-size selection for serving workloads.
  • Generalization: Late entry works across architectures (Ouro's stick-breaking readout and Huginn's middle-recurrent Coda), recurrence depths (T∈{4,8,16,32,64}T \in \{4, 8, 16, 32, 64\}), and drafters (trained mask-placeholder and parameter-free n-gram).

Conclusion

This paper establishes that saving passes from the two ends of a looped language model is fundamentally asymmetric: cached context depth substitutes for token-local early passes across Ouro-1.4B, Ouro-2.6B (T=4T=4), and middle-recurrent Huginn-3.5B (T∈{8,16,32,64}T \in \{8, 16, 32, 64\}). Pairing late-pass block verification with pipelined cache completion and zero-extra-pass Wake verification turns this asymmetry into a unified τ∈[0,∞]\tau \in [0, \infty] family that accelerates both greedy-exact TT-pass decoding (up to 5.2× at T=4T=4 and 8.4× at T=64T=64) and controlled lossy decoding (up to 6.8× and 12.5×).

Limitations and Future Directions

  1. Hardware dependence: Single-request speedups exploit HBM weight-streaming headroom on H100 GPUs; multi-request serving requires dynamic block-size reduction guided by the Roofline law.
  2. Greedy-only verification: Proposition 1 targets greedy decoding; extending Wake rollback to temperature sampling via speculative rejection sampling is left to future work.
  3. Fine-tuning requirement: Unlocking late entry on non-adapted checkpoints requires a one-time Stage-2 SFT pass (~1.9B tokens), motivating native late-entry exposure during pretraining and per-token adaptive pass scheduling.

Engine code, training scripts, and checkpoints are available at github.com/Xalp/LateComer and huggingface.co/collections/Xalphinions/late-entry.

Related papers