Late Entry in Looped Language Models
Authors: Ziqi Jin, Jiaxi Li, Weiwen Xu, Aixin Sun
Summary (Overview)
-
Core idea: In looped language models (LLMs) that iterate a shared layer stack times per token, the paper proposes late entry — having new tokens skip initial passes and enter directly into the final passes over a deep context cache — as opposed to conventional early exit which halts after initial passes with shallow context caches.
-
Key finding: Deep cached context substitutes for token-local early passes. One late pass over a four-pass context (.962 agreement) outperforms two early passes over a two-pass context (.954) on Ouro-2.6B, with the advantage holding across model families and recurrence depths up to .
-
Method: The paper introduces Wake Verification, a unified mechanism that completes exact pass caches via cross-step pipelining at zero extra serial calls, then projects finished wake states inside the same output GEMM to unify greedy-exact (), margin-gated (), and lossy () decoding in a single CUDA graph.
-
Results: On Ouro-2.6B (), Wake reaches 5.2× greedy-exact speedup at (+26% over block decoding) and 6.8× at . On Huginn-3.5B, scaling to yields 8.4× greedy-exact speedup at (+153% over block) and up to 12.5× lossy speedup.
-
Theory: A formal proposition proves that Wake Verification produces bit-for-bit identical token sequences to sequential -pass greedy decoding with finite-step termination, verified across 258/258 adversarial fp64 tests.
Introduction and Theoretical Foundation
Background
Looped language models scale test-time compute without increasing model size by passing each token repeatedly through a shared Transformer stack for recurrent passes. During autoregressive decoding, running all passes per generated token turns recurrence depth directly into sequential latency. When each new token is restricted to serial passes, the question arises: which passes should it execute?
- Early exit (conventional): runs the first passes and halts, leaving past key-value (KV) caches truncated at shallow depth , so all future tokens attend to immature context states.
- Late entry (proposed): skips initial passes and feeds each new token's embedding directly into the final passes over a deep context KV cache.
Theoretical Foundation
The looped forward pass is formalized as follows. Given a prefix , let denote the input embedding of token , and let denote the shared stack of Transformer layers iterated across passes . At pass , maps the preceding normalized hidden state (initialized at ) over the pass-aligned KV cache :
where denotes layer normalization.
Key insight: Passes through progressively refine prefix representations and store multi-pass contextual reasoning inside the deep KV cache . Once the model is adapted so pass can form valid attention queries directly from a raw token embedding , the new token can read out that distilled context in a single pass rather than re-deriving depth locally.
Why Raw Late Entry Fails Initially
Applying late entry directly to off-the-shelf models breaks predictions because late passes expect normalized hidden states from earlier passes. On Ouro-1.4B, raw late entry achieves only .552 agreement with full depth versus .938 for simple one-pass truncation. The issue is a norm mismatch: pass 1 expects raw token embeddings (median norm 0.76), whereas pass 4 expects layer-normalized hidden states emitted by pass 3 (norms 37–51).
Solution: A two-stage fine-tuning protocol:
- Stage 1: Standard full-depth instruction SFT
- Stage 2: Exposes half of the training segments directly to raw token embeddings at pass (with ) over a full-depth prefix cache
Methodology
1. Late-Pass Block Verification
The paper uses a parameter-free mask-placeholder block decoder as its execution vehicle. For , , block size :
- The last committed anchor token and candidate proposals each append a branch of mask placeholders, forming parallel rows that enter directly at pass .
- A structured causal mask restricts candidate to attend only to and so verification never leaks future tokens.
- The decoder evaluates greedy late-entry predictions, accepts the longest matching prefix plus one bonus token, and reads next-step drafts from placeholder branches at zero separate drafting calls.
2. Pipelined Cache Completion
When tentatively accepted tokens advance the frontier after running only passes , their KV cache slots for passes are empty. Rather than pausing generation:
- In-flight tentative token groups and the active -row candidate block execute different pass indices side by side as parallel batch rows inside the same forward calls.
- Each tentative group advances through true passes at passes per step, finishing within steps.
- This preserves exact causal KV dependencies because tentative groups enter the wake queue in chronological order.
3. Wake Verification
Wake verification projects finished -pass hidden states through the output projection inside the same single output GEMM as the active block rows, adding zero Transformer layer calls and zero extra kernel launches.
Formally, let denote the exact -pass logit vector at wake position , let be the true -pass greedy next token, and let be the tentatively accepted token. Parameterized by logit-margin tolerance , Wake verification triggers a rollback at the earliest index satisfying:
The single knob unifies three regimes:
- (Exact Wake): Guarantees -pass greedy equivalence in exact arithmetic
- (Margin-Gated Wake): Ignores benign near-tie flips while rolling back confident reasoning deviations
- (Lossy Late Entry): Non-rollback decoding
Proposition 1: Exactness and Termination
In exact arithmetic, Wake verification at with EOS wake drain produces bit-for-bit identical token sequences to sequential -pass greedy decoding for any and , and terminates in at most verified steps.
The proof relies on:
- Inductive prefix exactness: Every position equals the sequential -pass greedy prefix with exact KV states
- Strict progress: Every rollback advances the permanently verified prefix from to , ruling out livelock
Empirical Validation / Results
Pass-Allocation Asymmetry (Section 2)
Table 1: Diagnosing raw late-entry failure on off-the-shelf checkpoints
| Input state supplied to pass 4 ( over four-pass context) | Ouro-1.4B | Ouro-2.6B |
|---|---|---|
| Raw token embedding (unadapted) | .552 | .751 |
| Closed-form linear projection | .816 | .919 |
| True pass-3 hidden state (oracle) | .991 | .992 |
| Reference: 1-pass early exit | .938 | .949 |
Table 2: First vs. Last passes (teacher-forced agreement)
| Passes/token | Ouro-2.6B GSM8K (First K / Last K) | Ouro-2.6B IFEval (First K / Last K) | Ouro-1.4B GSM8K (First K / Last K) |
|---|---|---|---|
| .898 / .962 | .757 / .887 | .842 / .948 | |
| .954 / .984 | .866 / .943 | .922 / .976 | |
| .982 / .992 | .949 / .969 | .967 / .990 |
Key result: Across all 12 cells, spending passes at the end beats the beginning on identical weights, with margins of +6.4 to +14.1 points at .
End-to-End Results on Ouro-2.6B ()
Table 3: Full-dataset accuracy and throughput (block size , 21,403 items)
| Benchmark | Full depth Acc. | Gap/Spd. | Gap/Spd. (vs. ) | Gap/Spd. (vs. ) |
|---|---|---|---|---|
| GSM8K | .873 | -0.6 / 4.05× | -2.2† / 5.09× (+26%) | -3.4† / 6.73× (+66%) |
| IFEval | .732 | +0.7 / 2.16× | +0.2 / 2.82× (+31%) | -3.3† / 3.78× (+75%) |
| MATH500 | .532 | -0.6 / 4.06× | +0.4 / 4.99× (+23%) | -4.0† / 6.49× (+60%) |
| MBPP | .616 | +1.0 / 3.65× | +0.2 / 4.48× (+23%) | -1.4 / 5.80× (+59%) |
| BBH | .577 | 0.0 / 3.30× | -1.3† / 4.30× (+30%) | -2.8† / 6.16× (+87%) |
| MMLU-Pro | .504 | +0.1 / 3.49× | -0.3 / 4.45× (+28%) | -1.6† / 5.91× (+69%) |
Controlled Comparison: Trained Early Exit vs. Late Entry
Table 4: GSM8K matched fine-tuning comparison
| Passes/token | Early exit Acc. (tok/s) | Late entry Acc. (tok/s) | Gap (pts) | Sign-test p |
|---|---|---|---|---|
| .330 (657) | .629 (544) | +29.9† | ||
| .669 (431) | .839 (427) | +16.9† | ||
| .785 (314) | .851 (323) | +6.5† | ||
| .822 (246) | .867 (257) | +4.5† |
Scaling to Deep Middle-Recurrence: Huginn-3.5B ()
Table 6: Full-dataset results on Huginn-3.5B (, 8,871 items)
| Benchmark | Full depth Acc. (tok/s) | Gap/Spd. | Gap/Spd. (vs. ) | Gap/Spd. (vs. ) |
|---|---|---|---|---|
| GSM8K | .393 (21.7) | +0.4 / 3.37× | +1.1 / 5.84× (+74%) | -3.9† / 9.13× (+171%) |
| IFEval | .551 (21.6) | 0.0 / 2.93× | +1.5 / 5.24× (+79%) | -0.9 / 8.96× (+206%) |
| BBH | .283 (22.6) | +0.2 / 3.79× | +0.3 / 6.65× (+75%) | -1.0† / 10.86× (+186%) |
| MBPP | .234 (22.4) | +0.2 / 4.46× | -0.2 / 7.76× (+74%) | -0.6 / 12.41× (+178%) |
Wake Verification Spectrum
Table 7 highlights:
| Benchmark | Budget | Block Spd. | Exact Wake () Acc./rb/Spd. | Margin () Acc./Spd. | Lossy () Acc./Spd. |
|---|---|---|---|---|---|
| GSM8K (Ouro) | 4.13× | .867 / 2.02 / 5.20× (+26%) | .877 / 5.52× | .842† / 6.82× | |
| GSM8K (Huginn) | 3.35× | .392 / 2.22 / 6.09× (+82%) | .397 / 7.15× | .353† / 9.13× | |
| MBPP (Huginn) | 4.47× | .236 / 1.41 / 7.69× (+72%) | .236 / 8.93× | .228 / 12.50× |
Intra-Model Recurrence Depth Scaling
Table 8: Huginn-3.5B across on GSM8K
| Depth | Seq. Acc. (tok/s) | Spd. | Acc./rb at | Spd. at | Acc./rb at | Spd. at | Acc./Spd. at |
|---|---|---|---|---|---|---|---|
| .267 (78.6) | 2.86× | .265 / 5.56 | 2.63× | .262 / 12.46 | 1.96× | .030† / 4.82× | |
| .382 (42.1) | 3.32× | .379 / 1.85 | 4.21× | .375 / 6.25 | 3.47× | .146† / 6.56× | |
| .391 (21.9) | 3.35× | .396 / 0.48 | 5.21× | .392 / 2.22 | 6.09× | .353† / 9.13× | |
| .397 (11.1) | 3.34× | .397 / 0.26 | 5.42× | .395 / 0.58 | 8.42× | .392 / 10.57× |
Key finding: Rollbacks per 100 tokens fall 21.5× across at both (5.56 → 0.26) and (12.46 → 0.58).
Roofline Model
The paper derives an Roofline law for single-pass GPU latency:
Calibrated on modes only ( ms, ms, ms/row, ), it predicts unseen multi-request throughput across at 8.3% MAPE and identifies optimal block sizes on 11/12 (91.7%) operating points.
Theoretical and Practical Implications
Theoretical Contributions
-
Pass-allocation asymmetry: The paper formally demonstrates that in looped language models, deep cached context substitutes for token-local early passes — a structural property of the recurrence, not an artifact of training.
-
Error localization: fp32 interventions reveal that residual discrepancies concentrate in the 6% uncertain tail (), with agreement reaching 1.000 () and .998 () on the 68% of confident tokens ().
-
Unified decoding framework: Wake verification unifies greedy-exact (), margin-gated (), and lossy () decoding in one engine with provable exactness and termination at .
Practical Implications
- Speedups without accuracy loss: Greedy-exact decoding at surpasses full-depth block baselines by +22% to +153% across model families, making exact -pass decoding faster than previously possible.
- Controlled accuracy-throughput trade-offs: The knob provides a single continuous control for trading bounded accuracy for throughput, from exact () to fully lossy ().
- Hardware-aware scaling: The Roofline law enables zero-shot prediction of multi-request throughput and dynamic block-size selection for serving workloads.
- Generalization: Late entry works across architectures (Ouro's stick-breaking readout and Huginn's middle-recurrent Coda), recurrence depths (), and drafters (trained mask-placeholder and parameter-free n-gram).
Conclusion
This paper establishes that saving passes from the two ends of a looped language model is fundamentally asymmetric: cached context depth substitutes for token-local early passes across Ouro-1.4B, Ouro-2.6B (), and middle-recurrent Huginn-3.5B (). Pairing late-pass block verification with pipelined cache completion and zero-extra-pass Wake verification turns this asymmetry into a unified family that accelerates both greedy-exact -pass decoding (up to 5.2× at and 8.4× at ) and controlled lossy decoding (up to 6.8× and 12.5×).
Limitations and Future Directions
- Hardware dependence: Single-request speedups exploit HBM weight-streaming headroom on H100 GPUs; multi-request serving requires dynamic block-size reduction guided by the Roofline law.
- Greedy-only verification: Proposition 1 targets greedy decoding; extending Wake rollback to temperature sampling via speculative rejection sampling is left to future work.
- Fine-tuning requirement: Unlocking late entry on non-adapted checkpoints requires a one-time Stage-2 SFT pass (~1.9B tokens), motivating native late-entry exposure during pretraining and per-token adaptive pass scheduling.
Engine code, training scripts, and checkpoints are available at github.com/Xalp/LateComer and huggingface.co/collections/Xalphinions/late-entry.
Related papers
- BaseRSI: Recursive Self-Improvement of a Base Vision-Language Model via Training-Free Harness Internalization
BaseRSI lets a vision-language model improve its own weights via training-free per-block activation edits, gaining +11.7 points in-distribution and +6.1 out-of-distribution across ten benchmarks.
- SWE-growth: Teacher-Guided Adaptive Self-Play for Software Engineering Agents
SWE-Growth uses a frozen teacher to guide self-play curricula, improving software agents to 31.7% on SWE-bench Pro and 17.2% on TerminalBench, surpassing student-only training.
- Rethinking Fixed-Horizon Latent Reasoning
Adaptive latent inference that stops reasoning when progress stalls improves accuracy by up to 7.2 points while cutting computation by 29%, as completing fixed latent horizons can degrade performance.