Summary (Overview)

• ECHO (Efficient Causal Self-Recursion) redefines multi-token prediction (MTP) by recursively applying the backbone's own final layer to predict every future horizon, eliminating the need for auxiliary prediction modules.
• Dynamic Depth Sampling (DDS) reduces MTP training overhead by sampling prediction depths with decaying probabilities, achieving ECHO-1.5⁽³⁾ (expected depth 1.5) matching ECHO-3's downstream performance at half the cost.
• Direct multi-horizon supervision lands on the parameters that generate the next token, yielding substantial gains on reasoning (+2.63) and code (+2.23) at depth 3 over DeepSeek's auxiliary MTP.
• Depth extrapolation enables the pretrained model to serve as its own drafter beyond trained depths, outperforming post-hoc drafters (EAGLE-3, DFlash, DSpark) on high-entropy, open-ended generation.
• Robustness validated across a doubled token budget, repeated pretraining runs, deeper sampling (ECHO-2⁽⁷⁾), and supervised finetuning.

Introduction and Theoretical Foundation

Large language models (LLMs) are predominantly trained with Next-Token Prediction (NTP), which provides supervision only at the immediate next-token horizon. Multi-Token Prediction (MTP) addresses this by adding future-token targets, but existing approaches (both parallel and causal) remain auxiliary by construction:

1. Indirect Multi-Horizon Supervision: MTP losses primarily train auxiliary prediction modules, meaning the backbone receives multi-horizon signals only indirectly.
2. Costly Long-Horizon Supervision: Each additional prediction depth adds a module in memory and another pass in compute, limiting practical horizons (typically a single draft token).

ECHO solves both problems by reusing the backbone's final decoder layer $T^L(\cdot)$ and output head at every horizon, connected by a lightweight fusion layer $F(\cdot)$:

$$h_t^L = \begin{cases} h_1^L, & t = 1, \\ T^L\big(F(h_{t-1}^L, e(x_t))\big), & t > 1, \end{cases} \tag{1}$$

The theoretical foundation distinguishes ECHO from looped transformers: while looped models recurse over the same token position, ECHO recurses along the horizon, conditioning each step on the next future token.

Methodology

Final-Layer Sharing

ECHO follows the autoregressive design of DeepSeek-V3, where each horizon is predicted from the previous final-layer hidden state $h_{t-1}^L$ and the embedding $e(y_t)$ of the previously predicted token. The key innovation is that every parameter used for future prediction is also used for next-token generation (apart from the lightweight fusion layer).

Dynamic Depth Sampling (DDS)

At each optimizer step, DDS draws a prediction depth $D \sim \pi(k)$ over $\{1, \dots, K\}$, with expected MTP overhead:

$$\mathbb{E}[D] = \sum_{k=1}^{K} k\pi(k) \tag{2}$$

For ECHO-1.5⁽³⁾: $\pi = (0.6, 0.3, 0.1)$ with $K=3$, giving $\mathbb{E}[D] = 1.5$ (half the overhead of training all depths).

The depth training frequency follows:

$$f_k = \mathbb{P}(D \geq k) = \sum_{j=k}^{K} \pi(j) \tag{4}$$

For $\pi = (0.6, 0.3, 0.1)$: $f = (1, 0.4, 0.1)$.

Theoretical Bound on Acceptance

The expected accepted length in speculative decoding satisfies:

$$\mathbb{E}[\tau] \geq 1 + K - \frac{1}{\sqrt{2}}\mathcal{L}_{\text{MTP}}(f) \geq 1 + K - \frac{1}{\sqrt{2}}\mathcal{L}_{\text{MTP}}(\mathbf{1}) \tag{5}$$

where $\mathcal{L}_{\text{MTP}}(f) = \sqrt{\sum_k c_k^2/f_k}\sqrt{\sum_k f_k \mathrm{KL}(d_k \| q_k)}$ bounds the MTP loss, and $c_k = K - k + 1$ accounts for truncation across depths. Decaying $f_k$ tightens this bound, making reduced training loss translate more directly into accepted length.

Empirical Validation / Results

Setup

• Backbone: 10B-A1B Latent MoE with 24 layers, multi-head latent attention, 256 routed experts (8 active per token)
• Pretraining: 600B tokens, sequence length 4,096, batch size 1,920, bfloat16 with Muon optimizer
• Hardware: 192 NVIDIA H800 GPUs (~3 days pretraining, 5 hours SFT)

Key Results

Robustness Findings

• Data scaling: Doubling tokens (78k → 156k steps) preserves ECHO's lead (+0.52 → +0.50 overall; +0.88 → +0.89 on reasoning & code)
• Run repeat: ECHO leads MTP-1 by +0.52 and +0.78 across two runs; reasoning & code lead shifts by only 0.11
• Deeper sampling: ECHO-2⁽⁷⁾ trails ECHO-1.5⁽³⁾ in pretraining but overtakes after SFT (+0.49), reaching the largest post-SFT lead (+2.10 over MTP-Instruct)

Inference Results

ECHO keeps gaining acceptance length beyond trained depth (K > 3), while auxiliary MTP saturates. On high-entropy tasks, ECHO outperforms post-hoc drafters (warm-started EAGLE-3, DFlash, DSpark), with the largest gains on SWE-bench and other open-ended generation tasks.

Theoretical and Practical Implications

Theoretical Contributions

1. Direct supervision paradigm: Demonstrates that multi-horizon supervision on shared parameters outperforms module-based supervision, providing theoretical justification for why auxiliary MTP saturates at depth
2. Training-compute alignment: DDS encodes the natural decay of acceptance probabilities into training frequency, providing a principled connection between training schedule and speculative decoding dynamics
3. Bound on acceptance: Establishes a formal bound (Equation 5) relating MTP training loss to expected accepted length, showing why decaying depth supervision improves inference efficiency

Practical Implications

1. Cost reduction: ECHO-1.5⁽³⁾ matches ECHO-3's performance at half the MTP training overhead, making deeper supervision practical at scale
2. Elimination of drafter finetuning: The pretrained model serves as its own drafter, avoiding post-hoc drafter training costs (22–55 hours of additional GPU time for baselines) and overfitting to academic benchmarks
3. Real-world applicability: Causal drafting outperforms block-parallel drafting on realistic tasks, which matter since open-ended generation (practical guidance, information seeking, writing) accounts for ~80% of consumer ChatGPT messages

Conclusion

ECHO transforms multi-token prediction from an auxiliary task into part of the autoregressive objective. By reusing the backbone's final layer at every horizon with Dynamic Depth Sampling, it:

• Improves downstream performance (especially reasoning and code) where auxiliary MTP saturates
• Enables depth extrapolation at inference for speculative decoding
• Maintains advantages under doubled token budgets, repeated runs, and supervised finetuning

Limitations: Pretraining was repeated only once, and finetuning used a single seed. Future work includes exploring DDS at greater depths, understanding why deeper sampling pays off only after finetuning, and investigating the limits of depth extrapolation gains.