# ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction

> ECHO reuses the backbone's final layer for multi-token prediction, matching DeepSeek's auxiliary MTP performance at half the training cost while enabling depth extrapolation for speculative decoding.

- **Source:** [arXiv](https://arxiv.org/abs/2610.24766)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/BDaZ9P
- **Whiteboard:** https://picx.dev/p/BDaZ9P/image

## Summary

## Summary (Overview)

- **ECHO (Efficient Causal Self-Recursion)** redefines multi-token prediction (MTP) by recursively applying the backbone's own final layer to predict every future horizon, eliminating the need for auxiliary prediction modules.
- **Dynamic Depth Sampling (DDS)** reduces MTP training overhead by sampling prediction depths with decaying probabilities, achieving ECHO-1.5⁽³⁾ (expected depth 1.5) matching ECHO-3's downstream performance at half the cost.
- **Direct multi-horizon supervision** lands on the parameters that generate the next token, yielding substantial gains on reasoning (+2.63) and code (+2.23) at depth 3 over DeepSeek's auxiliary MTP.
- **Depth extrapolation** enables the pretrained model to serve as its own drafter beyond trained depths, outperforming post-hoc drafters (EAGLE-3, DFlash, DSpark) on high-entropy, open-ended generation.
- **Robustness validated** across a doubled token budget, repeated pretraining runs, deeper sampling (ECHO-2⁽⁷⁾), and supervised finetuning.

## Introduction and Theoretical Foundation

Large language models (LLMs) are predominantly trained with Next-Token Prediction (NTP), which provides supervision only at the immediate next-token horizon. Multi-Token Prediction (MTP) addresses this by adding future-token targets, but existing approaches (both parallel and causal) remain **auxiliary by construction**:

1. **Indirect Multi-Horizon Supervision**: MTP losses primarily train auxiliary prediction modules, meaning the backbone receives multi-horizon signals only indirectly.
2. **Costly Long-Horizon Supervision**: Each additional prediction depth adds a module in memory and another pass in compute, limiting practical horizons (typically a single draft token).

ECHO solves both problems by reusing the backbone's final decoder layer $T^L(\cdot)$ and output head at every horizon, connected by a lightweight fusion layer $F(\cdot)$:

$$h_t^L = \begin{cases} h_1^L, & t = 1, \\ T^L\big(F(h_{t-1}^L, e(x_t))\big), & t > 1, \end{cases} \tag{1}$$

The theoretical foundation distinguishes ECHO from looped transformers: while looped models recurse over the same token position, ECHO recurses **along the horizon**, conditioning each step on the next future token.

## Methodology

### Final-Layer Sharing

ECHO follows the autoregressive design of DeepSeek-V3, where each horizon is predicted from the previous final-layer hidden state $h_{t-1}^L$ and the embedding $e(y_t)$ of the previously predicted token. The key innovation is that **every parameter used for future prediction is also used for next-token generation** (apart from the lightweight fusion layer).

### Dynamic Depth Sampling (DDS)

At each optimizer step, DDS draws a prediction depth $D \sim \pi(k)$ over $\{1, \dots, K\}$, with expected MTP overhead:

$$\mathbb{E}[D] = \sum_{k=1}^{K} k\pi(k) \tag{2}$$

For ECHO-1.5⁽³⁾: $\pi = (0.6, 0.3, 0.1)$ with $K=3$, giving $\mathbb{E}[D] = 1.5$ (half the overhead of training all depths).

The depth training frequency follows:
$$f_k = \mathbb{P}(D \geq k) = \sum_{j=k}^{K} \pi(j) \tag{4}$$

For $\pi = (0.6, 0.3, 0.1)$: $f = (1, 0.4, 0.1)$.

### Theoretical Bound on Acceptance

The expected accepted length in speculative decoding satisfies:

$$\mathbb{E}[\tau] \geq 1 + K - \frac{1}{\sqrt{2}}\mathcal{L}_{\text{MTP}}(f) \geq 1 + K - \frac{1}{\sqrt{2}}\mathcal{L}_{\text{MTP}}(\mathbf{1}) \tag{5}$$

where $\mathcal{L}_{\text{MTP}}(f) = \sqrt{\sum_k c_k^2/f_k}\sqrt{\sum_k f_k \mathrm{KL}(d_k \| q_k)}$ bounds the MTP loss, and $c_k = K - k + 1$ accounts for truncation across depths. Decaying $f_k$ tightens this bound, making reduced training loss translate more directly into accepted length.

## Empirical Validation / Results

### Setup

- **Backbone**: 10B-A1B Latent MoE with 24 layers, multi-head latent attention, 256 routed experts (8 active per token)
- **Pretraining**: 600B tokens, sequence length 4,096, batch size 1,920, bfloat16 with Muon optimizer
- **Hardware**: 192 NVIDIA H800 GPUs (~3 days pretraining, 5 hours SFT)

### Key Results

| Configuration | MTP Overhead | Benchmark Avg | Reasoning | Code |
|--------------|-------------|---------------|-----------|------|
| MTP-1 (DeepSeek) | 1.0 | baseline | baseline | baseline |
| MTP-3 (DeepSeek) | 3.0 | +0.16 | — | — |
| ECHO-1 | 1.0 | +0.52 | +0.88 | — |
| ECHO-3 | 3.0 | +0.63 | +2.63 | +2.23 |
| ECHO-1.5⁽³⁾ | 1.5 | +0.63 (matches ECHO-3) | +1.31 | +2.37 |

### Robustness Findings

- **Data scaling**: Doubling tokens (78k → 156k steps) preserves ECHO's lead (+0.52 → +0.50 overall; +0.88 → +0.89 on reasoning & code)
- **Run repeat**: ECHO leads MTP-1 by +0.52 and +0.78 across two runs; reasoning & code lead shifts by only 0.11
- **Deeper sampling**: ECHO-2⁽⁷⁾ trails ECHO-1.5⁽³⁾ in pretraining but overtakes after SFT (+0.49), reaching the largest post-SFT lead (+2.10 over MTP-Instruct)

### Inference Results

ECHO keeps gaining acceptance length beyond trained depth (K > 3), while auxiliary MTP saturates. On high-entropy tasks, ECHO outperforms post-hoc drafters (warm-started EAGLE-3, DFlash, DSpark), with the largest gains on SWE-bench and other open-ended generation tasks.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Direct supervision paradigm**: Demonstrates that multi-horizon supervision on shared parameters outperforms module-based supervision, providing theoretical justification for why auxiliary MTP saturates at depth
2. **Training-compute alignment**: DDS encodes the natural decay of acceptance probabilities into training frequency, providing a principled connection between training schedule and speculative decoding dynamics
3. **Bound on acceptance**: Establishes a formal bound (Equation 5) relating MTP training loss to expected accepted length, showing why decaying depth supervision improves inference efficiency

### Practical Implications

1. **Cost reduction**: ECHO-1.5⁽³⁾ matches ECHO-3's performance at half the MTP training overhead, making deeper supervision practical at scale
2. **Elimination of drafter finetuning**: The pretrained model serves as its own drafter, avoiding post-hoc drafter training costs (22–55 hours of additional GPU time for baselines) and overfitting to academic benchmarks
3. **Real-world applicability**: Causal drafting outperforms block-parallel drafting on realistic tasks, which matter since open-ended generation (practical guidance, information seeking, writing) accounts for ~80% of consumer ChatGPT messages

## Conclusion

ECHO transforms multi-token prediction from an auxiliary task into part of the autoregressive objective. By reusing the backbone's final layer at every horizon with Dynamic Depth Sampling, it:
- Improves downstream performance (especially reasoning and code) where auxiliary MTP saturates
- Enables depth extrapolation at inference for speculative decoding
- Maintains advantages under doubled token budgets, repeated runs, and supervised finetuning

**Limitations**: Pretraining was repeated only once, and finetuning used a single seed. Future work includes exploring DDS at greater depths, understanding why deeper sampling pays off only after finetuning, and investigating the limits of depth extrapolation gains.

---

_Markdown view of https://picx.dev/p/BDaZ9P, served by PicX — AI-generated visual whiteboard summaries of research papers._