# LightMTP: Lightweight Latent Multi-Token Prediction

> LightMTP predicts future hidden states with a lightweight head, matching next-token performance while boosting planning, reasoning, and code generation at under 3B parameters.

- **Source:** [arXiv](https://arxiv.org/abs/2610.06031)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/sx33ix
- **Whiteboard:** https://picx.dev/p/sx33ix/image

## Summary

## Summary (Overview)

- **LightMTP** is a novel lightweight latent multi-token prediction (MTP) pretraining objective that uses the model's **own future hidden states** as supervision targets, eliminating the need for auxiliary encoders, codebooks, or helper models.
- Two variants are proposed: **LightMTP-S** (single-target vector prediction) and **LightMTP-C** (concatenated multi-target vector prediction), both adding at most **1% extra parameters** and being discarded after training (inference unchanged).
- LightMTP retains **NTP-level performance on general language modeling benchmarks** (FineWeb-Edu, LAMBADA, CORE), unlike token-level MTP baselines (Meta, Meta-L, DeepSeek) which consistently degrade.
- On the **Countdown planning task**, LightMTP improves over NTP by up to **10 points** (56.7% vs. 46.4% at k=8) while using 98% fewer MTP parameters than Meta-L (1.2M vs. 75M).
- LightMTP demonstrates superior **anticipation cost** (next-token degradation per unit of future-token gain), with the top four most cost-efficient models being LightMTP variants, outperforming baselines by up to two orders of magnitude.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Next-token prediction (NTP) is the standard pretraining objective for large language models, requiring no supervision beyond raw text and computing loss at every position in a single teacher-forced forward pass. However, NTP provides an **explicit training signal only for the immediate next token**. Information about more distant tokens is learned only insofar as it helps predict the next one, which can lead models to exploit local patterns instead of capturing longer-range structure. Bachmann & Nagarajan (2024) show that teacher-forced NTP can fail to learn even simple planning tasks.

### Multi-Token Prediction (MTP)

Multi-token prediction addresses this limitation by additionally training the model to predict tokens $x_{t+2}, \ldots, x_{t+k}$ from its representation at position $t$. This densifies the training signal and rewards representations that encode information beyond the next token (Gloeckle et al., 2024; DeepSeek-AI et al., 2025). However, token-level MTP methods face key challenges:

- **Computational cost**: Each offset adds a vocabulary projection, so compute and memory grow with both $k$ and vocabulary size.
- **Parameter overhead**: Many designs add highly parametrized transformer layers per offset.
- **Interference with NTP**: Gloeckle et al. (2024) find MTP hurts models below roughly 3B parameters.

### Latent MTP

Latent MTP methods avoid the vocabulary projection by predicting dense representations of future tokens. However, existing approaches (Semformer, Next-Concept Prediction, Future Summaries) rely on **auxiliary encoders, codebooks, or separately trained helper models**, adding parameters and training stages.

**LightMTP** addresses these issues by bootstrapping targets from the model's own hidden states, requiring no auxiliary model and avoiding vocabulary projection entirely.

---

## Methodology

### Architecture

A transformer trunk with $L$ layers maps a prefix $\mathbf{X}_{\le t}$ to hidden states $h_t^l$ for $l = 1, \ldots, L$. Standard NTP trains a linear head $\mathbf{\Psi} \in \mathbb{R}^{|V| \times d}$ on $h_t^L$ with cross-entropy loss $\mathcal{L}_{\text{NTP}}$. LightMTP adds a second head $g_\phi$ (a two-layer MLP with GELU nonlinearity) that reads $h_t^L$ and predicts hidden states the trunk produces at future positions.

### Targets and Loss

**Targets**: The trunk's own hidden states $h_\tau^l$ at a chosen target layer $0 \le l \le L$, taken at future position $\tau > t$. Gradients are stopped through the target branch: $\text{stopgrad}(h_\tau^l)$.

**Cosine Loss**: Both predicted and target vectors are normalized:

$$
\ell_{\cos}(\hat{\mathbf{v}}, \mathbf{v}) = 1 - \frac{\hat{\mathbf{v}} \cdot \mathbf{v}}{\|\hat{\mathbf{v}}\|_2 \|\mathbf{v}\|_2} \tag{1}
$$

**Variance Regularization**: To prevent representational collapse (all positions becoming similar), a within-sequence regularizer penalizes cross-positional similarity:

$$
\hat{\boldsymbol{h}}_{b,t}^l = \text{norm}(\boldsymbol{h}_{b,t}^l), \quad \bar{\boldsymbol{h}}_b^l = \frac{1}{T} \sum_{t=1}^{T} \hat{\boldsymbol{h}}_{b,t}^l, \quad \mathcal{L}_{\text{VarReg}} = \frac{1}{B} \sum_{b=1}^{B} \|\bar{\boldsymbol{h}}_b^l\|_2 \tag{2}
$$

**Overall Loss**:

$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{NTP}} + \lambda_{\text{MTP}} \mathcal{L}_{\text{MTP}} + \lambda_{\text{VarReg}} \mathcal{L}_{\text{VarReg}} \tag{3}
$$

### LightMTP-S: Single-Target Vector Prediction

Predicts a single future representation $k$ steps ahead:

$$
\hat{\boldsymbol{v}}_t = \text{norm}\left(g_\phi\left(\boldsymbol{h}_t^L\right)\right), \quad \boldsymbol{v}_{t+k} = \text{norm}\left(\text{stopgrad}\left(\boldsymbol{h}_{t+k}^l\right)\right) \tag{4}
$$

$$
\mathcal{L}_{\text{MTP}}^{\text{single}} = \frac{1}{T-k} \sum_{t=1}^{T-k} \ell(\hat{\boldsymbol{v}}_t, \boldsymbol{v}_{t+k}) \tag{5}
$$

### LightMTP-C: Concatenated Multi-Target Vector Prediction

Predicts all $k$ future hidden states jointly via concatenation:

$$
[\hat{\boldsymbol{v}}_t^{(1)}; \dots; \hat{\boldsymbol{v}}_t^{(k)}] = g_\phi(\boldsymbol{h}_t^L) \in \mathbb{R}^{k \cdot d} \tag{6}
$$

$$
\hat{\boldsymbol{v}}_t^{(i)} \leftarrow \text{norm}\left(\hat{\boldsymbol{v}}_t^{(i)}\right), \qquad \boldsymbol{v}_{t+i} = \text{norm}\left(\text{sg}(\boldsymbol{h}_{t+i}^l)\right), \qquad i = 1, \ldots, k \tag{7}
$$

Per-step losses with masking for sequence boundaries:

$$
\ell_i = \frac{\sum_t m_{t,i} \ell(\hat{\boldsymbol{v}}_t^{(i)}, \boldsymbol{v}_{t+i})}{\sum_t m_{t,i}} \tag{8}
$$

$$
\mathcal{L}_{\text{MTP}}^{\text{concat}} = \sum_{i=1}^{k} \ell_i \tag{9}
$$

### Experimental Setup

- **Models**: 24-layer decoder-only models with 1.4–1.5B parameters (a scale where MTP often fails).
- **Baselines**: NTP, Meta-style MTP, Meta-L (larger), DeepSeek-style MTP, Future Summaries.
- **LightMTP configuration**: Target layer $l = 16$ (two-thirds of depth), horizon $k = 4$ (also tested $k = 10$).
- **Token budgets**: Parameter-dependent, with Meta (+20.7%/+11.6%) and DeepSeek (+13.6%) seeing more tokens.
- **Batch size**: 524,288 tokens; tokenizer size 32,768.

---

## Empirical Validation / Results

### 5.1 Future Token Anticipation

Linear probing on frozen last hidden-layer outputs reveals a tradeoff: MTP methods improve future-token prediction ($k \geq 2$) at a cost to NTP ($k = 1$). The **anticipation cost** quantifies this:

$$
\text{Anticipation Cost} = \frac{\Delta \text{Bpb}_{k=1}}{\Delta \text{Acc}_{k>1}}
$$

Four LightMTP variants exhibit the **lowest anticipation cost** of all methods, with Future Summaries and DeepSeek having costs **two orders of magnitude higher**.

### 5.2 Language Modeling Performance

**Table 1: General LM benchmark results** (best bolded, runner-up underlined)

| Method | FineWeb-Edu (PPL) | Lambada (PPL) | Lambada (Acc) | CORE-avg |
|---|---|---|---|---|
| NTP | $1.683_{\pm 0.001}$ | $16.00_{\pm 0.72}$ | $43.59_{\pm 0.85}$ | $40.43_{\pm 0.68}$ |
| Meta | $1.712_{\pm 0.003}$ | $18.66_{\pm 0.84}$ | $42.14_{\pm 0.91}$ | $39.31_{\pm 0.69}$ |
| Meta-L | $1.698_{\pm 0.001}$ | $18.13_{\pm 0.48}$ | $42.84_{\pm 0.70}$ | $39.89_{\pm 0.42}$ |
| DeepSeek | $1.762_{\pm 0.038}$ | $31.49_{\pm 11.42}$ | $35.68_{\pm 4.28}$ | $35.81_{\pm 2.60}$ |
| Future Summaries | $1.685_{\pm 0.002}$ | $15.57_{\pm 0.33}$ | $44.43_{\pm 0.45}$ | $40.25_{\pm 0.37}$ |
| LightMTP-C | $1.683_{\pm 0.000}$ | $16.35_{\pm 0.20}$ | $43.22_{\pm 0.42}$ | $40.30_{\pm 0.23}$ |
| +VarReg | $1.689_{\pm 0.000}$ | $16.84_{\pm 0.41}$ | $42.65_{\pm 0.51}$ | $40.18_{\pm 0.43}$ |
| k=10 | $1.685_{\pm 0.005}$ | $15.97_{\pm 1.04}$ | $44.06_{\pm 1.47}$ | $40.57_{\pm 0.34}$ |
| LightMTP-S | $1.684_{\pm 0.000}$ | $16.28_{\pm 0.48}$ | $43.31_{\pm 0.71}$ | $40.13_{\pm 0.67}$ |
| +VarReg | $1.694_{\pm 0.002}$ | $17.53_{\pm 0.35}$ | $42.33_{\pm 0.14}$ | $40.29_{\pm 0.46}$ |
| k=10 | $1.685_{\pm 0.001}$ | $15.92_{\pm 0.46}$ | $43.97_{\pm 0.16}$ | $40.19_{\pm 0.34}$ |

Key findings:
- All token-level MTP baselines (Meta, Meta-L, DeepSeek) **regress relative to NTP** on every metric.
- **LightMTP-C matches NTP exactly** on FineWeb-Edu perplexity; most variants stay within 0.01 of NTP.
- **LightMTP-C with k=10 achieves the best CORE-avg score** overall.
- **VarReg consistently harms** general language modeling despite improving anticipation.

### 5.3 Planning: Countdown

**Table 2: Countdown strict accuracy on held-out targets (%)**

| Objective | Held-out targets |
|---|---|
| NTP | $46.4 \pm 1.4$ |
| Meta, k=4 | $51.4 \pm 3.1$ |
| Meta, k=8 | $58.1 \pm 0.5$ |
| Meta-L, k=4 | $55.8 \pm 0.9$ |
| Meta-L, k=8 | $58.9 \pm 1.2$ |
| DeepSeek, k=1 | $47.8 \pm 2.5$ |
| DeepSeek, k=4 | $55.4 \pm 0.4$ |
| DeepSeek, k=8 | $55.1 \pm 0.3$ |
| Future Summaries‡ | $34.9 \pm 15.5$ |
| LightMTP-C, k=1 | $50.1 \pm 2.6$ |
| LightMTP-C, k=4 | $54.0 \pm 3.0$ |
| LightMTP-C, k=8 | $56.1 \pm 1.4$ |
| LightMTP-S, k=1 | $50.6 \pm 0.9$ |
| LightMTP-S, k=4 | $55.5 \pm 1.2$ |
| LightMTP-S, k=8 | $56.7 \pm 0.9$ |

‡ Held-out strict accuracy destabilizes late in training on 2 of 3 seeds.

- Both LightMTP variants **substantially outperform NTP** and improve consistently with increasing $k$.
- **LightMTP-S outperforms LightMTP-C at every $k$** with lower variance at the largest horizon, using only **1.2M MTP parameters vs. 75M for Meta-L** (98% reduction).
- LightMTP-S trails Meta's best by only **2.2 points** (56.7% vs. 58.9%).
- **DeepSeek is the only method where additional lookahead is counterproductive** (declines at k=8).
- Solution-space analysis shows MTP models preserve more paths to the target throughout generation.

### 5.4 Structure: GSM8K and MBPP

**Table 3: GSM8K results** (relaxed = gold answer anywhere; strict = correct answer format)

| Method | p@5 relaxed | p@5 strict |
|---|---|---|
| NTP | 41.97 ± 2.29 | 4.94 ± 1.44 |
| Meta | 41.45 ± 1.43 | 4.09 ± 0.08 |
| Meta-L | 41.91 ± 0.88 | 4.82 ± 1.19 |
| DeepSeek MTP | 39.17 ± 2.11 | 3.85 ± 0.68 |
| Future Summaries | 41.19 ± 0.53 | 4.95 ± 1.25 |
| **LightMTP-C** | **43.62 ± 1.51** | 4.98 ± 0.91 |
| **LightMTP-S** | **43.82 ± 1.33** | **5.69 ± 1.09** |

**Table 4: MBPP code-generation pass@k (%)** (d24 python-mix models, k=4)

| Method | zero-shot p@5 | zero-shot p@10 | finetuned p@5 | finetuned p@10 |
|---|---|---|---|---|
| NTP | 5.02 ± 0.97 | 8.52 ± 1.70 | 11.39 ± 1.22 | 15.19 ± 2.31 |
| Meta | 4.80 ± 1.80 | 7.04 ± 2.31 | 13.22 ± 2.64 | 17.78 ± 3.85 |
| Meta-L | 6.01 ± 2.10 | 9.26 ± 2.80 | 16.43 ± 0.48 | 21.85 ± 0.64 |
| DeepSeek | 0.19 ± 0.32 | 0.37 ± 0.64 | 2.47 ± 1.90 | 3.70 ± 2.57 |
| Fut. Sum. | 6.57 ± 0.63 | 8.89 ± 1.11 | 15.62 ± 1.14 | 21.11 ± 1.11 |
| **LightMTP-C** | **6.94 ± 0.10** | **9.63 ± 0.64** | 16.42 ± 1.18 | **23.33 ± 2.94** |
| LightMTP-S | 5.30 ± 2.28 | 8.89 ± 4.44 | 12.55 ± 1.63 | 17.78 ± 2.22 |

- **LightMTP-C achieves best zero-shot MBPP performance** at both pass@5 and pass@10.
- **DeepSeek MTP collapses on code** (near zero zero-shot), indicating poor transfer.
- **All MTP methods exhibit fewer structural errors** (NameError, SyntaxError, IndentationError, ModuleNotFoundError) than NTP, supporting the hypothesis that MTP teaches local look-ahead skills.

---

## Theoretical and Practical Implications

### Theoretical Significance

1. **Latent supervision without external models**: LightMTP demonstrates that a model's own intermediate hidden states can serve as effective supervision targets, eliminating the need for auxiliary encoders or helper models—a key architectural simplification over prior latent MTP approaches.

2. **Anticipation–NTP tradeoff**: The paper introduces the concept of **anticipation cost**, providing a principled framework for quantifying the tradeoff between future-token anticipation and next-token performance. This metric could guide future objective design.

3. **Intermediate-layer targets**: Taking targets from an intermediate layer ($l = 16$ of 24) rather than the final layer provides a more favorable tradeoff, suggesting that **representation depth matters** for MTP supervision quality.

### Practical Implications

1. **Scalability at smaller model sizes**: LightMTP works at 1.4–1.5B parameters, a regime where token-level MTP methods fail. This makes it practical as a **general-purpose pretraining objective** for smaller models.

2. **Computational efficiency**: Adding at most 1% parameters with no inference overhead makes LightMTP nearly **free to adopt** in existing training pipelines.

3. **Structural task improvement**: Consistent gains on planning (Countdown), reasoning (GSM8K), and code generation (MBPP) with reduced structural errors suggest LightMTP improves **local coherence and format adherence** without sacrificing general capabilities.

4. **Limitations of VarReg**: Variance regularization improves anticipation but consistently harms downstream performance, suggesting **gradient interference** between the anticipation objective and NTP—a caution for future regularization approaches.

---

## Conclusion

**LightMTP** introduces a lightweight latent multi-token prediction objective that uses a model's own future hidden states for supervision via a small auxiliary head. By predicting in latent space rather than over the vocabulary, it avoids the compute and memory overhead of token-level MTP, adding at most 1% extra parameters and leaving the trunk and inference path untouched.

The method balances between anticipating future tokens and retaining next-token performance better than all considered MTP baselines, making it particularly relevant at **smaller model scales (< 3B)**—a regime where prior MTP work either reports degraded language modeling or omits entirely.

**Future directions** include:
- Systematically characterizing the choices affecting anticipation cost (target granularity, prediction horizon, model scale).
- Providing a principled basis for designing multi-token objectives rather than selecting them empirically.
- Exploring whether the approach scales to larger models and whether intermediate-layer supervision continues to provide favorable tradeoffs.

---

_Markdown view of https://picx.dev/p/sx33ix, served by PicX — AI-generated visual whiteboard summaries of research papers._
