Summary (Overview)

  • LightMTP is a novel lightweight latent multi-token prediction (MTP) pretraining objective that uses the model's own future hidden states as supervision targets, eliminating the need for auxiliary encoders, codebooks, or helper models.
  • Two variants are proposed: LightMTP-S (single-target vector prediction) and LightMTP-C (concatenated multi-target vector prediction), both adding at most 1% extra parameters and being discarded after training (inference unchanged).
  • LightMTP retains NTP-level performance on general language modeling benchmarks (FineWeb-Edu, LAMBADA, CORE), unlike token-level MTP baselines (Meta, Meta-L, DeepSeek) which consistently degrade.
  • On the Countdown planning task, LightMTP improves over NTP by up to 10 points (56.7% vs. 46.4% at k=8) while using 98% fewer MTP parameters than Meta-L (1.2M vs. 75M).
  • LightMTP demonstrates superior anticipation cost (next-token degradation per unit of future-token gain), with the top four most cost-efficient models being LightMTP variants, outperforming baselines by up to two orders of magnitude.

Introduction and Theoretical Foundation

Background and Motivation

Next-token prediction (NTP) is the standard pretraining objective for large language models, requiring no supervision beyond raw text and computing loss at every position in a single teacher-forced forward pass. However, NTP provides an explicit training signal only for the immediate next token. Information about more distant tokens is learned only insofar as it helps predict the next one, which can lead models to exploit local patterns instead of capturing longer-range structure. Bachmann & Nagarajan (2024) show that teacher-forced NTP can fail to learn even simple planning tasks.

Multi-Token Prediction (MTP)

Multi-token prediction addresses this limitation by additionally training the model to predict tokens xt+2,…,xt+kx_{t+2}, \ldots, x_{t+k} from its representation at position tt. This densifies the training signal and rewards representations that encode information beyond the next token (Gloeckle et al., 2024; DeepSeek-AI et al., 2025). However, token-level MTP methods face key challenges:

  • Computational cost: Each offset adds a vocabulary projection, so compute and memory grow with both kk and vocabulary size.
  • Parameter overhead: Many designs add highly parametrized transformer layers per offset.
  • Interference with NTP: Gloeckle et al. (2024) find MTP hurts models below roughly 3B parameters.

Latent MTP

Latent MTP methods avoid the vocabulary projection by predicting dense representations of future tokens. However, existing approaches (Semformer, Next-Concept Prediction, Future Summaries) rely on auxiliary encoders, codebooks, or separately trained helper models, adding parameters and training stages.

LightMTP addresses these issues by bootstrapping targets from the model's own hidden states, requiring no auxiliary model and avoiding vocabulary projection entirely.


Methodology

Architecture

A transformer trunk with LL layers maps a prefix X≤t\mathbf{X}_{\le t} to hidden states htlh_t^l for l=1,…,Ll = 1, \ldots, L. Standard NTP trains a linear head Ψ∈R∣V∣×d\mathbf{\Psi} \in \mathbb{R}^{|V| \times d} on htLh_t^L with cross-entropy loss LNTP\mathcal{L}_{\text{NTP}}. LightMTP adds a second head gϕg_\phi (a two-layer MLP with GELU nonlinearity) that reads htLh_t^L and predicts hidden states the trunk produces at future positions.

Targets and Loss

Targets: The trunk's own hidden states hτlh_\tau^l at a chosen target layer 0≤l≤L0 \le l \le L, taken at future position τ>t\tau > t. Gradients are stopped through the target branch: stopgrad(hτl)\text{stopgrad}(h_\tau^l).

Cosine Loss: Both predicted and target vectors are normalized:

ℓcos⁡(v^,v)=1−v^⋅v∥v^∥2∥v∥2(1)\ell_{\cos}(\hat{\mathbf{v}}, \mathbf{v}) = 1 - \frac{\hat{\mathbf{v}} \cdot \mathbf{v}}{\|\hat{\mathbf{v}}\|_2 \|\mathbf{v}\|_2} \tag{1}

Variance Regularization: To prevent representational collapse (all positions becoming similar), a within-sequence regularizer penalizes cross-positional similarity:

h^b,tl=norm(hb,tl),hˉbl=1T∑t=1Th^b,tl,LVarReg=1B∑b=1B∥hˉbl∥2(2)\hat{\boldsymbol{h}}_{b,t}^l = \text{norm}(\boldsymbol{h}_{b,t}^l), \quad \bar{\boldsymbol{h}}_b^l = \frac{1}{T} \sum_{t=1}^{T} \hat{\boldsymbol{h}}_{b,t}^l, \quad \mathcal{L}_{\text{VarReg}} = \frac{1}{B} \sum_{b=1}^{B} \|\bar{\boldsymbol{h}}_b^l\|_2 \tag{2}

Overall Loss:

Ltotal=LNTP+λMTPLMTP+λVarRegLVarReg(3)\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{NTP}} + \lambda_{\text{MTP}} \mathcal{L}_{\text{MTP}} + \lambda_{\text{VarReg}} \mathcal{L}_{\text{VarReg}} \tag{3}

LightMTP-S: Single-Target Vector Prediction

Predicts a single future representation kk steps ahead:

v^t=norm(gϕ(htL)),vt+k=norm(stopgrad(ht+kl))(4)\hat{\boldsymbol{v}}_t = \text{norm}\left(g_\phi\left(\boldsymbol{h}_t^L\right)\right), \quad \boldsymbol{v}_{t+k} = \text{norm}\left(\text{stopgrad}\left(\boldsymbol{h}_{t+k}^l\right)\right) \tag{4} LMTPsingle=1T−k∑t=1T−kℓ(v^t,vt+k)(5)\mathcal{L}_{\text{MTP}}^{\text{single}} = \frac{1}{T-k} \sum_{t=1}^{T-k} \ell(\hat{\boldsymbol{v}}_t, \boldsymbol{v}_{t+k}) \tag{5}

LightMTP-C: Concatenated Multi-Target Vector Prediction

Predicts all kk future hidden states jointly via concatenation:

[v^t(1);… ;v^t(k)]=gϕ(htL)∈Rk⋅d(6)[\hat{\boldsymbol{v}}_t^{(1)}; \dots; \hat{\boldsymbol{v}}_t^{(k)}] = g_\phi(\boldsymbol{h}_t^L) \in \mathbb{R}^{k \cdot d} \tag{6} v^t(i)←norm(v^t(i)),vt+i=norm(sg(ht+il)),i=1,…,k(7)\hat{\boldsymbol{v}}_t^{(i)} \leftarrow \text{norm}\left(\hat{\boldsymbol{v}}_t^{(i)}\right), \qquad \boldsymbol{v}_{t+i} = \text{norm}\left(\text{sg}(\boldsymbol{h}_{t+i}^l)\right), \qquad i = 1, \ldots, k \tag{7}

Per-step losses with masking for sequence boundaries:

ℓi=∑tmt,iℓ(v^t(i),vt+i)∑tmt,i(8)\ell_i = \frac{\sum_t m_{t,i} \ell(\hat{\boldsymbol{v}}_t^{(i)}, \boldsymbol{v}_{t+i})}{\sum_t m_{t,i}} \tag{8} LMTPconcat=∑i=1kℓi(9)\mathcal{L}_{\text{MTP}}^{\text{concat}} = \sum_{i=1}^{k} \ell_i \tag{9}

Experimental Setup

  • Models: 24-layer decoder-only models with 1.4–1.5B parameters (a scale where MTP often fails).
  • Baselines: NTP, Meta-style MTP, Meta-L (larger), DeepSeek-style MTP, Future Summaries.
  • LightMTP configuration: Target layer l=16l = 16 (two-thirds of depth), horizon k=4k = 4 (also tested k=10k = 10).
  • Token budgets: Parameter-dependent, with Meta (+20.7%/+11.6%) and DeepSeek (+13.6%) seeing more tokens.
  • Batch size: 524,288 tokens; tokenizer size 32,768.

Empirical Validation / Results

5.1 Future Token Anticipation

Linear probing on frozen last hidden-layer outputs reveals a tradeoff: MTP methods improve future-token prediction (k≥2k \geq 2) at a cost to NTP (k=1k = 1). The anticipation cost quantifies this:

Anticipation Cost=ΔBpbk=1ΔAcck>1\text{Anticipation Cost} = \frac{\Delta \text{Bpb}_{k=1}}{\Delta \text{Acc}_{k>1}}

Four LightMTP variants exhibit the lowest anticipation cost of all methods, with Future Summaries and DeepSeek having costs two orders of magnitude higher.

5.2 Language Modeling Performance

Table 1: General LM benchmark results (best bolded, runner-up underlined)

MethodFineWeb-Edu (PPL)Lambada (PPL)Lambada (Acc)CORE-avg
NTP1.683±0.0011.683_{\pm 0.001}16.00±0.7216.00_{\pm 0.72}43.59±0.8543.59_{\pm 0.85}40.43±0.6840.43_{\pm 0.68}
Meta1.712±0.0031.712_{\pm 0.003}18.66±0.8418.66_{\pm 0.84}42.14±0.9142.14_{\pm 0.91}39.31±0.6939.31_{\pm 0.69}
Meta-L1.698±0.0011.698_{\pm 0.001}18.13±0.4818.13_{\pm 0.48}42.84±0.7042.84_{\pm 0.70}39.89±0.4239.89_{\pm 0.42}
DeepSeek1.762±0.0381.762_{\pm 0.038}31.49±11.4231.49_{\pm 11.42}35.68±4.2835.68_{\pm 4.28}35.81±2.6035.81_{\pm 2.60}
Future Summaries1.685±0.0021.685_{\pm 0.002}15.57±0.3315.57_{\pm 0.33}44.43±0.4544.43_{\pm 0.45}40.25±0.3740.25_{\pm 0.37}
LightMTP-C1.683±0.0001.683_{\pm 0.000}16.35±0.2016.35_{\pm 0.20}43.22±0.4243.22_{\pm 0.42}40.30±0.2340.30_{\pm 0.23}
+VarReg1.689±0.0001.689_{\pm 0.000}16.84±0.4116.84_{\pm 0.41}42.65±0.5142.65_{\pm 0.51}40.18±0.4340.18_{\pm 0.43}
k=101.685±0.0051.685_{\pm 0.005}15.97±1.0415.97_{\pm 1.04}44.06±1.4744.06_{\pm 1.47}40.57±0.3440.57_{\pm 0.34}
LightMTP-S1.684±0.0001.684_{\pm 0.000}16.28±0.4816.28_{\pm 0.48}43.31±0.7143.31_{\pm 0.71}40.13±0.6740.13_{\pm 0.67}
+VarReg1.694±0.0021.694_{\pm 0.002}17.53±0.3517.53_{\pm 0.35}42.33±0.1442.33_{\pm 0.14}40.29±0.4640.29_{\pm 0.46}
k=101.685±0.0011.685_{\pm 0.001}15.92±0.4615.92_{\pm 0.46}43.97±0.1643.97_{\pm 0.16}40.19±0.3440.19_{\pm 0.34}

Key findings:

  • All token-level MTP baselines (Meta, Meta-L, DeepSeek) regress relative to NTP on every metric.
  • LightMTP-C matches NTP exactly on FineWeb-Edu perplexity; most variants stay within 0.01 of NTP.
  • LightMTP-C with k=10 achieves the best CORE-avg score overall.
  • VarReg consistently harms general language modeling despite improving anticipation.

5.3 Planning: Countdown

Table 2: Countdown strict accuracy on held-out targets (%)

ObjectiveHeld-out targets
NTP46.4±1.446.4 \pm 1.4
Meta, k=451.4±3.151.4 \pm 3.1
Meta, k=858.1±0.558.1 \pm 0.5
Meta-L, k=455.8±0.955.8 \pm 0.9
Meta-L, k=858.9±1.258.9 \pm 1.2
DeepSeek, k=147.8±2.547.8 \pm 2.5
DeepSeek, k=455.4±0.455.4 \pm 0.4
DeepSeek, k=855.1±0.355.1 \pm 0.3
Future Summaries‡34.9±15.534.9 \pm 15.5
LightMTP-C, k=150.1±2.650.1 \pm 2.6
LightMTP-C, k=454.0±3.054.0 \pm 3.0
LightMTP-C, k=856.1±1.456.1 \pm 1.4
LightMTP-S, k=150.6±0.950.6 \pm 0.9
LightMTP-S, k=455.5±1.255.5 \pm 1.2
LightMTP-S, k=856.7±0.956.7 \pm 0.9

‡ Held-out strict accuracy destabilizes late in training on 2 of 3 seeds.

  • Both LightMTP variants substantially outperform NTP and improve consistently with increasing kk.
  • LightMTP-S outperforms LightMTP-C at every kk with lower variance at the largest horizon, using only 1.2M MTP parameters vs. 75M for Meta-L (98% reduction).
  • LightMTP-S trails Meta's best by only 2.2 points (56.7% vs. 58.9%).
  • DeepSeek is the only method where additional lookahead is counterproductive (declines at k=8).
  • Solution-space analysis shows MTP models preserve more paths to the target throughout generation.

5.4 Structure: GSM8K and MBPP

Table 3: GSM8K results (relaxed = gold answer anywhere; strict = correct answer format)

Methodp@5 relaxedp@5 strict
NTP41.97 ± 2.294.94 ± 1.44
Meta41.45 ± 1.434.09 ± 0.08
Meta-L41.91 ± 0.884.82 ± 1.19
DeepSeek MTP39.17 ± 2.113.85 ± 0.68
Future Summaries41.19 ± 0.534.95 ± 1.25
LightMTP-C43.62 ± 1.514.98 ± 0.91
LightMTP-S43.82 ± 1.335.69 ± 1.09

Table 4: MBPP code-generation pass@k (%) (d24 python-mix models, k=4)

Methodzero-shot p@5zero-shot p@10finetuned p@5finetuned p@10
NTP5.02 ± 0.978.52 ± 1.7011.39 ± 1.2215.19 ± 2.31
Meta4.80 ± 1.807.04 ± 2.3113.22 ± 2.6417.78 ± 3.85
Meta-L6.01 ± 2.109.26 ± 2.8016.43 ± 0.4821.85 ± 0.64
DeepSeek0.19 ± 0.320.37 ± 0.642.47 ± 1.903.70 ± 2.57
Fut. Sum.6.57 ± 0.638.89 ± 1.1115.62 ± 1.1421.11 ± 1.11
LightMTP-C6.94 ± 0.109.63 ± 0.6416.42 ± 1.1823.33 ± 2.94
LightMTP-S5.30 ± 2.288.89 ± 4.4412.55 ± 1.6317.78 ± 2.22
  • LightMTP-C achieves best zero-shot MBPP performance at both pass@5 and pass@10.
  • DeepSeek MTP collapses on code (near zero zero-shot), indicating poor transfer.
  • All MTP methods exhibit fewer structural errors (NameError, SyntaxError, IndentationError, ModuleNotFoundError) than NTP, supporting the hypothesis that MTP teaches local look-ahead skills.

Theoretical and Practical Implications

Theoretical Significance

  1. Latent supervision without external models: LightMTP demonstrates that a model's own intermediate hidden states can serve as effective supervision targets, eliminating the need for auxiliary encoders or helper models—a key architectural simplification over prior latent MTP approaches.

  2. Anticipation–NTP tradeoff: The paper introduces the concept of anticipation cost, providing a principled framework for quantifying the tradeoff between future-token anticipation and next-token performance. This metric could guide future objective design.

  3. Intermediate-layer targets: Taking targets from an intermediate layer (l=16l = 16 of 24) rather than the final layer provides a more favorable tradeoff, suggesting that representation depth matters for MTP supervision quality.

Practical Implications

  1. Scalability at smaller model sizes: LightMTP works at 1.4–1.5B parameters, a regime where token-level MTP methods fail. This makes it practical as a general-purpose pretraining objective for smaller models.

  2. Computational efficiency: Adding at most 1% parameters with no inference overhead makes LightMTP nearly free to adopt in existing training pipelines.

  3. Structural task improvement: Consistent gains on planning (Countdown), reasoning (GSM8K), and code generation (MBPP) with reduced structural errors suggest LightMTP improves local coherence and format adherence without sacrificing general capabilities.

  4. Limitations of VarReg: Variance regularization improves anticipation but consistently harms downstream performance, suggesting gradient interference between the anticipation objective and NTP—a caution for future regularization approaches.


Conclusion

LightMTP introduces a lightweight latent multi-token prediction objective that uses a model's own future hidden states for supervision via a small auxiliary head. By predicting in latent space rather than over the vocabulary, it avoids the compute and memory overhead of token-level MTP, adding at most 1% extra parameters and leaving the trunk and inference path untouched.

The method balances between anticipating future tokens and retaining next-token performance better than all considered MTP baselines, making it particularly relevant at smaller model scales (< 3B)—a regime where prior MTP work either reports degraded language modeling or omits entirely.

Future directions include:

  • Systematically characterizing the choices affecting anticipation cost (target granularity, prediction horizon, model scale).
  • Providing a principled basis for designing multi-token objectives rather than selecting them empirically.
  • Exploring whether the approach scales to larger models and whether intermediate-layer supervision continues to provide favorable tradeoffs.

Related papers