Summary (Overview)
- LightMTP is a novel lightweight latent multi-token prediction (MTP) pretraining objective that uses the model's own future hidden states as supervision targets, eliminating the need for auxiliary encoders, codebooks, or helper models.
- Two variants are proposed: LightMTP-S (single-target vector prediction) and LightMTP-C (concatenated multi-target vector prediction), both adding at most 1% extra parameters and being discarded after training (inference unchanged).
- LightMTP retains NTP-level performance on general language modeling benchmarks (FineWeb-Edu, LAMBADA, CORE), unlike token-level MTP baselines (Meta, Meta-L, DeepSeek) which consistently degrade.
- On the Countdown planning task, LightMTP improves over NTP by up to 10 points (56.7% vs. 46.4% at k=8) while using 98% fewer MTP parameters than Meta-L (1.2M vs. 75M).
- LightMTP demonstrates superior anticipation cost (next-token degradation per unit of future-token gain), with the top four most cost-efficient models being LightMTP variants, outperforming baselines by up to two orders of magnitude.
Introduction and Theoretical Foundation
Background and Motivation
Next-token prediction (NTP) is the standard pretraining objective for large language models, requiring no supervision beyond raw text and computing loss at every position in a single teacher-forced forward pass. However, NTP provides an explicit training signal only for the immediate next token. Information about more distant tokens is learned only insofar as it helps predict the next one, which can lead models to exploit local patterns instead of capturing longer-range structure. Bachmann & Nagarajan (2024) show that teacher-forced NTP can fail to learn even simple planning tasks.
Multi-Token Prediction (MTP)
Multi-token prediction addresses this limitation by additionally training the model to predict tokens from its representation at position . This densifies the training signal and rewards representations that encode information beyond the next token (Gloeckle et al., 2024; DeepSeek-AI et al., 2025). However, token-level MTP methods face key challenges:
- Computational cost: Each offset adds a vocabulary projection, so compute and memory grow with both and vocabulary size.
- Parameter overhead: Many designs add highly parametrized transformer layers per offset.
- Interference with NTP: Gloeckle et al. (2024) find MTP hurts models below roughly 3B parameters.
Latent MTP
Latent MTP methods avoid the vocabulary projection by predicting dense representations of future tokens. However, existing approaches (Semformer, Next-Concept Prediction, Future Summaries) rely on auxiliary encoders, codebooks, or separately trained helper models, adding parameters and training stages.
LightMTP addresses these issues by bootstrapping targets from the model's own hidden states, requiring no auxiliary model and avoiding vocabulary projection entirely.
Methodology
Architecture
A transformer trunk with layers maps a prefix to hidden states for . Standard NTP trains a linear head on with cross-entropy loss . LightMTP adds a second head (a two-layer MLP with GELU nonlinearity) that reads and predicts hidden states the trunk produces at future positions.
Targets and Loss
Targets: The trunk's own hidden states at a chosen target layer , taken at future position . Gradients are stopped through the target branch: .
Cosine Loss: Both predicted and target vectors are normalized:
Variance Regularization: To prevent representational collapse (all positions becoming similar), a within-sequence regularizer penalizes cross-positional similarity:
Overall Loss:
LightMTP-S: Single-Target Vector Prediction
Predicts a single future representation steps ahead:
LightMTP-C: Concatenated Multi-Target Vector Prediction
Predicts all future hidden states jointly via concatenation:
Per-step losses with masking for sequence boundaries:
Experimental Setup
- Models: 24-layer decoder-only models with 1.4–1.5B parameters (a scale where MTP often fails).
- Baselines: NTP, Meta-style MTP, Meta-L (larger), DeepSeek-style MTP, Future Summaries.
- LightMTP configuration: Target layer (two-thirds of depth), horizon (also tested ).
- Token budgets: Parameter-dependent, with Meta (+20.7%/+11.6%) and DeepSeek (+13.6%) seeing more tokens.
- Batch size: 524,288 tokens; tokenizer size 32,768.
Empirical Validation / Results
5.1 Future Token Anticipation
Linear probing on frozen last hidden-layer outputs reveals a tradeoff: MTP methods improve future-token prediction () at a cost to NTP (). The anticipation cost quantifies this:
Four LightMTP variants exhibit the lowest anticipation cost of all methods, with Future Summaries and DeepSeek having costs two orders of magnitude higher.
5.2 Language Modeling Performance
Table 1: General LM benchmark results (best bolded, runner-up underlined)
| Method | FineWeb-Edu (PPL) | Lambada (PPL) | Lambada (Acc) | CORE-avg |
|---|---|---|---|---|
| NTP | ||||
| Meta | ||||
| Meta-L | ||||
| DeepSeek | ||||
| Future Summaries | ||||
| LightMTP-C | ||||
| +VarReg | ||||
| k=10 | ||||
| LightMTP-S | ||||
| +VarReg | ||||
| k=10 |
Key findings:
- All token-level MTP baselines (Meta, Meta-L, DeepSeek) regress relative to NTP on every metric.
- LightMTP-C matches NTP exactly on FineWeb-Edu perplexity; most variants stay within 0.01 of NTP.
- LightMTP-C with k=10 achieves the best CORE-avg score overall.
- VarReg consistently harms general language modeling despite improving anticipation.
5.3 Planning: Countdown
Table 2: Countdown strict accuracy on held-out targets (%)
| Objective | Held-out targets |
|---|---|
| NTP | |
| Meta, k=4 | |
| Meta, k=8 | |
| Meta-L, k=4 | |
| Meta-L, k=8 | |
| DeepSeek, k=1 | |
| DeepSeek, k=4 | |
| DeepSeek, k=8 | |
| Future Summaries‡ | |
| LightMTP-C, k=1 | |
| LightMTP-C, k=4 | |
| LightMTP-C, k=8 | |
| LightMTP-S, k=1 | |
| LightMTP-S, k=4 | |
| LightMTP-S, k=8 |
‡ Held-out strict accuracy destabilizes late in training on 2 of 3 seeds.
- Both LightMTP variants substantially outperform NTP and improve consistently with increasing .
- LightMTP-S outperforms LightMTP-C at every with lower variance at the largest horizon, using only 1.2M MTP parameters vs. 75M for Meta-L (98% reduction).
- LightMTP-S trails Meta's best by only 2.2 points (56.7% vs. 58.9%).
- DeepSeek is the only method where additional lookahead is counterproductive (declines at k=8).
- Solution-space analysis shows MTP models preserve more paths to the target throughout generation.
5.4 Structure: GSM8K and MBPP
Table 3: GSM8K results (relaxed = gold answer anywhere; strict = correct answer format)
| Method | p@5 relaxed | p@5 strict |
|---|---|---|
| NTP | 41.97 ± 2.29 | 4.94 ± 1.44 |
| Meta | 41.45 ± 1.43 | 4.09 ± 0.08 |
| Meta-L | 41.91 ± 0.88 | 4.82 ± 1.19 |
| DeepSeek MTP | 39.17 ± 2.11 | 3.85 ± 0.68 |
| Future Summaries | 41.19 ± 0.53 | 4.95 ± 1.25 |
| LightMTP-C | 43.62 ± 1.51 | 4.98 ± 0.91 |
| LightMTP-S | 43.82 ± 1.33 | 5.69 ± 1.09 |
Table 4: MBPP code-generation pass@k (%) (d24 python-mix models, k=4)
| Method | zero-shot p@5 | zero-shot p@10 | finetuned p@5 | finetuned p@10 |
|---|---|---|---|---|
| NTP | 5.02 ± 0.97 | 8.52 ± 1.70 | 11.39 ± 1.22 | 15.19 ± 2.31 |
| Meta | 4.80 ± 1.80 | 7.04 ± 2.31 | 13.22 ± 2.64 | 17.78 ± 3.85 |
| Meta-L | 6.01 ± 2.10 | 9.26 ± 2.80 | 16.43 ± 0.48 | 21.85 ± 0.64 |
| DeepSeek | 0.19 ± 0.32 | 0.37 ± 0.64 | 2.47 ± 1.90 | 3.70 ± 2.57 |
| Fut. Sum. | 6.57 ± 0.63 | 8.89 ± 1.11 | 15.62 ± 1.14 | 21.11 ± 1.11 |
| LightMTP-C | 6.94 ± 0.10 | 9.63 ± 0.64 | 16.42 ± 1.18 | 23.33 ± 2.94 |
| LightMTP-S | 5.30 ± 2.28 | 8.89 ± 4.44 | 12.55 ± 1.63 | 17.78 ± 2.22 |
- LightMTP-C achieves best zero-shot MBPP performance at both pass@5 and pass@10.
- DeepSeek MTP collapses on code (near zero zero-shot), indicating poor transfer.
- All MTP methods exhibit fewer structural errors (NameError, SyntaxError, IndentationError, ModuleNotFoundError) than NTP, supporting the hypothesis that MTP teaches local look-ahead skills.
Theoretical and Practical Implications
Theoretical Significance
-
Latent supervision without external models: LightMTP demonstrates that a model's own intermediate hidden states can serve as effective supervision targets, eliminating the need for auxiliary encoders or helper models—a key architectural simplification over prior latent MTP approaches.
-
Anticipation–NTP tradeoff: The paper introduces the concept of anticipation cost, providing a principled framework for quantifying the tradeoff between future-token anticipation and next-token performance. This metric could guide future objective design.
-
Intermediate-layer targets: Taking targets from an intermediate layer ( of 24) rather than the final layer provides a more favorable tradeoff, suggesting that representation depth matters for MTP supervision quality.
Practical Implications
-
Scalability at smaller model sizes: LightMTP works at 1.4–1.5B parameters, a regime where token-level MTP methods fail. This makes it practical as a general-purpose pretraining objective for smaller models.
-
Computational efficiency: Adding at most 1% parameters with no inference overhead makes LightMTP nearly free to adopt in existing training pipelines.
-
Structural task improvement: Consistent gains on planning (Countdown), reasoning (GSM8K), and code generation (MBPP) with reduced structural errors suggest LightMTP improves local coherence and format adherence without sacrificing general capabilities.
-
Limitations of VarReg: Variance regularization improves anticipation but consistently harms downstream performance, suggesting gradient interference between the anticipation objective and NTP—a caution for future regularization approaches.
Conclusion
LightMTP introduces a lightweight latent multi-token prediction objective that uses a model's own future hidden states for supervision via a small auxiliary head. By predicting in latent space rather than over the vocabulary, it avoids the compute and memory overhead of token-level MTP, adding at most 1% extra parameters and leaving the trunk and inference path untouched.
The method balances between anticipating future tokens and retaining next-token performance better than all considered MTP baselines, making it particularly relevant at smaller model scales (< 3B)—a regime where prior MTP work either reports degraded language modeling or omits entirely.
Future directions include:
- Systematically characterizing the choices affecting anticipation cost (target granularity, prediction horizon, model scale).
- Providing a principled basis for designing multi-token objectives rather than selecting them empirically.
- Exploring whether the approach scales to larger models and whether intermediate-layer supervision continues to provide favorable tradeoffs.
Related papers
- Do Halting Heads Really Save Compute in Looped Language Models?
Halting heads in looped language models save far less compute than reported because mixture-trained heads rarely stop tokens and batching overhead eliminates most remaining gains.
- Inside the Loop: A Mechanistic Study of Looped Transformers on Permutation Traversal
Looped transformers solve sequential algorithmic tasks untied models of equal effective depth cannot, by executing one algorithm step per iteration without intermediate supervision.
- Rethinking Fixed-Horizon Latent Reasoning
Adaptive latent inference that stops reasoning when progress stalls improves accuracy by up to 7.2 points while cutting computation by 29%, as completing fixed latent horizons can degrade performance.