Summary of NCP-ArchPreview: Moving towards Latent Space Language Models through Next Concept Prediction

Summary (Overview)

• NCP-ArchPreview is an 8.9B-parameter latent-space language model that introduces Next Concept Prediction (NCP) alongside standard Next Token Prediction (NTP), trained on 5.73T tokens from Dolma-3 — the largest demonstration of latent-space language modeling to date.

• The model achieves 1.95× faster convergence (matching OLMo-3-7B's final loss with only 51.3% of training tokens) and 2.45 points higher downstream macro-average, including a notable +5.99 gain on GSM8K.

• The architecture combines a 16-layer Token Encoder, 8-layer Concept Module (using product quantization for discrete concept vocabulary), and 16-layer Token Decoder, with hierarchical residual connections (IRC + CRC) for cross-module information flow.

• Compute efficiency gains: NCP-ArchPreview achieves a 1.74× Pareto compute efficiency improvement over standard training, approaching a parameter-aligned 40-layer baseline with only 85% of its computation.

• The learned concept space enables lightweight domain adaptation via VQ-only training (17M parameters, 2.02× faster than full training), and improves speculative decoding acceptance length by 4.17% when integrated into a DFlash2 drafter.

Introduction and Theoretical Foundation

Background and Motivation

Standard Next Token Prediction (NTP) supervises only granular token-level predictions, leaving high-level semantic abstractions to emerge as indirect byproducts. The authors argue that explicit concept-level objectives are needed to guide how semantic structure unfolds across multi-token spansches.

Theoretical Basis

The work builds on several research threads:

• Latent diffusion in visual synthesis (Rombach et al., 2022) demonstrated that modeling in compact continuous representations dramatically improves efficiency.
• Joint-Embedding Predictive Architectures (JEPA) (LeCun et al., 2022) predict future representations in latent space to capture invariant semantic structures.
• Multi-Token Prediction (MTP) (Gloeckle et al., 2024) supervises multiple future positions but remains tethered to surface tokens.
• ConceptLM (Liu et al., 2026) introduced NCP with a learned discrete concept vocabulary, but only evaluated up to 1.5B parameters.

Key Distinctions from Prior Work

Prior hierarchical approaches differ in two dimensions:

1. How latent representations are formed: Fixed hierarchies (Hourglass Transformer, ContextLM, MegaByte) vs. dynamic/input-adaptive chunking (BLT, DLCM, H-Net)
2. What they predict: Continuous embeddings (Large Concept Model, JEPA variants) vs. discrete concepts (ConceptLM)

NCP-ArchPreview uniquely combines:

• Product-quantized discrete concept vocabulary learned from hidden states
• Joint NTP + NCP training from the start of pre-training
• Trillion-token scale (5.73T) validation

Methodology

Architecture Overview

The model organizes the backbone into three modules:

1. Token Encoder (16 layers): Produces token-level hidden states $h_{1:T}$
2. Concept Module (8 layers): Predicts future concepts at a compressed sequence level
3. Token Decoder (16 layers): Consumes fused token + concept representations for next-token prediction

Concept Compression and Quantization

Token states are compressed into concepts via mean pooling with compression factor $k=4$:

$$\mathbf{c}_m = f_c(\mathbf{h}_{(m-1)k+1:mk}) = \frac{1}{k}\sum_{i=1}^{k}\mathbf{h}_{(m-1)k+i}, \qquad \mathbf{c}_m \in \mathbb{R}^d$$

Product Quantization partitions each concept into $S$ segments, each with its own codebook of $N$ entries:

$$\mathbf{c}_m = \text{concat}(\mathbf{c}_m^1, \dots, \mathbf{c}_m^S), \quad \mathbf{c}_m^s \in \mathbb{R}^{d/S}$$

Each segment is assigned to its nearest codeword:

$$n_m^s = \arg\min_{n \in \{1,\dots,N\}} \|\mathbf{c}_m^s - \mathbf{e}_n^s\|_2^2$$

This yields $N^S$ possible concept combinations while keeping codebooks small (32 codebooks × 128 entries).

Next Concept Prediction

The Concept Module predicts probability distributions over codebook entries, forming differentiable predicted concepts via expectation:

$$\hat{\mathbf{c}}_m^s = \sum_{n=1}^{N} \pi_{m,n}^s \mathbf{e}_n^s$$

The NCP loss uses mean-squared error with stop-gradient on targets:

$$\mathcal{L}_{\text{NCP}} = \frac{1}{M-1}\sum_{m=2}^{M}\|\hat{\mathbf{c}}_m - \text{sg}(\mathbf{c}_m)\|_2^2$$

Concept Injection into Token Stream

Predicted concepts are repeated $k$ times, causally shifted by $\Delta = k$, and added to token states:

$$\mathbf{b}_t = \begin{cases} \mathbf{0}, & 1 \leq t < \Delta \\ \hat{\mathbf{c}}_{\lfloor (t-\Delta)/k\rfloor+2}, & \Delta \leq t < T \end{cases}$$ $$\tilde{\mathbf{h}}_t = \mathbf{h}_t + \mathbf{b}_t$$

Hierarchical Residual Connections

• Intra-Module Residual Connections (IRC): Weighted combination of states from multiple depths within a module, learned via lightweight MLPs

$$\mathbf{H}_{\ell+1}^s = \sum_{j=1}^{\ell+1} w_{\ell,j}^s \mathbf{X}_{\ell,j}^s$$

• Cross-Module Residual Connections (CRC): Transfer representations between modules with softmax-weighted combination and diagonal scaling

Joint Training Objective

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{NTP}} + \alpha \mathcal{L}_{\text{NCP}} + \beta \mathcal{L}_{\text{VQ}}$$

Where:

• $\mathcal{L}_{\text{NTP}}$: standard next-token prediction
• $\mathcal{L}_{\text{NCP}}$: next-concept prediction (MSE loss)
• $\mathcal{L}_{\text{VQ}}$: vector quantization codebook fitting

Optimization

Uses Moonlight Muon optimizer for matrix parameters:

$$\mathbf{W}_t = \mathbf{W}_{t-1} - \eta_t\left(\lambda_{\text{Muon}}\mathbf{O}_t\sqrt{\max(d_{\text{in}}, d_{\text{out}})} + \lambda_{wd}\mathbf{W}_{t-1}\right)$$

Empirical Validation / Results

Main Results (Training Loss)

Downstream Performance (Selected results)

Ablation Studies

Progressive component ablation (training loss over 200B tokens):

• Vanilla → Vanilla + Concept Module: loss decreases
• • Residual connections: further decreases
• • NCP loss: additional improvement

Matched parameter/computation baselines:

NCP-ArchPreview outperforms both Vanilla and computation-aligned baselines, approaching size-aligned performance with only 85% of its compute.

Hierarchical Residual Connection Ablation (1B model, 150B tokens)

Scaling Laws

NCP-ArchPreview achieves 1.74× computational efficiency compared to OLMo-3 with compute-optimal training.

Numerical Stability

Identified instability from interaction between full-matrix Muon updates and layer-wise Q/K normalization. Per-head Q/K normalization stabilizes training by preventing outlier attention heads from dominating.

Domain Adaptation via VQ-Only Training

VQ training achieves 2.02× higher throughput than LoRA and 1.50× faster than full training, with lowest memory footprint (32.4% vs 49.0% for LoRA, 90.3% for full).

Speculative Decoding Enhancement

Adding concept representations to a DFlash2 drafter improves mean accepted length by 4.17% (+7.59% on HumanEval) with negligible parameter overhead (0.04M for a 1.1B drafter).

Theoretical and Practical Implications

Theoretical Contributions

1. Latent spaces as first-class prediction targets: Demonstrates that concept-level prediction objectives provide complementary supervision that improves token-level modeling at trillion-token scale.

2. Product-quantized discrete concepts: Provides a structured, bounded prediction space that avoids the "arbitrary vector" problem of continuous regression while enabling differentiability through expectation-based predictions.

3. Hierarchical residual connections: Input-dependent routing of information across abstraction levels outperforms both standard residuals and fixed grouping schemes, with minimal FLOPs overhead.

4. Stability considerations: Reveals important interactions between matrix optimizers (Muon) and normalization strategies that affect all large-scale training efforts.

Practical Implications

1. Compute efficiency: 1.74× Pareto efficiency suggests latent-space modeling could reduce the cost of large-scale pre-training.

2. Lightweight adaptation: VQ-only training offers a highly efficient interface (17M parameters, 2× throughput) for domain adaptation with less catastrophic forgetting than full or LoRA fine-tuning.

3. Speculative decoding: Concept representations improve draft acceptance with negligible overhead, enabling faster inference.

4. Data efficiency: 1.95× faster convergence suggests equivalent performance with roughly half the training data.

Conclusion

NCP-ArchPreview establishes that latent representations learned by a language model can serve as first-class prediction targets at trillion-token scale. The architecture jointly models tokens and learned discrete concepts while preserving standard autoregressive generation, achieving:

• Lower training loss with 1.95× faster convergence
• +2.45 downstream macro-average improvement
• 1.74× compute efficiency gains
• Effective lightweight adaptation and speculative decoding benefits

Future directions include:

• Extending to long-context training (where compressed concept sequences may provide particular advantages)
• Bridging the gap between lower language-modeling loss and downstream capability (which varies across stages and domains)
• Exploring capability-aware data selection and combined checkpoint selection criteria

The results position latent-space prediction not merely as an auxiliary objective, but as a highly efficient and scalable architectural blueprint for next-generation foundation models.