# NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

> NCP-ArchPreview, an 8.9B latent-space language model adding next-concept prediction, achieves 1.95x faster convergence and +2.45 average downstream gains over standard 7B training at 5.73T tokens.

- **Source:** [arXiv](https://arxiv.org/abs/2609.10715)
- **Published:** 2026-09-12
- **Permalink:** https://picx.dev/p/k8KVc2
- **Whiteboard:** https://picx.dev/p/k8KVc2/image

## Summary

# Summary of NCP-ArchPreview: Moving towards Latent Space Language Models through Next Concept Prediction

## Summary (Overview)

- **NCP-ArchPreview** is an 8.9B-parameter latent-space language model that introduces **Next Concept Prediction (NCP)** alongside standard Next Token Prediction (NTP), trained on 5.73T tokens from Dolma-3 — the largest demonstration of latent-space language modeling to date.

- The model achieves **1.95× faster convergence** (matching OLMo-3-7B's final loss with only 51.3% of training tokens) and **2.45 points higher downstream macro-average**, including a notable **+5.99 gain on GSM8K**.

- The architecture combines a 16-layer Token Encoder, 8-layer Concept Module (using **product quantization** for discrete concept vocabulary), and 16-layer Token Decoder, with **hierarchical residual connections** (IRC + CRC) for cross-module information flow.

- **Compute efficiency gains**: NCP-ArchPreview achieves a **1.74× Pareto compute efficiency improvement** over standard training, approaching a parameter-aligned 40-layer baseline with only 85% of its computation.

- The learned concept space enables lightweight **domain adaptation via VQ-only training** (17M parameters, 2.02× faster than full training), and improves **speculative decoding acceptance length by 4.17%** when integrated into a DFlash2 drafter.

## Introduction and Theoretical Foundation

### Background and Motivation

Standard Next Token Prediction (NTP) supervises only granular token-level predictions, leaving high-level semantic abstractions to emerge as indirect byproducts. The authors argue that explicit concept-level objectives are needed to guide how semantic structure unfolds across multi-token spansches.

### Theoretical Basis

The work builds on several research threads:

- **Latent diffusion** in visual synthesis (Rombach et al., 2022) demonstrated that modeling in compact continuous representations dramatically improves efficiency.
- **Joint-Embedding Predictive Architectures (JEPA)** (LeCun et al., 2022) predict future representations in latent space to capture invariant semantic structures.
- **Multi-Token Prediction (MTP)** (Gloeckle et al., 2024) supervises multiple future positions but remains tethered to surface tokens.
- **ConceptLM** (Liu et al., 2026) introduced NCP with a learned discrete concept vocabulary, but only evaluated up to 1.5B parameters.

### Key Distinctions from Prior Work

Prior hierarchical approaches differ in two dimensions:
1. **How latent representations are formed**: Fixed hierarchies (Hourglass Transformer, ContextLM, MegaByte) vs. dynamic/input-adaptive chunking (BLT, DLCM, H-Net)
2. **What they predict**: Continuous embeddings (Large Concept Model, JEPA variants) vs. discrete concepts (ConceptLM)

NCP-ArchPreview uniquely combines:
- Product-quantized discrete concept vocabulary learned from hidden states
- Joint NTP + NCP training from the start of pre-training
- Trillion-token scale (5.73T) validation

## Methodology

### Architecture Overview

The model organizes the backbone into three modules:

1. **Token Encoder** (16 layers): Produces token-level hidden states $h_{1:T}$
2. **Concept Module** (8 layers): Predicts future concepts at a compressed sequence level
3. **Token Decoder** (16 layers): Consumes fused token + concept representations for next-token prediction

### Concept Compression and Quantization

Token states are compressed into concepts via **mean pooling** with compression factor $k=4$:

$$
\mathbf{c}_m = f_c(\mathbf{h}_{(m-1)k+1:mk}) = \frac{1}{k}\sum_{i=1}^{k}\mathbf{h}_{(m-1)k+i}, \qquad \mathbf{c}_m \in \mathbb{R}^d
$$

**Product Quantization** partitions each concept into $S$ segments, each with its own codebook of $N$ entries:

$$
\mathbf{c}_m = \text{concat}(\mathbf{c}_m^1, \dots, \mathbf{c}_m^S), \quad \mathbf{c}_m^s \in \mathbb{R}^{d/S}
$$

Each segment is assigned to its nearest codeword:
$$
n_m^s = \arg\min_{n \in \{1,\dots,N\}} \|\mathbf{c}_m^s - \mathbf{e}_n^s\|_2^2
$$

This yields $N^S$ possible concept combinations while keeping codebooks small (32 codebooks × 128 entries).

### Next Concept Prediction

The Concept Module predicts probability distributions over codebook entries, forming **differentiable predicted concepts** via expectation:

$$
\hat{\mathbf{c}}_m^s = \sum_{n=1}^{N} \pi_{m,n}^s \mathbf{e}_n^s
$$

The NCP loss uses **mean-squared error** with stop-gradient on targets:

$$
\mathcal{L}_{\text{NCP}} = \frac{1}{M-1}\sum_{m=2}^{M}\|\hat{\mathbf{c}}_m - \text{sg}(\mathbf{c}_m)\|_2^2
$$

### Concept Injection into Token Stream

Predicted concepts are repeated $k$ times, causally shifted by $\Delta = k$, and added to token states:

$$
\mathbf{b}_t = \begin{cases}
\mathbf{0}, & 1 \leq t < \Delta \\
\hat{\mathbf{c}}_{\lfloor (t-\Delta)/k\rfloor+2}, & \Delta \leq t < T
\end{cases}
$$

$$
\tilde{\mathbf{h}}_t = \mathbf{h}_t + \mathbf{b}_t
$$

### Hierarchical Residual Connections

- **Intra-Module Residual Connections (IRC)**: Weighted combination of states from multiple depths within a module, learned via lightweight MLPs

$$
\mathbf{H}_{\ell+1}^s = \sum_{j=1}^{\ell+1} w_{\ell,j}^s \mathbf{X}_{\ell,j}^s
$$

- **Cross-Module Residual Connections (CRC)**: Transfer representations between modules with softmax-weighted combination and diagonal scaling

### Joint Training Objective

$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{NTP}} + \alpha \mathcal{L}_{\text{NCP}} + \beta \mathcal{L}_{\text{VQ}}
$$

Where:
- $\mathcal{L}_{\text{NTP}}$: standard next-token prediction
- $\mathcal{L}_{\text{NCP}}$: next-concept prediction (MSE loss)
- $\mathcal{L}_{\text{VQ}}$: vector quantization codebook fitting

### Optimization

Uses **Moonlight Muon** optimizer for matrix parameters:

$$
\mathbf{W}_t = \mathbf{W}_{t-1} - \eta_t\left(\lambda_{\text{Muon}}\mathbf{O}_t\sqrt{\max(d_{\text{in}}, d_{\text{out}})} + \lambda_{wd}\mathbf{W}_{t-1}\right)
$$

## Empirical Validation / Results

### Main Results (Training Loss)

| Metric | OLMo-3-7B | NCP-ArchPreview | Gain |
|--------|-----------|-----------------|------|
| Stage-1 final loss | — | 0.091 lower | 1.95× convergence |
| Stage-1 tokens to match | 100% | 51.3% | — |
| Stage-2 loss gap | — | 0.027 lower | 1.51× convergence |
| Stage-2 tokens to match | 100% | 66.2% | — |

### Downstream Performance (Selected results)

| Domain | Stage-1 Vanilla | Stage-1 NCP | Δ | Stage-2 Vanilla | Stage-2 NCP | Δ |
|--------|-----------------|-------------|---|-----------------|-------------|---|
| MMLU Avg | 54.50 | 56.73 | **+2.24** | 58.32 | 59.94 | **+1.62** |
| MATH Avg | 20.79 | 24.54 | **+3.75** | 55.63 | 57.39 | **+1.76** |
| GSM8K | 39.27 | 45.26 | **+5.99** | 79.68 | 83.02 | **+3.34** |
| Code Avg | 25.15 | 27.79 | **+2.64** | 39.42 | 38.77 | -0.65 |
| MC-Non-STEM Avg | 70.08 | 74.71 | **+4.63** | 76.85 | 77.76 | **+0.91** |
| **Overall Avg** | **46.59** | **49.04** | **+2.45** | **56.98** | **57.57** | **+0.59** |

### Ablation Studies

**Progressive component ablation** (training loss over 200B tokens):
- Vanilla → Vanilla + Concept Module: loss decreases
- + Residual connections: further decreases
- + NCP loss: additional improvement

**Matched parameter/computation baselines**:

| Model | Parameters | Computation |
|-------|-----------|-------------|
| NCP-ArchPreview | 40P_blk | 34F_blk |
| Vanilla | 32P_blk | 32F_blk |
| Vanilla size-aligned | 40P_blk | 40F_blk |
| Vanilla computation-aligned | 34P_blk | 34F_blk |

NCP-ArchPreview outperforms both Vanilla and computation-aligned baselines, approaching size-aligned performance with only 85% of its compute.

### Hierarchical Residual Connection Ablation (1B model, 150B tokens)

| Variant | Δ Loss | Δ FLOPs |
|---------|--------|---------|
| No hierarchical residuals | 0.0000 | 0.000% |
| IRC + CRC | **-0.0323** | +0.051% |
| IRC only | -0.0273 | +0.024% |
| Block AttnRes + CRC | -0.0180 | +0.026% |

### Scaling Laws

NCP-ArchPreview achieves **1.74× computational efficiency** compared to OLMo-3 with compute-optimal training.

### Numerical Stability

Identified instability from interaction between **full-matrix Muon updates** and **layer-wise Q/K normalization**. Per-head Q/K normalization stabilizes training by preventing outlier attention heads from dominating.

### Domain Adaptation via VQ-Only Training

| Method | Trainable Params | Code Avg Gain | Math Avg Gain | General Avg Δ |
|--------|-----------------|---------------|---------------|---------------|
| +Full | 8.9B | +0.11 (from code) | +6.16 | -0.48 |
| +LoRA | 17M | +1.34 | +3.15 | -0.42 |
| +VQ | 17M | **+2.65** | **+4.27** | **+0.39** |

VQ training achieves 2.02× higher throughput than LoRA and 1.50× faster than full training, with lowest memory footprint (32.4% vs 49.0% for LoRA, 90.3% for full).

### Speculative Decoding Enhancement

Adding concept representations to a DFlash2 drafter improves **mean accepted length by 4.17%** (+7.59% on HumanEval) with negligible parameter overhead (0.04M for a 1.1B drafter).

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Latent spaces as first-class prediction targets**: Demonstrates that concept-level prediction objectives provide complementary supervision that improves token-level modeling at trillion-token scale.

2. **Product-quantized discrete concepts**: Provides a structured, bounded prediction space that avoids the "arbitrary vector" problem of continuous regression while enabling differentiability through expectation-based predictions.

3. **Hierarchical residual connections**: Input-dependent routing of information across abstraction levels outperforms both standard residuals and fixed grouping schemes, with minimal FLOPs overhead.

4. **Stability considerations**: Reveals important interactions between matrix optimizers (Muon) and normalization strategies that affect all large-scale training efforts.

### Practical Implications

1. **Compute efficiency**: 1.74× Pareto efficiency suggests latent-space modeling could reduce the cost of large-scale pre-training.

2. **Lightweight adaptation**: VQ-only training offers a highly efficient interface (17M parameters, 2× throughput) for domain adaptation with less catastrophic forgetting than full or LoRA fine-tuning.

3. **Speculative decoding**: Concept representations improve draft acceptance with negligible overhead, enabling faster inference.

4. **Data efficiency**: 1.95× faster convergence suggests equivalent performance with roughly half the training data.

## Conclusion

NCP-ArchPreview establishes that latent representations learned by a language model can serve as first-class prediction targets at trillion-token scale. The architecture jointly models tokens and learned discrete concepts while preserving standard autoregressive generation, achieving:
- Lower training loss with 1.95× faster convergence
- +2.45 downstream macro-average improvement
- 1.74× compute efficiency gains
- Effective lightweight adaptation and speculative decoding benefits

**Future directions** include:
- Extending to long-context training (where compressed concept sequences may provide particular advantages)
- Bridging the gap between lower language-modeling loss and downstream capability (which varies across stages and domains)
- Exploring capability-aware data selection and combined checkpoint selection criteria

The results position **latent-space prediction not merely as an auxiliary objective, but as a highly efficient and scalable architectural blueprint** for next-generation foundation models.

---

_Markdown view of https://picx.dev/p/k8KVc2, served by PicX — AI-generated visual whiteboard summaries of research papers._