# Rethinking Pre-pretraining

> Synthetic pre-pretraining never beats compute-matched natural-language training; reported gains stem from suboptimal baselines and are forgotten across the phase transition.

- **Source:** [arXiv](https://arxiv.org/abs/2610.40051)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/TRrzNr
- **Whiteboard:** https://picx.dev/p/TRrzNr/image

## Summary

# Rethinking Pre-Pretraining

## Summary (Overview)

- **Core finding**: Allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in compute-matched experiments across four PPT curricula (k-Dyck, neural cellular automata (NCA), declarative reference corpora, and code) and three training regimes.
- **Initialization sensitivity**: PPT gains reported in prior work are sensitive to weight initialization schemes; an alternative scheme improves both methods, but the no-PPT baseline more, closing the gap.
- **Additive compute**: PPT can help at small natural-language budgets (50M–100M tokens) but the advantage disappears by 1B tokens.
- **Forgetting mechanism**: Models forget what they learn during PPT when transitioning to natural-language training; interleaving prevents forgetting but does not significantly improve natural-language performance.
- **Recommendation**: Results motivate rethinking the PPT phase (front-loading synthetic data) rather than ruling out synthetic data entirely; gradual integration through interleaving may be preferable.

## Introduction and Theoretical Foundation

The paper addresses a fundamental question in language model training: can synthetic data provide useful linguistic and computational priors that transfer to natural language?

**Motivation**: 
- Natural language scaling benefits haven't hit a ceiling, but increasingly large data is needed (Kaplan et al., 2020)
- Humans acquire language from far less linguistic experience (Warstadt et al., 2023)
- Reasoning abilities are largely dissociable from linguistic proficiency (Fedorenko et al., 2024)
- Synthetic distributions supply effectively unlimited training data once high-quality natural language becomes scarce

**Pre-pretraining (PPT)** is operationalized as a training phase on synthetic data that precedes conventional pretraining (PT), intended to produce an initialization with useful priors for learning language. Different synthetic distributions target different computational priors:

- **k-Dyck**: encourages hierarchical dependency tracking (Hu et al., 2025)
- **Neural cellular automata (NCA)**: expose models to latent transition rules and long-range dependencies (Lee et al., 2026)
- **Procedural tasks**: train operations such as copying, sorting, and retrieval (Jiang et al., 2026)

Prior work reported gains including up to 8.6% lower perplexity for NCA, both when PPT receives additional compute and when it shares a fixed total compute budget.

**Key theoretical questions**:
1. Are PPT gains robust across different training configurations and regimes?
2. Does PPT provide useful inductive biases when not competing with natural-language training for compute?
3. Is PPT worth its compute when competing directly with natural-language training?

## Methodology

### Experimental Setup

**Architecture**: All models trained using a fork of NanoChat with:
- Muon optimizer (Jordan et al., 2024; Liu et al., 2025; Karpathy, 2025) for transformer blocks' weight matrices
- AdamW for embedding and unembedding tables
- Hidden size set to 64d for depth d, so parameter count follows from depth alone

**Data**:
- **Natural language**: FineWeb (Penedo et al., 2024), C4 (Raffel et al., 2023), GitHub Code dataset (CodeParrot, 2022)
- **Synthetic**: k-Dyck, NCA (generated using prior work's code), and DeclRef (new synthetic corpus targeting retrieval from context)

### Training Regimes

**Additive compute**: PPT prepended to natural-language training without substituting any of its budget. Model size fixed at 336M parameters; PT budgets {50M, 100M, 1B} tokens; PPT tokens varied over {5M, 20M, 50M, 100M, 200M, 500M, 1B}.

**Compute-matched**: Total compute budget fixed; α gives fraction spent on PPT, with α ∈ {0, 0.01, 0.05, 0.10}. Three configurations:

1. **Data-constrained**: 100M natural-language tokens, $1 \times 10^{18}$ FLOPs total
2. **Data-unconstrained**: abundant natural language, sweeps over $1 \times 10^{18}$, $3 \times 10^{18}$, $9 \times 10^{18}$ FLOPs
3. **Over-Chinchilla**: 881M parameters, 50B tokens (∼57 tokens per parameter, ~12× compute-optimal)

### Weight Initialization Ablation

Three schemes tested:
1. **Hu et al. (2025)**: all parameters from $\mathcal{N}(0, 0.02)$
2. **Lee et al. (2026)**: token embeddings from $\mathcal{N}(0, 1)$, remaining from $\mathcal{N}(0, 0.02)$
3. **Spectral-based (NanoChat)**: linear-layer weights from $\mathcal{N}(0, \sigma)$ with $\sigma = \min(1, \sqrt{d_{out}/d_{in}})/\sqrt{d_{in}}$, token embeddings from $\mathcal{N}(0, 1)$, zero-initialized LM head and residual projections

### Tokenization

- **NCA**: Each grid split into $2 \times 2$ patches, each becoming one token; grids concatenated in time order with start/end tokens
- **k-Dyck**: Opening and closing bracket types each their own token
- **DeclRef**: Keywords, scope brackets, names, and values as unique tokens
- Synthetic languages use their own smaller vocabularies; at phase boundary, vocabulary expanded and embedding table/LM head reinitialized

### Hyperparameter Optimization

- **LLMTuna**: language-model-based hyperparameter optimizer (released with paper)
- Found more sample-efficient than Bayesian optimization (Optuna)
- Model size searched separately for every method and budget in data-constrained and data-unconstrained regimes

### Evaluation

- **Primary metric**: Bits per byte (BPB) on held-out FineWeb split (~10M tokens)
- **Secondary**: DCLM CORE suite (Li et al., 2024), BLiMP (Warstadt et al., 2020)
- All metrics reported as mean over three seeds with standard deviation

## Empirical Validation / Results

### 1. Initialization Sensitivity (Reproduction)

Reproducing Lee et al. (2026)'s NCA experiment with a 400M-parameter Llama model on ~9B OpenWebText tokens:

- Under original initialization (N(0,1) embeddings): NCA PPT improves perplexity by **5.4%**
- Changing embeddings to N(0, 0.02): baseline improves by **9.0%** (more than the reported gain)
- With same initialization, NCA and baseline become indistinguishable

### 2. Additive Compute Results

| Natural-language budget | PPT improvement over best baseline |
|------------------------|-----------------------------------|
| 50M tokens | Both k-Dyck and NCA improve |
| 100M tokens | Both k-Dyck and NCA improve |
| 1B tokens | Gain disappears under spectral-based initialization |

Key finding: At 1B tokens, under spectral-based initialization, all PPT methods fall within the baseline's seed spread. However, under alternative baseline initialization, PPT still appears to help.

### 3. Compute-Matched Results

**Data-constrained** (100M NL tokens, $1 \times 10^{18}$ FLOPs):
- Diverting any fraction of budget to synthetic PPT increases validation BPB
- Effect levels off beyond 5% PPT
- Compute better spent on additional epochs over same natural-language data

**Over-Chinchilla** (881M params, 50B tokens):
- Neither NCA nor k-Dyck improves over baseline on validation loss or DCLM CORE accuracy

**Data-unconstrained**:
- No PPT method significantly outperforms baseline in natural-language validation BPB
- Power law fitting shows no PPT method has lower multiplicative constant or significantly larger scaling exponent than baseline

### 4. Downstream Benchmark Exceptions

| Benchmark | Setting | PPT method | Baseline (0% PPT) | PPT score |
|-----------|---------|------------|-------------------|-----------|
| BLiMP (↑) | Data-constrained (100M NL, 1×10¹⁸) | DeclRef 10% | 0.721 ± 0.004 | 0.731 ± 0.005 |
| DCLM CORE (↑) | Additive (50M NL) | k-Dyck | 0.024 ± 0.004 | 0.032 ± 0.003 |
| DCLM CORE (↑) | Data-unconstrained (1×10¹⁸) | C4 1% | 0.088 ± 0.006 | 0.106 ± 0.004 |
| DCLM CORE (↑) | Data-unconstrained (1×10¹⁸) | Code 1% | 0.088 ± 0.006 | 0.104 ± 0.004 |
| DCLM CORE (↑) | Data-unconstrained (1×10¹⁸) | DeclRef 1% | 0.088 ± 0.006 | 0.097 ± 0.002 |
| DCLM CORE (↑) | Data-unconstrained (1×10¹⁸) | DeclRef 5% | 0.088 ± 0.006 | 0.108 ± 0.011 |

Note: C4 (natural-language control) reaches DCLM CORE scores in the same range as PPT methods.

### 5. Forgetting Across PPT-to-PT Transition

At $2.7 \times 10^{19}$ FLOPs:
- During PPT phase: all methods learn their respective distributions
- At PPT-to-PT boundary: BPB immediately increases on PPT distributions (forgetting)
- **Interleaving** (5% of every batch as PPT data throughout training): prevents forgetting, improves performance on PPT distribution over entire run
- Interleaving improves FineWeb BPB for PPT methods, though not significantly for all

## Theoretical and Practical Implications

### Implications for Training Methodology

1. **Initialization matters more than curriculum**: The choice of weight initialization scheme can close gaps attributed to PPT, suggesting some reported gains may be artifacts of suboptimal baselines.

2. **Front-loading synthetic data is inefficient**: The abrupt distribution shift between PPT and PT causes forgetting of learned priors. Gradual integration (interleaving) is more effective for retention.

3. **Natural language is surprisingly effective**: Even when unique natural-language tokens are scarce, repeated training on them remains more useful than training on the tested synthetic corpora.

### For Future Research

- **Robustness checks**: Future PPT studies should vary initialization schemes for both PPT and no-PPT methods
- **Retention measurement**: Track model performance on synthetic distributions throughout training as a proxy for whether inductive biases are retained
- **Interleaving schedules**: Investigate better schedules for integrating synthetic data throughout training
- **Targeted evaluations**: PPT may shape narrower capabilities not captured by aggregate benchmarks (e.g., specific BLiMP phenomena)

### Caveats and Limitations

- Full hyperparameter search not possible above $9 \times 10^{18}$ FLOPs (cost constraints)
- Effect sizes smaller than prior work partly due to different tokenizer, architecture, and dataset (NanoChat vs. prior setups)
- BPB differences not directly comparable to perplexity differences in prior work
- Study limited to language modeling; PPT could prove more useful in other modalities

## Conclusion

The paper's central conclusion is that **synthetic pre-pretraining as a distinct training stage provides limited benefits** compared to spending the same compute on natural-language training. Key takeaways:

1. Compute-matched comparisons show natural-language training performs as well as or better than PPT across all regimes tested
2. PPT gains are sensitive to baseline initialization; improved baselines close the gap
3. Learned PPT priors are forgotten across the PPT-to-PT phase boundary
4. Interleaving synthetic data throughout training prevents forgetting but doesn't significantly improve natural-language performance
5. The results motivate **rethinking how synthetic data is used** — suggesting gradual integration rather than front-loading — rather than ruling out synthetic data entirely

**Future directions**: Solving the forgetting problem across phase boundaries, investigating better interleaving schedules, and determining whether such schedules can make synthetic data a better use of compute than natural language.

---

_Markdown view of https://picx.dev/p/TRrzNr, served by PicX — AI-generated visual whiteboard summaries of research papers._