Rethinking Pre-Pretraining

Summary (Overview)

• Core finding: Allocating compute to synthetic pre-pretraining (PPT) never outperforms pure natural-language training in compute-matched experiments across four PPT curricula (k-Dyck, neural cellular automata (NCA), declarative reference corpora, and code) and three training regimes.
• Initialization sensitivity: PPT gains reported in prior work are sensitive to weight initialization schemes; an alternative scheme improves both methods, but the no-PPT baseline more, closing the gap.
• Additive compute: PPT can help at small natural-language budgets (50M–100M tokens) but the advantage disappears by 1B tokens.
• Forgetting mechanism: Models forget what they learn during PPT when transitioning to natural-language training; interleaving prevents forgetting but does not significantly improve natural-language performance.
• Recommendation: Results motivate rethinking the PPT phase (front-loading synthetic data) rather than ruling out synthetic data entirely; gradual integration through interleaving may be preferable.

Introduction and Theoretical Foundation

The paper addresses a fundamental question in language model training: can synthetic data provide useful linguistic and computational priors that transfer to natural language?

Motivation:

• Natural language scaling benefits haven't hit a ceiling, but increasingly large data is needed (Kaplan et al., 2020)
• Humans acquire language from far less linguistic experience (Warstadt et al., 2023)
• Reasoning abilities are largely dissociable from linguistic proficiency (Fedorenko et al., 2024)
• Synthetic distributions supply effectively unlimited training data once high-quality natural language becomes scarce

Pre-pretraining (PPT) is operationalized as a training phase on synthetic data that precedes conventional pretraining (PT), intended to produce an initialization with useful priors for learning language. Different synthetic distributions target different computational priors:

• k-Dyck: encourages hierarchical dependency tracking (Hu et al., 2025)
• Neural cellular automata (NCA): expose models to latent transition rules and long-range dependencies (Lee et al., 2026)
• Procedural tasks: train operations such as copying, sorting, and retrieval (Jiang et al., 2026)

Prior work reported gains including up to 8.6% lower perplexity for NCA, both when PPT receives additional compute and when it shares a fixed total compute budget.

Key theoretical questions:

1. Are PPT gains robust across different training configurations and regimes?
2. Does PPT provide useful inductive biases when not competing with natural-language training for compute?
3. Is PPT worth its compute when competing directly with natural-language training?

Methodology

Experimental Setup

Architecture: All models trained using a fork of NanoChat with:

• Muon optimizer (Jordan et al., 2024; Liu et al., 2025; Karpathy, 2025) for transformer blocks' weight matrices
• AdamW for embedding and unembedding tables
• Hidden size set to 64d for depth d, so parameter count follows from depth alone

Data:

• Natural language: FineWeb (Penedo et al., 2024), C4 (Raffel et al., 2023), GitHub Code dataset (CodeParrot, 2022)
• Synthetic: k-Dyck, NCA (generated using prior work's code), and DeclRef (new synthetic corpus targeting retrieval from context)

Training Regimes

Additive compute: PPT prepended to natural-language training without substituting any of its budget. Model size fixed at 336M parameters; PT budgets {50M, 100M, 1B} tokens; PPT tokens varied over {5M, 20M, 50M, 100M, 200M, 500M, 1B}.

Compute-matched: Total compute budget fixed; α gives fraction spent on PPT, with α ∈ {0, 0.01, 0.05, 0.10}. Three configurations:

1. Data-constrained: 100M natural-language tokens, $1 \times 10^{18}$ FLOPs total
2. Data-unconstrained: abundant natural language, sweeps over $1 \times 10^{18}$, $3 \times 10^{18}$, $9 \times 10^{18}$ FLOPs
3. Over-Chinchilla: 881M parameters, 50B tokens (∼57 tokens per parameter, ~12× compute-optimal)

Weight Initialization Ablation

Three schemes tested:

1. Hu et al. (2025): all parameters from $\mathcal{N}(0, 0.02)$
2. Lee et al. (2026): token embeddings from $\mathcal{N}(0, 1)$, remaining from $\mathcal{N}(0, 0.02)$
3. Spectral-based (NanoChat): linear-layer weights from $\mathcal{N}(0, \sigma)$ with $\sigma = \min(1, \sqrt{d_{out}/d_{in}})/\sqrt{d_{in}}$, token embeddings from $\mathcal{N}(0, 1)$, zero-initialized LM head and residual projections

Tokenization

• NCA: Each grid split into $2 \times 2$ patches, each becoming one token; grids concatenated in time order with start/end tokens
• k-Dyck: Opening and closing bracket types each their own token
• DeclRef: Keywords, scope brackets, names, and values as unique tokens
• Synthetic languages use their own smaller vocabularies; at phase boundary, vocabulary expanded and embedding table/LM head reinitialized

Hyperparameter Optimization

• LLMTuna: language-model-based hyperparameter optimizer (released with paper)
• Found more sample-efficient than Bayesian optimization (Optuna)
• Model size searched separately for every method and budget in data-constrained and data-unconstrained regimes

Evaluation

• Primary metric: Bits per byte (BPB) on held-out FineWeb split (~10M tokens)
• Secondary: DCLM CORE suite (Li et al., 2024), BLiMP (Warstadt et al., 2020)
• All metrics reported as mean over three seeds with standard deviation

Empirical Validation / Results

1. Initialization Sensitivity (Reproduction)

Reproducing Lee et al. (2026)'s NCA experiment with a 400M-parameter Llama model on ~9B OpenWebText tokens:

• Under original initialization (N(0,1) embeddings): NCA PPT improves perplexity by 5.4%
• Changing embeddings to N(0, 0.02): baseline improves by 9.0% (more than the reported gain)
• With same initialization, NCA and baseline become indistinguishable

2. Additive Compute Results

Key finding: At 1B tokens, under spectral-based initialization, all PPT methods fall within the baseline's seed spread. However, under alternative baseline initialization, PPT still appears to help.

3. Compute-Matched Results

Data-constrained (100M NL tokens, $1 \times 10^{18}$ FLOPs):

• Diverting any fraction of budget to synthetic PPT increases validation BPB
• Effect levels off beyond 5% PPT
• Compute better spent on additional epochs over same natural-language data

Over-Chinchilla (881M params, 50B tokens):

• Neither NCA nor k-Dyck improves over baseline on validation loss or DCLM CORE accuracy

Data-unconstrained:

• No PPT method significantly outperforms baseline in natural-language validation BPB
• Power law fitting shows no PPT method has lower multiplicative constant or significantly larger scaling exponent than baseline

4. Downstream Benchmark Exceptions

Note: C4 (natural-language control) reaches DCLM CORE scores in the same range as PPT methods.

5. Forgetting Across PPT-to-PT Transition

At $2.7 \times 10^{19}$ FLOPs:

• During PPT phase: all methods learn their respective distributions
• At PPT-to-PT boundary: BPB immediately increases on PPT distributions (forgetting)
• Interleaving (5% of every batch as PPT data throughout training): prevents forgetting, improves performance on PPT distribution over entire run
• Interleaving improves FineWeb BPB for PPT methods, though not significantly for all

Theoretical and Practical Implications

Implications for Training Methodology

1. Initialization matters more than curriculum: The choice of weight initialization scheme can close gaps attributed to PPT, suggesting some reported gains may be artifacts of suboptimal baselines.

2. Front-loading synthetic data is inefficient: The abrupt distribution shift between PPT and PT causes forgetting of learned priors. Gradual integration (interleaving) is more effective for retention.

3. Natural language is surprisingly effective: Even when unique natural-language tokens are scarce, repeated training on them remains more useful than training on the tested synthetic corpora.

For Future Research

• Robustness checks: Future PPT studies should vary initialization schemes for both PPT and no-PPT methods
• Retention measurement: Track model performance on synthetic distributions throughout training as a proxy for whether inductive biases are retained
• Interleaving schedules: Investigate better schedules for integrating synthetic data throughout training
• Targeted evaluations: PPT may shape narrower capabilities not captured by aggregate benchmarks (e.g., specific BLiMP phenomena)

Caveats and Limitations

• Full hyperparameter search not possible above $9 \times 10^{18}$ FLOPs (cost constraints)
• Effect sizes smaller than prior work partly due to different tokenizer, architecture, and dataset (NanoChat vs. prior setups)
• BPB differences not directly comparable to perplexity differences in prior work
• Study limited to language modeling; PPT could prove more useful in other modalities

Conclusion

The paper's central conclusion is that synthetic pre-pretraining as a distinct training stage provides limited benefits compared to spending the same compute on natural-language training. Key takeaways:

1. Compute-matched comparisons show natural-language training performs as well as or better than PPT across all regimes tested
2. PPT gains are sensitive to baseline initialization; improved baselines close the gap
3. Learned PPT priors are forgotten across the PPT-to-PT phase boundary
4. Interleaving synthetic data throughout training prevents forgetting but doesn't significantly improve natural-language performance
5. The results motivate rethinking how synthetic data is used — suggesting gradual integration rather than front-loading — rather than ruling out synthetic data entirely

Future directions: Solving the forgetting problem across phase boundaries, investigating better interleaving schedules, and determining whether such schedules can make synthetic data a better use of compute than natural language.