# Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting

> Time-incremental continued pretraining on web-scale post-cutoff data reliably teaches new facts across LLM families without catastrophic forgetting, though acquisition tracks pretraining saturation.

- **Source:** [arXiv](https://arxiv.org/abs/2609.23916)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/JPLsvP
- **Whiteboard:** https://picx.dev/p/JPLsvP/image

## Summary

## Summary (Overview)

- This paper studies **time-incremental continued pretraining (CPT)** of LLMs on web-scale data drawn strictly from after each model's knowledge cutoff, a realistic regime where successive crawls share substantial URL overlap.
- Across **six open-weight models** spanning three families (OLMo2, LLAMA-3.1/3.2, GEMMA-3-1B) and four parameter scales (1B–8B), CPT **consistently acquires new knowledge without catastrophic forgetting**: five of six models also improve on pre-cutoff factual recall.
- **General capabilities are largely preserved**: macro-average accuracy across a thirteen-task suite stays within 0.01 of the base for every model, with eleven of thirteen tasks within ±0.02.
- **Data quality dominates quantity**: a curated 6B-token slice matches a 40B-token broader slice; the optimal learning rates for knowledge acquisition (~3e-5) and general capability retention differ by roughly an order of magnitude; LoRA at sufficient rank (≥128) matches full CPT.
- **CPT gains survive deployment**: gains transfer through SFT, while DPO's effect is family-dependent (helps OLMo2-7B, slightly hurts LLAMA-3.1-8B).

## Introduction and Theoretical Foundation

Large language models become temporally outdated once pretraining ends, yet retraining from scratch is prohibitively expensive. Continued pretraining (CPT) is the natural remedy, but prior continual learning literature has been shaped by an assumption of **disjoint, incremental data streams** — an assumption that does not hold at web scale.

**Key structural facts motivating this work:**

1. Web-scale curation pipelines (e.g., Common Crawl) retain substantial cross-snapshot URL overlap — roughly 30–40% of URLs in fresh dumps are new, with the remainder being revisits of previously-seen URLs.
2. The pretraining corpora of almost all open-weight LLMs are closed, making strictly disjoint CPT **operationally infeasible** — one cannot enumerate which content is genuinely novel except by timestamp.
3. Global cross-snapshot deduplication actually degrades data quality (Penedo et al., 2024).

The paper formalizes the problem as: *do LLMs efficiently and reliably absorb new knowledge as it emerges without losing existing knowledge and abilities?*

**Theoretical foundation:** The central concern is catastrophic forgetting, whereby acquiring new information overwrites previously learned capabilities. The paper also leverages the **stability–plasticity trade-off**: facts are stored in distributed, overlapping directions, so plain CPT has no term distinguishing directions that carry retained knowledge from free ones.

## Methodology

**Models:** Six open-weight models with documented knowledge cutoffs:

| Model | Parameters | Pretraining Tokens | Cutoff |
|-------|-----------|-------------------|--------|
| GEMMA-3-1B | 1B | ~2T | Aug 2024 |
| OLMo2-1B | 1B | ~4.2T | Dec 2023 |
| OLMo2-7B | 7B | ~4.2T | Dec 2023 |
| LLAMA-3.2-1B | 1B | up to 9T | Dec 2023 |
| LLAMA-3.2-3B | 3B | up to 9T | Dec 2023 |
| LLAMA-3.1-8B | 8B | ~15T | Dec 2023 |

**Pretraining data:** FineWeb-Edu dumps (post-cutoff, CC week 10/2024 – week 26/2025), with two quality-filtered slices:
- High-quality slice: score ≥ 4, **6B tokens**
- Broader slice: score ≥ 3.5, **40B tokens**

**Evaluation:**
- **Daily Oracle benchmark**: QA pairs from daily news (True/False and Multiple Choice), partitioned into before-cutoff and after-cutoff sets
- **Thirteen downstream tasks**: PIQA, HellaSwag, WinoGrande, OpenBookQA, BoolQ, SciQ, ARC-Easy, ARC-Challenge, COPA, CommonsenseQA, SocialIQA, Basic Arithmetic, MMLU

**Sample-level metrics:**
- **Acquisition**: fraction of questions answered wrong → right
- **Forgetting**: fraction answered right → wrong
- **Net = acquisition − forgetting** = accuracy change

**Training details:** Learning rate sweeps (2–9 rates per model), linear warmup (500 steps), linear decay to 0.1× peak; trained on 8×H200 GPUs.

## Empirical Validation / Results

### 3.1 Knowledge Acquisition is Heterogeneous but Without Catastrophic Forgetting

- **After-cutoff gains track pretraining saturation**: OLMo2-7B (+0.09 Macro-F1) and GEMMA-3-1B (+0.07) improve substantially; LLAMA models gain less. Rank correlation with tokens-per-parameter: ρ = −0.63.
- **Five of six models improve on before-cutoff questions** (mean +0.027, p = 0.034), attributed to cross-snapshot URL overlap implicitly replaying pre-cutoff content.
- **Gains widen over time**: the CPT advantage grows for more recent events.
- **Gains are not due to memorization**: Macro-F1 deltas on in-corpus and out-of-corpus question sets are comparable; CPT improves five of six models on a temporal holdout (mean +0.028).

### 3.2 Sample-Level View: Acquisition and Forgetting are Coupled

**Table 2: Sample-level transitions (acquisition / forgetting / net, as % of questions)**

| Model | TF Before-cutoff | TF After-cutoff | MC Before-cutoff | MC After-cutoff |
|-------|-----------------|-----------------|------------------|-----------------|
| OLMo2-7B | 26.0 / 9.3 / +16.7 | 25.3 / 13.1 / +12.2 | 7.8 / 5.7 / +2.0 | 11.8 / 6.0 / +5.8 |
| GEMMA-3-1B | 20.7 / 13.1 / +7.7 | 20.6 / 15.6 / +5.1 | 2.9 / 4.0 / −1.0 | 6.2 / 1.5 / +4.7 |
| OLMo2-1B | 20.0 / 12.3 / +7.7 | 21.5 / 16.7 / +4.8 | 15.1 / 11.7 / +3.4 | 11.3 / 8.2 / +3.2 |
| LLAMA-3.2-3B | 13.3 / 10.4 / +3.0 | 18.1 / 13.2 / +4.9 | 8.2 / 10.8 / −2.6 | 8.8 / 2.4 / +6.4 |
| LLAMA-3.1-8B | 8.2 / 8.5 / −0.3 | 16.0 / 14.2 / +1.9 | 4.8 / 3.7 / +1.1 | 5.2 / 2.1 / +3.1 |
| LLAMA-3.2-1B | 7.7 / 10.2 / −2.5 | 10.7 / 11.8 / −1.1 | 1.7 / 1.6 / +0.2 | 2.3 / 1.1 / +1.2 |

Key findings:
- **Forgetting is real at the sample level** (9–16% of before-cutoff TF answers flip), but outweighed by acquisition.
- **Acquisition and forgetting are strongly correlated** (Pearson r = 0.68, p = 1.3e-27), with best fit: forg = 0.60·acq + 0.025. Roughly 0.6 previously-correct answers are lost per additional answer gained.
- **Learning-rate sweet spot**: the net is maximized at intermediate rates (1e-5–5e-5).

### 3.3 General Capabilities are Largely Preserved

- Every model lands within **0.01 of its base** on macro-average accuracy across thirteen tasks.
- Eleven of thirteen tasks stay within ±0.02; a TOST test against ±0.01 margin gives 90% CI of [−0.001, +0.007].
- **ARC-Challenge improves consistently** across all six models (+0.04 on OLMo2-1B).
- **CommonsenseQA is the only benchmark** where any model loses >0.02, concentrated on OLMo2 family (worst case: −0.06) — a localized trade-off between factual acquisition and commonsense reasoning.

### 3.4 Learning Recipe

**Data quality dominates quantity:**
- A curated 6B-token slice (score ≥ 4) captures most of the CPT signal; the 40B slice (score ≥ 3.5) yields only marginal gains (only OLMo2-1B benefits, 0.49 → 0.51).
- Training 7 epochs on high-quality data provides no additional benefit over a single pass.

**Learning rate divergence:**
- Daily Oracle peaks at **3e-5**, then degrades sharply at higher rates.
- Downstream accuracy continues to rise even at **3e-4** (which degrades Daily Oracle).
- The two optima are separated by roughly an order of magnitude.

**LoRA matches full CPT at sufficient rank:**
- Higher-rank variants match or slightly exceed full CPT on both before- and after-cutoff questions.
- For LLAMA-3.2-3B: LoRA ranks 128/256 reach after-cutoff Macro-F1 of 0.525/0.534 vs. 0.540 for full CPT (base: 0.484).
- Primary advantage is **memory efficiency, not speed** (no wall-clock speedup observed).

### 3.5 Deployment Survival

- **CPT gains transfer through SFT**: all CPT variants outperform the base after instruction tuning.
- **DPO's effect is family-dependent**: helps OLMo2-7B, slightly hurts LLAMA-3.1-8B (reminiscent of the alignment-tax phenomenon).

## Theoretical and Practical Implications

1. **The disjoint-stream assumption does not hold at web scale**: constructing strictly disjoint corpora is infeasible (closed pretraining corpora) and counterproductive (deduplication degrades quality). Catastrophic forgetting is a property of disjoint regimes, not CPT per se.

2. **Temporal plasticity tracks pretraining saturation more than scale**: models trained on fewer tokens (GEMMA-3-1B at ~2T, OLMo2 at ~4.2T) are more plastic than heavily-trained ones (LLAMA-3.1-8B at ~15T). Multi-family evaluation should be the minimum standard for CPT research.

3. **Operational recipe for time-incremental updates**:
   - (i) Source CPT data from a curated, high-quality post-cutoff slice (score ≥ 4 or equivalent)
   - (ii) Train at a moderate learning rate (~3e-5) for factual recall, tuning against a held-out temporal slice
   - (iii) Use LoRA at rank ≥ 128 if memory-bound
   - (iv) Plan for short runs (performance plateaus within the first third of training)
   - (v) Apply CPT before SFT in the post-training pipeline

## Conclusion

This work provides an optimistic, empirically-grounded picture of time-incremental CPT in the realistic web-scale regime. The key takeaways:

- **CPT works**: knowledge is acquired heterogeneously across families, without catastrophic forgetting of either pre-cutoff facts or general capabilities.
- **The cost is low**: general abilities are statistically equivalent to base within a ±0.01 margin.
- **The recipe is simple**: quality over quantity, learning rate tuned to the deployment objective, LoRA at sufficient rank, short runs, and CPT before SFT.
- **Future directions**: replicating the learning-rate sweep across additional families, disentangling whether DPO's family-dependent effects reflect base-model differences or interactions with the CPT signal, and exploring the stability–plasticity trade-off at even larger scales.

---

_Markdown view of https://picx.dev/p/JPLsvP, served by PicX — AI-generated visual whiteboard summaries of research papers._