# The Finetuner’s Fallacy When to Pretrain with Your Finetuning Data

> Specialized Pretraining, mixing small domain datasets into pretraining before finetuning, beats standard pretrain-then-finetune across domains, cutting compute up to 1.75x and enabling smaller models to outperform larger ones.

- **Source:** [arXiv](https://arxiv.org/abs/2603.16177)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/6vvmyr
- **Whiteboard:** https://picx.dev/p/6vvmyr/image

## Summary

# The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data

## Summary (Overview)

- **Core finding**: The paper introduces **Specialized Pretraining (SPT)**, a strategy where a small domain dataset (typically reserved for finetuning) is mixed into pretraining as a fraction of total tokens (1–5%), repeated 10–50× throughout pretraining, followed by standard finetuning on the same data.

- **Consistent gains across domains**: SPT outperforms standard pretraining followed by finetuning (NPT→FT) across three specialized domains—MusicPile (symbolic music), ChemPile (chemistry), and ProofPile (mathematical proofs)—achieving lower domain test loss, better retention of general knowledge, and improved downstream task performance.

- **Compute efficiency**: SPT reduces pretraining tokens needed to reach a given domain performance by up to **1.75×**, and a 1B SPT model can outperform a 3B NPT model on domains far from web text (closing 133% of the performance gap on ProofPile).

- **The "finetuner's fallacy"**: While finetuning appears cheapest (avoiding pretraining cost), amortized over inference, pretraining a smaller model with domain data is cheaper. A 1B SPT model breaks even with a 3B finetuned model after ~1 trillion inference tokens.

- **Theoretical contribution**: The authors derive **overfitting scaling laws** that decompose test loss into training loss (power law with negative exponent) and train-test gap (power law with positive exponent), enabling prediction of optimal mixture fractions without exhaustive sweeps.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Real-world deployments demand strong performance on narrow domains where data is scarce. The conventional recipe is:

1. Start from a strong open-weights model pretrained on web-scale data
2. Finetune on the proprietary dataset

This treats pretraining and finetuning as disjoint phases: first learn general knowledge at scale, then specialize. However, growing evidence suggests that data encountered during pretraining shapes model behavior more durably than data introduced later:

- Reasoning data in pretraining outperforms reasoning data only in finetuning (Akter et al., 2025; Hatamizadeh et al., 2025)
- Unsafe behaviors learned during pretraining are harder to remove via post-training (Maini et al., 2025)
- Cross-language transfer during pretraining improves low-resource language performance (Longpre et al., 2025)

### Key Research Question

> When the target domain is poorly represented in the pretraining corpus, is reserving all domain-specific data for finetuning optimal?

The authors hypothesize that introducing domain data only during finetuning may require large representational updates, leading to weaker generalization and greater forgetting of general knowledge.

### Theoretical Basis: The Finetuner's Fallacy

The paper formalizes a cost analysis comparing two strategies:

- **NPT→FT**: Pretrain a large model (e.g., 3B) on general data, then finetune on domain data
- **SPT→FT**: Pretrain a smaller model (e.g., 1B) with domain data mixed in, then finetune

While NPT→FT appears cheaper upfront (avoids pretraining), the 3× larger model costs more to serve. The break-even point arrives after approximately 1 trillion inference tokens, after which SPT saves both compute and money.

---

## Methodology

### Notation and Setup

**Specialized Pretraining (SPT)**: Let $\delta \in [0, 1]$ denote the fraction of pretraining tokens drawn from the domain-specific dataset. The remaining $1 - \delta$ fraction comes from general web data. Since domain datasets are much smaller than the total pretraining budget, domain examples are repeated as necessary. Given $T$ pretraining tokens, the total epochs of domain data $\mathcal{D}_{\mathrm{dom}}$ seen is:

$$E = (T \cdot \delta) / |\mathcal{D}_{\mathrm{dom}}|$$

**Naive Pretraining (NPT)**: Standard pretraining with $\delta = 0$.

Both are followed by finetuning (FT) on the domain dataset.

### Experimental Setup

**Base Architecture**: OLMo-1B trained on the Dolma corpus (200B tokens)

**Domain Datasets** (each ~300M tokens):
- **MusicPile**: Symbolic music notation (Yuan et al., 2024)
- **ChemPile**: Chemistry text (Mirza et al., 2025b)
- **ProofPile**: Formal mathematical proofs (Hoskinson Center for Formal Mathematics, 2022)

**Mixture Fractions Tested**: $\delta \in \{0, 0.1\%, 1\%, 2\%, 5\%\}$

**Model Sizes**: 300M, 600M, 1B, and 3B variants (adjusted by model depth and hidden dimensions)

**Finetuning Protocol**:
- Training exclusively on domain dataset with WSD learning rate schedule
- **Early stopping** applied (data repeated only as long as test loss decreases)
- Warmup steps and learning rate tuned by grid search
- Lowest domain test loss reported across all configurations

### Controlled Overlap Study

For the distributional overlap analysis, the authors use an **English→Japanese translation task**:
- 160M-parameter LLaMA models pretrained on 20B tokens of FineWeb2 English/Japanese monolingual data
- 1B tokens of parallel data from JParaCrawl v3.0
- Japanese monolingual proportion varied over $\{0\%, 0.001\%, 0.01\%, 0.1\%, 1\%, 10\%\}$

### Scaling Law Formulation

The test loss is decomposed into training loss and train-test gap, each modeled as a power law:

$$\mathcal{L}_{\mathrm{train}}(T, \delta) = A_{\mathrm{train}} T^{b_{\mathrm{train}}(\delta)} + C_{\mathrm{train}}(\delta) \tag{1}$$

$$\mathcal{L}_{\mathrm{gap}}(T, \delta) = A_{\mathrm{gap}}(\delta) \cdot T^{b_{\mathrm{gap}}(\delta)} \tag{2}$$

$$\mathcal{L}_{\mathrm{test}}(T, \delta) = \mathcal{L}_{\mathrm{train}}(T, \delta) + \mathcal{L}_{\mathrm{gap}}(T, \delta) \tag{3}$$

where the exponents are modeled as:

$$b_{x}(\delta) = \delta b_{x,s} + (1 - \delta) b_{x,g}, \quad x \in \{\text{train}, \text{gap}\} \tag{4}$$

$$A_{\text{gap}}(\delta) = \alpha_{1} \delta^{\alpha_{2}} \exp(\alpha_{3} \delta) \tag{5}$$

$$C_{\mathrm{train}}(\delta) = \kappa_{0} - \kappa_{1} \log(\delta + \kappa_{2}) - \kappa_{3} \delta \tag{6}$$

**Key insight**: $b_{\mathrm{train}}(\delta)$ is strictly negative (training loss decreases monotonically), while $b_{\mathrm{gap}}$ is positive but remains below 1 across all mixture percentages up to 10%, implying overfitting grows sublinearly over extended pretraining.

### Post-Finetuning Loss Prediction

The change in domain test loss after finetuning is defined as:

$$\Delta \ell_{\mathrm{test}} = \ell_{\mathrm{test}}(\theta_{PT}) - \ell_{\mathrm{test}}(\theta_{PT+FT}) \tag{7}$$

where $\ell_{\mathrm{test}}(\theta_{PT})$ is the test loss of the pretrained model and $\ell_{\mathrm{test}}(\theta_{PT+FT})$ is the best loss after finetuning. This follows a power-law relationship: $\Delta \ell_{\mathrm{test}} = aT^{b} + c$.

---

## Empirical Validation / Results

### 1. SPT Improves Domain Performance

The relative gain metric:

$$\mathcal{R}_{\text{gain}}(\delta) = 100 \cdot \frac{L^{\mathrm{NPT} \rightarrow \mathrm{FT}} - L^{\mathrm{SPT}(\delta) \rightarrow \mathrm{FT}}}{L^{\mathrm{NPT} \rightarrow \mathrm{FT}}}$$

| Domain | Relative Gain | Compute Multiplier | Gap Closed (1B vs 3B) |
|--------|--------------|-------------------|----------------------|
| MusicPile | 2.0% | 1.75× | 81% |
| ProofPile | 1.5% | 1.56× | 133% |
| ChemPile | 0.8% | 1.40× | 23% |

### 2. SPT Reduces Forgetting

- SPT models start finetuning from a lower domain loss, requiring less aggressive adaptation
- This preserves general capabilities: at the same domain loss, SPT models achieve lower Dolma (general) loss
- **Downstream improvements**: At 200B pretraining tokens, SPT improves:
  - MATH accuracy by up to **6 percentage points**
  - MusicTheoryBench by up to **4 percentage points**

### 3. SPT Acts as a Regularizer

The key finding: SPT's benefit is a **regularization effect**, not an optimization effect:
- Domain test loss begins rising after ~5 epochs of finetuning for both SPT and NPT
- During pretraining, domain tokens are a small fraction of each batch, and surrounding general data acts as a natural regularizer
- A $\delta = 5\%$ model that has seen domain data 33× during pretraining still overfits less during finetuning than an NPT model seeing the data for the first time

### 4. Factors Governing SPT Gains

**Domain Similarity** (controlled study):
- As Japanese monolingual data decreases from 10% to 0.1% in pretraining, $\mathcal{R}_{\text{gain}}$ increases from ~2% to ~5%
- Only post-finetuning domain loss (a direct measure of NPT generalization) correctly ranks all three domains among distributional similarity metrics

**Domain Dataset Size**:
- 300M-token dataset: gains consistently high across all scales
- 3M–30M-token datasets: benefits diminish at longer training horizons; **Specialized Continued Pretraining (SCPT)** (introducing domain data at later stages) is preferable

**Model Size**:
- Relative gains increase with parameter count
- The 3B model exhibits the largest reduction in post-finetuning test loss under SPT

**Pretraining Compute Budget**:
- At small budgets (<30B tokens): $\delta = 10\%$ is optimal
- At ~50B tokens: $\delta = 5\%$ becomes optimal
- Past 240B tokens: $\delta = 2\%$ is best
- Regardless of budget, SPT with appropriate δ outperforms NPT

### 5. Replay Does Not Substitute for SPT

- Across replay rates $\{0\%, 10\%, 20\%\}$, SPT→FT consistently achieves lower domain test loss than NPT→FT
- 10% replay helps NPT, but NPT→FT with replay falls well short of SPT→FT without replay

### 6. Scaling Law Validation

- The scaling laws correctly predict that 10% SPT begins overfitting around 70B tokens (Figure 13a)
- Extrapolation correctly predicts that 2% SPT surpasses 5% SPT around 280B tokens (Figure 13b)
- These forecasts allow practitioners to identify optimal mixture fractions without running full training sweeps

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Overfitting scaling laws**: The decomposition of test loss into training loss (negative-exponent power law) and train-test gap (positive-exponent power law) provides a new framework for modeling repeated data, addressing a gap in prior work (Kaplan et al., 2020; Muennighoff et al., 2023) that lacked expressions for the overfitting stage.

2. **Regularization mechanism**: SPT's benefit is understood as a regularization effect—diffused exposure to domain data during pretraining creates representations that generalize better during finetuning, even after tens of repetitions.

3. **When data is seen matters**: The persistent advantage of SPT over replay-based approaches confirms that the timing of domain data exposure has a lasting impact on model performance.

### Practical Implications

1. **The finetuner's fallacy**: Finetuning a large off-the-shelf model is not necessarily the cheapest path to domain adaptation when inference costs are amortized. A 1B SPT model, despite higher training cost, becomes cheaper than a 3B finetuned model after ~1 trillion inference tokens.

2. **Guidance for practitioners**:
   - Use **higher δ** for short pretraining runs, **lower δ** for longer ones
   - SPT helps most when the target domain is **underrepresented** in the pretraining corpus
   - For **very small domain datasets** (<30M tokens), consider SCPT (introducing data later) instead of full SPT
   - Scaling laws enable selection of optimal δ from a **small number of pilot runs**

3. **Environmental impact**: For deployed models, inference dominates total energy consumption. Enabling the same capability with fewer parameters reduces the carbon footprint of domain-adapted language models.

4. **Compatibility**: SPT is compatible with other data curation strategies like synthetic augmentation, and the case for early integration strengthens as pretraining costs continue to fall.

---

## Conclusion

The paper exposes the **finetuner's fallacy**: the assumption that finetuning a large off-the-shelf model is the cheapest path to domain adaptation is often misleading. The authors demonstrate that:

> "To get the most out of domain data, incorporate it as early in training as possible."

### Key Takeaways

1. **SPT consistently outperforms NPT→FT** across chemistry, symbolic music, and mathematical proofs, achieving lower domain loss, better general knowledge retention, and improved downstream task performance.

2. **SPT provides a Pareto improvement** in training efficiency: better domain performance, faster convergence (up to 1.75× fewer tokens), and stronger parameter efficiency (1B SPT can beat 3B NPT on far-from-web domains).

3. **The optimal strategy depends on context**: domain data size, distributional overlap, model scale, and pretraining budget all interact to determine whether SPT, SCPT, or NPT is optimal.

4. **Overfitting scaling laws** enable practitioners to navigate this tradeoff from a small number of pilot runs rather than exhaustive search.

### Future Directions

- The precise mechanism behind the asymmetry between SPT and replay remains an open question
- Extending the scaling law framework to other domains and architectures
- Exploring the interaction between SPT and other data curation strategies (e.g., synthetic augmentation)

The broader lesson: scarce and specialized domain data should not be treated as a final-stage resource. As pretraining costs continue to fall, the case for early integration strengthens, pointing toward a shift from post-training patches applied to generic checkpoints toward natively specialized models that incorporate domain knowledge from the start.

---

_Markdown view of https://picx.dev/p/6vvmyr, served by PicX — AI-generated visual whiteboard summaries of research papers._