The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data

Summary (Overview)

• Core finding: The paper introduces Specialized Pretraining (SPT), a strategy where a small domain dataset (typically reserved for finetuning) is mixed into pretraining as a fraction of total tokens (1–5%), repeated 10–50× throughout pretraining, followed by standard finetuning on the same data.

• Consistent gains across domains: SPT outperforms standard pretraining followed by finetuning (NPT→FT) across three specialized domains—MusicPile (symbolic music), ChemPile (chemistry), and ProofPile (mathematical proofs)—achieving lower domain test loss, better retention of general knowledge, and improved downstream task performance.

• Compute efficiency: SPT reduces pretraining tokens needed to reach a given domain performance by up to 1.75×, and a 1B SPT model can outperform a 3B NPT model on domains far from web text (closing 133% of the performance gap on ProofPile).

• The "finetuner's fallacy": While finetuning appears cheapest (avoiding pretraining cost), amortized over inference, pretraining a smaller model with domain data is cheaper. A 1B SPT model breaks even with a 3B finetuned model after ~1 trillion inference tokens.

• Theoretical contribution: The authors derive overfitting scaling laws that decompose test loss into training loss (power law with negative exponent) and train-test gap (power law with positive exponent), enabling prediction of optimal mixture fractions without exhaustive sweeps.

---

Introduction and Theoretical Foundation

Background and Motivation

Real-world deployments demand strong performance on narrow domains where data is scarce. The conventional recipe is:

1. Start from a strong open-weights model pretrained on web-scale data
2. Finetune on the proprietary dataset

This treats pretraining and finetuning as disjoint phases: first learn general knowledge at scale, then specialize. However, growing evidence suggests that data encountered during pretraining shapes model behavior more durably than data introduced later:

• Reasoning data in pretraining outperforms reasoning data only in finetuning (Akter et al., 2025; Hatamizadeh et al., 2025)
• Unsafe behaviors learned during pretraining are harder to remove via post-training (Maini et al., 2025)
• Cross-language transfer during pretraining improves low-resource language performance (Longpre et al., 2025)

Key Research Question

When the target domain is poorly represented in the pretraining corpus, is reserving all domain-specific data for finetuning optimal?

The authors hypothesize that introducing domain data only during finetuning may require large representational updates, leading to weaker generalization and greater forgetting of general knowledge.

Theoretical Basis: The Finetuner's Fallacy

The paper formalizes a cost analysis comparing two strategies:

• NPT→FT: Pretrain a large model (e.g., 3B) on general data, then finetune on domain data
• SPT→FT: Pretrain a smaller model (e.g., 1B) with domain data mixed in, then finetune

While NPT→FT appears cheaper upfront (avoids pretraining), the 3× larger model costs more to serve. The break-even point arrives after approximately 1 trillion inference tokens, after which SPT saves both compute and money.

---

Methodology

Notation and Setup

Specialized Pretraining (SPT): Let $\delta \in [0, 1]$ denote the fraction of pretraining tokens drawn from the domain-specific dataset. The remaining $1 - \delta$ fraction comes from general web data. Since domain datasets are much smaller than the total pretraining budget, domain examples are repeated as necessary. Given $T$ pretraining tokens, the total epochs of domain data $\mathcal{D}_{\mathrm{dom}}$ seen is:

$$E = (T \cdot \delta) / |\mathcal{D}_{\mathrm{dom}}|$$

Naive Pretraining (NPT): Standard pretraining with $\delta = 0$.

Both are followed by finetuning (FT) on the domain dataset.

Experimental Setup

Base Architecture: OLMo-1B trained on the Dolma corpus (200B tokens)

Domain Datasets (each ~300M tokens):

• MusicPile: Symbolic music notation (Yuan et al., 2024)
• ChemPile: Chemistry text (Mirza et al., 2025b)
• ProofPile: Formal mathematical proofs (Hoskinson Center for Formal Mathematics, 2022)

Mixture Fractions Tested: $\delta \in \{0, 0.1\%, 1\%, 2\%, 5\%\}$

Model Sizes: 300M, 600M, 1B, and 3B variants (adjusted by model depth and hidden dimensions)

Finetuning Protocol:

• Training exclusively on domain dataset with WSD learning rate schedule
• Early stopping applied (data repeated only as long as test loss decreases)
• Warmup steps and learning rate tuned by grid search
• Lowest domain test loss reported across all configurations

Controlled Overlap Study

For the distributional overlap analysis, the authors use an English→Japanese translation task:

• 160M-parameter LLaMA models pretrained on 20B tokens of FineWeb2 English/Japanese monolingual data
• 1B tokens of parallel data from JParaCrawl v3.0
• Japanese monolingual proportion varied over $\{0\%, 0.001\%, 0.01\%, 0.1\%, 1\%, 10\%\}$

Scaling Law Formulation

The test loss is decomposed into training loss and train-test gap, each modeled as a power law:

$$\mathcal{L}_{\mathrm{train}}(T, \delta) = A_{\mathrm{train}} T^{b_{\mathrm{train}}(\delta)} + C_{\mathrm{train}}(\delta) \tag{1}$$ $$\mathcal{L}_{\mathrm{gap}}(T, \delta) = A_{\mathrm{gap}}(\delta) \cdot T^{b_{\mathrm{gap}}(\delta)} \tag{2}$$ $$\mathcal{L}_{\mathrm{test}}(T, \delta) = \mathcal{L}_{\mathrm{train}}(T, \delta) + \mathcal{L}_{\mathrm{gap}}(T, \delta) \tag{3}$$

where the exponents are modeled as:

$$b_{x}(\delta) = \delta b_{x,s} + (1 - \delta) b_{x,g}, \quad x \in \{\text{train}, \text{gap}\} \tag{4}$$ $$A_{\text{gap}}(\delta) = \alpha_{1} \delta^{\alpha_{2}} \exp(\alpha_{3} \delta) \tag{5}$$ $$C_{\mathrm{train}}(\delta) = \kappa_{0} - \kappa_{1} \log(\delta + \kappa_{2}) - \kappa_{3} \delta \tag{6}$$

Key insight: $b_{\mathrm{train}}(\delta)$ is strictly negative (training loss decreases monotonically), while $b_{\mathrm{gap}}$ is positive but remains below 1 across all mixture percentages up to 10%, implying overfitting grows sublinearly over extended pretraining.

Post-Finetuning Loss Prediction

The change in domain test loss after finetuning is defined as:

$$\Delta \ell_{\mathrm{test}} = \ell_{\mathrm{test}}(\theta_{PT}) - \ell_{\mathrm{test}}(\theta_{PT+FT}) \tag{7}$$

where $\ell_{\mathrm{test}}(\theta_{PT})$ is the test loss of the pretrained model and $\ell_{\mathrm{test}}(\theta_{PT+FT})$ is the best loss after finetuning. This follows a power-law relationship: $\Delta \ell_{\mathrm{test}} = aT^{b} + c$.

---

Empirical Validation / Results

1. SPT Improves Domain Performance

The relative gain metric:

$$\mathcal{R}_{\text{gain}}(\delta) = 100 \cdot \frac{L^{\mathrm{NPT} \rightarrow \mathrm{FT}} - L^{\mathrm{SPT}(\delta) \rightarrow \mathrm{FT}}}{L^{\mathrm{NPT} \rightarrow \mathrm{FT}}}$$

2. SPT Reduces Forgetting

• SPT models start finetuning from a lower domain loss, requiring less aggressive adaptation
• This preserves general capabilities: at the same domain loss, SPT models achieve lower Dolma (general) loss
• Downstream improvements: At 200B pretraining tokens, SPT improves:
  • MATH accuracy by up to 6 percentage points
  • MusicTheoryBench by up to 4 percentage points

3. SPT Acts as a Regularizer

The key finding: SPT's benefit is a regularization effect, not an optimization effect:

• Domain test loss begins rising after ~5 epochs of finetuning for both SPT and NPT
• During pretraining, domain tokens are a small fraction of each batch, and surrounding general data acts as a natural regularizer
• A $\delta = 5\%$ model that has seen domain data 33× during pretraining still overfits less during finetuning than an NPT model seeing the data for the first time

4. Factors Governing SPT Gains

Domain Similarity (controlled study):

• As Japanese monolingual data decreases from 10% to 0.1% in pretraining, $\mathcal{R}_{\text{gain}}$ increases from ~2% to ~5%
• Only post-finetuning domain loss (a direct measure of NPT generalization) correctly ranks all three domains among distributional similarity metrics

Domain Dataset Size:

• 300M-token dataset: gains consistently high across all scales
• 3M–30M-token datasets: benefits diminish at longer training horizons; Specialized Continued Pretraining (SCPT) (introducing domain data at later stages) is preferable

Model Size:

• Relative gains increase with parameter count
• The 3B model exhibits the largest reduction in post-finetuning test loss under SPT

Pretraining Compute Budget:

• At small budgets (<30B tokens): $\delta = 10\%$ is optimal
• At ~50B tokens: $\delta = 5\%$ becomes optimal
• Past 240B tokens: $\delta = 2\%$ is best
• Regardless of budget, SPT with appropriate δ outperforms NPT

5. Replay Does Not Substitute for SPT

• Across replay rates $\{0\%, 10\%, 20\%\}$, SPT→FT consistently achieves lower domain test loss than NPT→FT
• 10% replay helps NPT, but NPT→FT with replay falls well short of SPT→FT without replay

6. Scaling Law Validation

• The scaling laws correctly predict that 10% SPT begins overfitting around 70B tokens (Figure 13a)
• Extrapolation correctly predicts that 2% SPT surpasses 5% SPT around 280B tokens (Figure 13b)
• These forecasts allow practitioners to identify optimal mixture fractions without running full training sweeps

---

Theoretical and Practical Implications

Theoretical Implications

1. Overfitting scaling laws: The decomposition of test loss into training loss (negative-exponent power law) and train-test gap (positive-exponent power law) provides a new framework for modeling repeated data, addressing a gap in prior work (Kaplan et al., 2020; Muennighoff et al., 2023) that lacked expressions for the overfitting stage.

2. Regularization mechanism: SPT's benefit is understood as a regularization effect—diffused exposure to domain data during pretraining creates representations that generalize better during finetuning, even after tens of repetitions.

3. When data is seen matters: The persistent advantage of SPT over replay-based approaches confirms that the timing of domain data exposure has a lasting impact on model performance.

Practical Implications

1. The finetuner's fallacy: Finetuning a large off-the-shelf model is not necessarily the cheapest path to domain adaptation when inference costs are amortized. A 1B SPT model, despite higher training cost, becomes cheaper than a 3B finetuned model after ~1 trillion inference tokens.

2. Guidance for practitioners:

   • Use higher δ for short pretraining runs, lower δ for longer ones
   • SPT helps most when the target domain is underrepresented in the pretraining corpus
   • For very small domain datasets (<30M tokens), consider SCPT (introducing data later) instead of full SPT
   • Scaling laws enable selection of optimal δ from a small number of pilot runs

3. Environmental impact: For deployed models, inference dominates total energy consumption. Enabling the same capability with fewer parameters reduces the carbon footprint of domain-adapted language models.

4. Compatibility: SPT is compatible with other data curation strategies like synthetic augmentation, and the case for early integration strengthens as pretraining costs continue to fall.

---

Conclusion

The paper exposes the finetuner's fallacy: the assumption that finetuning a large off-the-shelf model is the cheapest path to domain adaptation is often misleading. The authors demonstrate that:

"To get the most out of domain data, incorporate it as early in training as possible."

Key Takeaways

1. SPT consistently outperforms NPT→FT across chemistry, symbolic music, and mathematical proofs, achieving lower domain loss, better general knowledge retention, and improved downstream task performance.

2. SPT provides a Pareto improvement in training efficiency: better domain performance, faster convergence (up to 1.75× fewer tokens), and stronger parameter efficiency (1B SPT can beat 3B NPT on far-from-web domains).

3. The optimal strategy depends on context: domain data size, distributional overlap, model scale, and pretraining budget all interact to determine whether SPT, SCPT, or NPT is optimal.

4. Overfitting scaling laws enable practitioners to navigate this tradeoff from a small number of pilot runs rather than exhaustive search.

Future Directions

• The precise mechanism behind the asymmetry between SPT and replay remains an open question
• Extending the scaling law framework to other domains and architectures
• Exploring the interaction between SPT and other data curation strategies (e.g., synthetic augmentation)

The broader lesson: scarce and specialized domain data should not be treated as a final-stage resource. As pretraining costs continue to fall, the case for early integration strengthens, pointing toward a shift from post-training patches applied to generic checkpoints toward natively specialized models that incorporate domain knowledge from the start.