Visual Summary | Scaling Domain Data Repetition in LLM Pretraining

Summary (Overview)

Core Finding: This paper studies data repetition in LLM pretraining under a fixed tokens-per-parameter (TPP) ratio, a setting that better reflects practical LLM scaling than fixed-data-budget settings.
Key Results: The optimal repetition count is strongly negatively correlated with final validation loss (Pearson r = -0.944), mildly increases with model size under fixed TPP, and is largely insensitive to the amount of unique high-quality data (r = 0.018).
Theoretical Contribution: A linear regression model is used to explain these observations, providing theorems on how noise fitting, knowledge acquisition, and model scaling interact to determine optimal repetition.
Practical Implication: Practitioners can use smaller proxy models (trained at the same TPP) to conservatively estimate safe repetition counts for larger models, and domain-specific validation loss is a strong predictor of repetition tolerance.

Introduction and Theoretical Foundation

The paper addresses a critical problem in LLM scaling: while general web data is abundant, high-quality domain-specific data (code, math, etc.) is limited and difficult to scale. As models grow, the training-token budget increases proportionally (fixed TPP ratio), diluting the fraction of high-quality domain data in the training mixture. This dilution risks under-learning knowledge-dense domains, as prior work shows that domains may only be learned once their mixing ratio exceeds a critical threshold.

Data repetition—reusing limited high-quality data multiple times—is proposed as a direct countermeasure. However, excessive repetition risks overfitting. The paper's central question is: What is the optimal repetition count, and how does it depend on model size, domain, and data quantity?

Key Difference from Prior Work: Previous studies typically hold the training-data size fixed when comparing models of different sizes. This paper argues that this is unrealistic; instead, the TPP ratio (D/N) should be held constant, meaning the training-token budget grows with model size, directly following the Chinchilla scaling paradigm.

Methodology

Experimental Setup

Model Sizes: Models of varying sizes N are trained, each with a total token budget of $D_N = \text{TPP} \cdot N$ , where TPP > 100.
High-Quality Domains: Four domains are studied: Code, Math, Wiki, and Medical. Each domain is trained separately with a mixture of its data and general web data.
Data Configuration: For each domain, a subset of $U_{N,\alpha} = \alpha D_N$ unique tokens is selected and repeated $e$ times. The total high-quality token count is $H_{N,\alpha,e} = eU_{N,\alpha} = e\alpha D_N$ , and the remaining tokens are web data. Parameters are set to $\alpha \in \{\frac{1}{40}, \frac{1}{20}, \frac{1}{10}\}$ and repetition count $e \in \{1, 2, 3, 4, 5, 6, 7\}$ .
Metrics: Both In-Domain (IID) validation loss and Out-Of-Domain (OOD) pretraining loss are measured.

Analysis of Optimal Repetition

The optimal repetition count is estimated by fitting a quadratic function to the validation loss as a function of repetition count:

\widehat{L}_{d, N, \alpha}(e) = a_{d, N, \alpha}e^2 + b_{d, N, \alpha}e + c_{d, N,\alpha}

and finding the minimum of the fitted curve: $\widehat{e}^*_{d, N, \alpha} = -b_{d, N, \alpha} / (2a_{d, N, \alpha})$ .

Theoretical Model

A one-hot linear regression problem in $\mathbb{R}^\infty$ is used. Each coordinate $k$ represents a "knowledge unit" with frequency following a power law. The model is trained via full-batch gradient descent. The expected risk is decomposed into three terms: unrepresented knowledge, knowledge-acquisition error, and noise-fitting error.

Empirical Validation / Results

Key Findings

Domain Dependence: Different domains exhibit substantially different optimal repetition counts. Math can tolerate many repetitions, while Wiki degrades quickly.
Model Size Dependence: Under fixed TPP, the optimal repetition count increases with model size, contrary to the fixed-data-budget setting where it decreases. This is a central result.
Data Quantity Insensitivity: The optimal repetition count is largely insensitive to the fraction of unique high-quality data (α).

Correlation Analysis

The paper quantifies the relationship between the optimal repetition count and various factors:

Factor	Pearson Correlation
Minimum Validation Loss	-0.944
Model Size	+0.400
Unique Data Fraction (α)	+0.018

Theoretical Results

The theoretical model provides three key theorems:

Theorem 4.2 (Noise Decay): Lower noise (σ²) in the data leads to a larger optimal repetition count, as overfitting is delayed. This explains the strong negative correlation with validation loss.
Theorem 4.3 (Fixed Budget): With a fixed token budget, increasing model size eventually causes the optimal repetition count to decrease, as the model overfits earlier.
Theorem 4.4 (Linear Scaling): Under fixed TPP scaling (D/N → C₀), the optimal repetition count grows as $r^*(D, N) = \Theta(D^{\alpha/\beta})$ , confirming that more repetitions are beneficial as the model and data budget scale up together.

Additional Analyses

Unique vs. Repeated Tokens: Replacing unique tokens with repeated tokens at a fixed domain fraction shows domain-dependent degradation. Math is robust, while Wiki degrades sharply with more than 2 repetitions.
OOD Performance: Repetition within a high-quality domain has limited effect on OOD pretraining performance when total domain fraction is controlled.
Learning-Rate Schedules: The learning-rate schedule affects repetition tolerance. A constant LR or later decay allows more repetitions before overfitting, while earlier decay leads to degradation after fewer repetitions.

Theoretical and Practical Implications

New Scaling Regime: The paper highlights that the choice of scaling regime (fixed data budget vs. fixed TPP) fundamentally changes the qualitative conclusions about data repetition, reconciling conflicting findings in prior literature.
Theoretical Framework: The linear regression model provides a principled framework for understanding the trade-off between knowledge acquisition and noise fitting during repeated training, offering a theoretical basis for empirical observations.
Proxy Model Tuning: The most significant practical contribution is a clear, low-cost strategy for selecting repetition counts. Since the optimal count is insensitive to α and correlates with validation loss, one can use a small proxy model to find a safe repetition count, which will be conservative for larger models.
Domain-Specific Strategy: The strong correlation between validation loss and optimal repetition suggests that domains with lower loss (e.g., math) should be repeated more than domains with higher loss (e.g., wiki).

Conclusion

This paper provides a systematic study of data repetition in LLM pretraining, emphasizing the importance of the tokens-per-parameter scaling regime. The authors find that the optimal repetition count is primarily determined by the domain's validation loss, increases mildly with model size under fixed TPP, and is insensitive to the amount of unique data. The theoretical model offers a robust explanation for these phenomena.

Future work will extend the analysis to mixtures with multiple simultaneously repeated domains and develop quantitative scaling rules for predicting optimal repetition counts across model scales.