Summary (Overview)

• Core problem: How to optimally mix scarce, valuable target data (e.g., low-resource languages, specialized domains) with abundant generic data during LLM pretraining, when the target data must be repeated due to its limited size.
• Key empirical finding: Across 2,000+ training runs, mixture training tolerates 15–20 repetitions of target data—far exceeding the widely cited <4 epoch ceiling from single-source training—because generic data acts as an implicit regularizer.
• Main contribution: A repetition-aware mixture scaling law that predicts target-domain loss as a function of target data size, mixture ratio, and model size, enabling optimal mixture recommendations without expensive hyperparameter sweeps.
• Additional findings: Larger models overfit faster but still achieve lower losses; broadening quality filters (accepting slightly lower-quality data for more unique tokens) beats aggressive repetition of narrow high-quality slices; the scaling law extends to multi-domain settings with multiple constrained sources.
• Practical impact: Given only target pool size and compute budget, the law prescribes the optimal mixture ratio, saving up to 74% of training compute compared to naive baselines.

---

Introduction and Theoretical Foundation

Background

Large language models (LLMs) achieve state-of-the-art performance partly due to massive pretraining corpora (trillions of tokens). However, many real-world scenarios involve data-constrained domains—low-resource languages, specialized domains (math, science), or curated high-quality datasets—where unique data is inherently limited.

The Mixing Trade-off

When scarce target data is mixed with abundant generic data, a fundamental tension emerges:

• Too little target data → underexposure to the target domain
• Too much target data → excessive repetition of the same examples → memorization → overfitting

This is formalized via the repetition factor:

$$r = h \cdot D_{\mathrm{total}} / D_{\mathrm{target}} \tag{3.1}$$

where $h \in [0,1]$ is the fraction of training devoted to target data, $D_{\mathrm{total}}$ is the total training token budget, and $D_{\mathrm{target}}$ is the unique target token count.

Gap in Prior Work

• Muennighoff et al. (2023) derived scaling laws for single-source data-constrained training, showing ~4 repetitions are nearly lossless. However, this applies when all tokens are repeated.
• Data mixing laws (Ye et al., 2025; Xie et al., 2023; Shukor et al., 2025) optimize mixture composition but assume unlimited data per source.
• This work sits at the intersection: jointly optimizing mixture weights and repetition when one source is constrained.

---

Methodology

Experimental Setup

Three data-constrained scenarios were studied:

Models

GPT-2-style autoregressive decoder-only Transformers with 101M to 805M non-embedding parameters, trained for ~100×N tokens (sufficient to observe both benefits and degradation from repetition). Data is mixed at the sample level: each mini-batch samples from target with probability $h$ and generic with probability $1-h$.

Scaling Law Formulation

The core innovation is the effective data concept, adapting Muennighoff et al.'s saturation function:

$$D_T = D_{\mathrm{target}}\big(1 + \rho(r)\big), \quad \text{with} \quad \rho(r) = r_1\Big(1 - e^{-(r-1)/r_1}\Big) \tag{5.1}$$

where $r_1$ controls the decay rate of repeated data value. Total effective data:

$$D_{\mathrm{eff}} = (1-h)D_{\mathrm{total}} + \tau D_T \tag{5.2}$$

where $\tau$ controls the relative value of target vs. generic tokens.

Two loss formulas are proposed:

$$L_{\mathrm{fix}} = E + \frac{A}{D_{\mathrm{eff}}^\alpha} + \gamma h, \quad L_{\mathrm{size}} = E + \frac{C}{N^\beta} + \frac{B N^\delta}{D_{\mathrm{eff}}^\alpha} + \gamma h \tag{5.3}$$

• $L_{\mathrm{fix}}$: 6 parameters $(E, A, \alpha, r_1, \tau, \gamma)$ for fixed model size
• $L_{\mathrm{size}}$: 9 parameters $(E, C, \beta, B, \delta, \alpha, r_1, \tau, \gamma)$ for variable model size
• The term $\gamma h$ is a linear penalty on target weight
• The coupling term $N^\delta$ ($\delta > 0$) encodes that larger models overfit faster

Fitting: Parameters estimated by minimizing reweighted Huber loss via basin-hopping with 100 random restarts:

$$\hat{\theta} = \arg\min_{\theta} \sum_i \omega_i \cdot \mathcal{H}\big(\ell_i - L_{\theta}(D_{\mathrm{total},i}, h_i, D_{\mathrm{target},i}, N_i)\big) \tag{5.4}$$

with weights $\omega_i = \max(r_i \cdot h_i, \epsilon)$ emphasizing the high-repetition regime.

---

Empirical Validation / Results

Key Empirical Findings

1. Repetition-driven overfitting is predictable: Across all settings (languages, domains, model sizes), overfitting onset is governed solely by the repetition factor $r$—not by how $r$ is achieved (which combination of $h$ and $D_{\mathrm{target}}$).

2. Mixture training unlocks high repetition tolerance: Optimal repetition reaches 15–20× (vs. <4× for single-source), increasing steadily with training budget. The target weight needed varies widely: from 9.5% (101M model) to 1.9% (539M model) for 50M target tokens.

3. Larger models overfit faster but still win: At fixed $h$, larger models plateau/overfit earlier, yet the best-loss envelope consistently favors larger models at every training budget.

4. Broadening quality filters beats repetition: When high-quality data is scarce, accepting slightly lower-quality data (broader filter) outperforms repeating a narrow high-quality slice. The optimal filter threshold shifts with training budget (Q95-99 best early, Q90-99 overtakes by 30B tokens: loss 2.33 vs. 2.80 for Q99-100).

Scaling Law Performance

Table 1: Test weighted $R^2$ (higher is better)

$L_{\mathrm{fix}}$ consistently outperforms baselines, especially in the quality-filtered setting where repetition-agnostic and utility-decay baselines fail dramatically.

Table 2: Fraction of training tokens wasted (lower is better)

The scaling law wastes only 26% median compute vs. 88% for repetition-agnostic baselines, demonstrating substantial practical savings.

Multi-Domain Extension

• Proportional weighting (target fractions ∝ dataset sizes) outperforms equal weighting when data is scarce; equal weighting wins in data-rich regimes.
• Optimal repetition grows with training budget (5× less data → $r \approx 30$; 5× more data → $r \approx 7$).
• A 10% compute confidence band shows that rough estimates of optimal $r$ yield near-optimal losses.
• Independent optimization per domain using the bilingual scaling law ($r_{\mathrm{wiki}} = 35$, $r_{\mathrm{peS2o}} = 17$) outperforms the best grid-searched proportional weighting, approaching the 2-domain oracle.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Unified framework: The scaling law bridges two previously separate research lines—data-constrained training (Muennighoff et al., 2023) and mixture optimization (Shukor et al., 2025)—showing that repetition and mixture composition must be optimized jointly.
2. Regularization mechanism: The unsaturated $(1-h)D_{\mathrm{total}}$ term in $D_{\mathrm{eff}}$ formalizes why generic data regularizes training: fresh generic tokens keep effective data growing even when target contribution saturates.
3. Predictable scaling: Best achievable loss follows remarkably regular curves as a function of model size and available target data, suggesting that "collecting more target data" and "scaling up the model" have predictable, complementary effects.

Practical Implications

1. Eliminates expensive sweeps: The scaling law can be fitted at small scales and extrapolated to larger scales, replacing per-configuration grid searches with a single principled prediction.
2. Compute savings: Up to 74% of training compute can be saved at the 90th percentile compared to naive baselines.
3. Quality vs. quantity guidance: When building domain datasets, practitioners should prefer broader filters (more unique tokens) over aggressive repetition of narrow high-quality slices, especially for longer training runs.
4. Multi-domain mixtures: Independent per-domain optimization using the two-domain law is sufficient—no need for combinatorial sweeps over all domain weight combinations.

---

Conclusion

This work provides the first systematic study of mixture pretraining under data constraints, spanning multilingual, multi-domain, and quality-filtered settings across 2,000+ training runs. The central findings are:

1. Repetition is the primary driver of target-domain performance in mixtures, with optimal repetition reaching 15–20×—far exceeding single-source limits—thanks to generic data regularization.
2. A repetition-aware scaling law accurately predicts target-domain loss and optimal mixture configurations, outperforming baselines that ignore repetition or domain structure.

Future Directions

• Extending the framework to more than two constrained domains with formal multi-domain optimization theory
• Investigating whether the scaling law transfers to decoder-only instruction-tuned models and continued pretraining
• Exploring the interaction between synthetic data generation and repetition under data constraints
• Validating the law at larger model scales (billions of parameters) where extrapolation behavior may differ

The work establishes that effective training on scarce data is not merely about collecting more data, but about optimally balancing repetition and mixture composition—a principle that scales predictably across data types, model sizes, and domain configurations.