# Scaling Laws for Mixture Pretraining Under Data Constraints

> Mixture training with generic data tolerates 15-20 repetitions of scarce target data, far exceeding single-source limits, and a repetition-aware scaling law predicts optimal mixtures saving up to 74% compute.

- **Source:** [arXiv](https://arxiv.org/abs/2605.12715)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/eALen5
- **Whiteboard:** https://picx.dev/p/eALen5/image

## Summary

## Summary (Overview)

- **Core problem**: How to optimally mix scarce, valuable target data (e.g., low-resource languages, specialized domains) with abundant generic data during LLM pretraining, when the target data must be repeated due to its limited size.
- **Key empirical finding**: Across 2,000+ training runs, mixture training tolerates **15–20 repetitions** of target data—far exceeding the widely cited <4 epoch ceiling from single-source training—because generic data acts as an implicit regularizer.
- **Main contribution**: A **repetition-aware mixture scaling law** that predicts target-domain loss as a function of target data size, mixture ratio, and model size, enabling optimal mixture recommendations without expensive hyperparameter sweeps.
- **Additional findings**: Larger models overfit faster but still achieve lower losses; broadening quality filters (accepting slightly lower-quality data for more unique tokens) beats aggressive repetition of narrow high-quality slices; the scaling law extends to multi-domain settings with multiple constrained sources.
- **Practical impact**: Given only target pool size and compute budget, the law prescribes the optimal mixture ratio, saving up to 74% of training compute compared to naive baselines.

---

## Introduction and Theoretical Foundation

### Background
Large language models (LLMs) achieve state-of-the-art performance partly due to massive pretraining corpora (trillions of tokens). However, many real-world scenarios involve **data-constrained domains**—low-resource languages, specialized domains (math, science), or curated high-quality datasets—where unique data is inherently limited.

### The Mixing Trade-off
When scarce target data is mixed with abundant generic data, a fundamental tension emerges:
- **Too little target data** → underexposure to the target domain
- **Too much target data** → excessive repetition of the same examples → memorization → overfitting

This is formalized via the **repetition factor**:

$$r = h \cdot D_{\mathrm{total}} / D_{\mathrm{target}} \tag{3.1}$$

where $h \in [0,1]$ is the fraction of training devoted to target data, $D_{\mathrm{total}}$ is the total training token budget, and $D_{\mathrm{target}}$ is the unique target token count.

### Gap in Prior Work
- **Muennighoff et al. (2023)** derived scaling laws for single-source data-constrained training, showing ~4 repetitions are nearly lossless. However, this applies when *all* tokens are repeated.
- **Data mixing laws** (Ye et al., 2025; Xie et al., 2023; Shukor et al., 2025) optimize mixture composition but assume unlimited data per source.
- **This work** sits at the intersection: jointly optimizing mixture weights *and* repetition when one source is constrained.

---

## Methodology

### Experimental Setup
Three data-constrained scenarios were studied:

| Scenario | Target Domain | Generic Domain | Target Sizes |
|---|---|---|---|
| **Multilingual** | German, French, Swahili (FineWeb2) | English (FineWeb) | 50M–1B tokens |
| **Multi-domain** | OpenWebMath, Wikipedia, peS2o | DCLM, FineWeb, Nemotron-CC | 10M–2.5B tokens |
| **Quality-filtered** | Top {1, 5, 10, 20}% of DCLM (by quality score) | Full DCLM base | 26M–982M tokens |

### Models
GPT-2-style autoregressive decoder-only Transformers with **101M to 805M non-embedding parameters**, trained for ~100×N tokens (sufficient to observe both benefits and degradation from repetition). Data is mixed at the sample level: each mini-batch samples from target with probability $h$ and generic with probability $1-h$.

### Scaling Law Formulation
The core innovation is the **effective data** concept, adapting Muennighoff et al.'s saturation function:

$$D_T = D_{\mathrm{target}}\big(1 + \rho(r)\big), \quad \text{with} \quad \rho(r) = r_1\Big(1 - e^{-(r-1)/r_1}\Big) \tag{5.1}$$

where $r_1$ controls the decay rate of repeated data value. Total effective data:

$$D_{\mathrm{eff}} = (1-h)D_{\mathrm{total}} + \tau D_T \tag{5.2}$$

where $\tau$ controls the relative value of target vs. generic tokens.

**Two loss formulas** are proposed:

$$L_{\mathrm{fix}} = E + \frac{A}{D_{\mathrm{eff}}^\alpha} + \gamma h, \quad L_{\mathrm{size}} = E + \frac{C}{N^\beta} + \frac{B N^\delta}{D_{\mathrm{eff}}^\alpha} + \gamma h \tag{5.3}$$

- $L_{\mathrm{fix}}$: 6 parameters $(E, A, \alpha, r_1, \tau, \gamma)$ for fixed model size
- $L_{\mathrm{size}}$: 9 parameters $(E, C, \beta, B, \delta, \alpha, r_1, \tau, \gamma)$ for variable model size
- The term $\gamma h$ is a linear penalty on target weight
- The coupling term $N^\delta$ ($\delta > 0$) encodes that larger models overfit faster

**Fitting**: Parameters estimated by minimizing reweighted Huber loss via basin-hopping with 100 random restarts:

$$\hat{\theta} = \arg\min_{\theta} \sum_i \omega_i \cdot \mathcal{H}\big(\ell_i - L_{\theta}(D_{\mathrm{total},i}, h_i, D_{\mathrm{target},i}, N_i)\big) \tag{5.4}$$

with weights $\omega_i = \max(r_i \cdot h_i, \epsilon)$ emphasizing the high-repetition regime.

---

## Empirical Validation / Results

### Key Empirical Findings

**1. Repetition-driven overfitting is predictable**: Across all settings (languages, domains, model sizes), overfitting onset is governed solely by the repetition factor $r$—not by how $r$ is achieved (which combination of $h$ and $D_{\mathrm{target}}$).

**2. Mixture training unlocks high repetition tolerance**: Optimal repetition reaches **15–20×** (vs. <4× for single-source), increasing steadily with training budget. The target weight needed varies widely: from 9.5% (101M model) to 1.9% (539M model) for 50M target tokens.

**3. Larger models overfit faster but still win**: At fixed $h$, larger models plateau/overfit earlier, yet the best-loss envelope consistently favors larger models at every training budget.

**4. Broadening quality filters beats repetition**: When high-quality data is scarce, accepting slightly lower-quality data (broader filter) outperforms repeating a narrow high-quality slice. The optimal filter threshold shifts with training budget (Q95-99 best early, Q90-99 overtakes by 30B tokens: loss 2.33 vs. 2.80 for Q99-100).

### Scaling Law Performance

**Table 1: Test weighted $R^2$** (higher is better)

| Formula | German | Maths | Quality | Wiki/peS2o |
|---|---|---|---|---|
| **$L_{\mathrm{fix}}$** | **0.95** | **0.88** | **0.71** | **0.80** |
| Repetition-agnostic | 0.78 | 0.78 | 0.14 | 0.72 |
| Utility decay | 0.72 | 0.55 | -0.64 | 0.79 |
| Domain-agnostic | -40.7 | -0.49 | -2.19 | -1.16 |

$L_{\mathrm{fix}}$ consistently outperforms baselines, especially in the quality-filtered setting where repetition-agnostic and utility-decay baselines fail dramatically.

**Table 2: Fraction of training tokens wasted** (lower is better)

| Formula | Median | Mean | p90 |
|---|---|---|---|
| **$L_{\mathrm{fix}}$** | **26%** | **34%** | **76%** |
| Rep-agn. | 88% | 73% | 99% |
| Util. dec. | 31% | 41% | 95% |
| Dom.-agn. | 47% | 59% | 98% |

The scaling law wastes only 26% median compute vs. 88% for repetition-agnostic baselines, demonstrating substantial practical savings.

### Multi-Domain Extension
- **Proportional weighting** (target fractions ∝ dataset sizes) outperforms equal weighting when data is scarce; equal weighting wins in data-rich regimes.
- Optimal repetition grows with training budget (5× less data → $r \approx 30$; 5× more data → $r \approx 7$).
- A **10% compute confidence band** shows that rough estimates of optimal $r$ yield near-optimal losses.
- **Independent optimization per domain** using the bilingual scaling law ($r_{\mathrm{wiki}} = 35$, $r_{\mathrm{peS2o}} = 17$) outperforms the best grid-searched proportional weighting, approaching the 2-domain oracle.

---

## Theoretical and Practical Implications

### Theoretical Contributions
1. **Unified framework**: The scaling law bridges two previously separate research lines—data-constrained training (Muennighoff et al., 2023) and mixture optimization (Shukor et al., 2025)—showing that repetition and mixture composition must be optimized jointly.
2. **Regularization mechanism**: The unsaturated $(1-h)D_{\mathrm{total}}$ term in $D_{\mathrm{eff}}$ formalizes why generic data regularizes training: fresh generic tokens keep effective data growing even when target contribution saturates.
3. **Predictable scaling**: Best achievable loss follows remarkably regular curves as a function of model size and available target data, suggesting that "collecting more target data" and "scaling up the model" have predictable, complementary effects.

### Practical Implications
1. **Eliminates expensive sweeps**: The scaling law can be fitted at small scales and extrapolated to larger scales, replacing per-configuration grid searches with a single principled prediction.
2. **Compute savings**: Up to 74% of training compute can be saved at the 90th percentile compared to naive baselines.
3. **Quality vs. quantity guidance**: When building domain datasets, practitioners should prefer broader filters (more unique tokens) over aggressive repetition of narrow high-quality slices, especially for longer training runs.
4. **Multi-domain mixtures**: Independent per-domain optimization using the two-domain law is sufficient—no need for combinatorial sweeps over all domain weight combinations.

---

## Conclusion

This work provides the first systematic study of **mixture pretraining under data constraints**, spanning multilingual, multi-domain, and quality-filtered settings across 2,000+ training runs. The central findings are:

1. **Repetition is the primary driver** of target-domain performance in mixtures, with optimal repetition reaching 15–20×—far exceeding single-source limits—thanks to generic data regularization.
2. **A repetition-aware scaling law** accurately predicts target-domain loss and optimal mixture configurations, outperforming baselines that ignore repetition or domain structure.

### Future Directions
- Extending the framework to **more than two constrained domains** with formal multi-domain optimization theory
- Investigating whether the scaling law transfers to **decoder-only instruction-tuned models** and **continued pretraining**
- Exploring the interaction between **synthetic data generation** and repetition under data constraints
- Validating the law at **larger model scales** (billions of parameters) where extrapolation behavior may differ

The work establishes that effective training on scarce data is not merely about collecting more data, but about **optimally balancing repetition and mixture composition**—a principle that scales predictably across data types, model sizes, and domain configurations.

---

_Markdown view of https://picx.dev/p/eALen5, served by PicX — AI-generated visual whiteboard summaries of research papers._