Summary (Overview)

• Core finding: The paper identifies repetition mismatch as a primary cause of failure when extrapolating data mixture experiments from small-scale to target training budgets in data-constrained settings.
• Key mechanism: When high-quality data is scarce, its repetition rate changes as the training budget grows, shifting the optimal mixture in ways that small-scale proxy experiments fail to anticipate.
• Proposed solution: A repetition-aware subsampling procedure that matches the target repetition rate while using only a fraction of the total tokens recovers near-optimal mixtures from minimal compute.
• Main results: A single repetition-controlled experiment using ~1/16 of target tokens recovers mixtures within 0.05 of the optimum for a 757M parameter model (vs. error of 0.75 without repetition control); with three data sources, two repetition-controlled horizons suffice at 757M scale.
• Implication: Data repetition should be treated as a first-class variable in mixture optimization rather than an incidental side effect of limited data.

Introduction and Theoretical Foundation

Background

Pre-training data mixtures significantly impact language model performance (Miranda et al., 2025). Typical corpora combine noisy web crawl data with cleaner, high-quality sources, and finding the optimal balance is crucial yet costly (Shukor et al., 2025). A common strategy is running smaller-scale experiments and extrapolating results to the target budget, but this frequently fails (Kang et al., 2025).

The Repetition Mismatch Problem

The paper identifies repetition mismatch as the key confounder:

• When high-quality data is limited, it must be repeated many times during training.
• The number of repetitions changes as the training budget grows.
• Standard scaling-based mixture selection ignores this effect, imposing a fundamentally different repetition regime on high-quality data in proxy experiments versus target runs.

Theoretical Foundations

The work builds on two strands of prior research:

1. Data mixing methods: Scaling law-based approaches (Ge et al., 2025; Shukor et al., 2025; Ye et al., 2025), proxy model methods (Xie et al., 2023; Fan et al., 2024), and regression-based approaches (Liu et al., 2025a,b) that predict mixtures from small-scale experiments.

2. Data repetition effects: Muennighoff et al. (2023) showed that up to ~4 repetitions of a dataset are as effective as new data, with diminishing returns beyond; Xue et al. (2023) found that larger models are more susceptible to overfitting from excessive repetition.

The paper builds on a repetition-aware subsampling procedure introduced by Li et al. (2025), but isolates repetition mismatch as a distinct phenomenon and characterizes when it matters.

Methodology

Datasets

Validation is performed exclusively on high-quality domains to isolate repetition and mixture effects.

Models

Modified NanoGPT (GPT-2 architecture) with Muon optimizer and RoPE embeddings, at four scales:

• 30M, 124M, 345M, and 757M parameters

Data Mixing Objective

Let $\mathcal{D} = \{\mathcal{D}_1, ..., \mathcal{D}_n\}$ be a set of datasets and $T_\star$ the target training horizon. The goal is to find the optimal target mixture vector $\mathbf{m}^*(T_\star) = [m_1^*, ..., m_n^*]$ with $\sum m_i^* = 1$ and $0 \leq m_i^* \leq 1$.

$$\mathbf{m}^*(T_\star) \in \arg\min_{\mathbf{m} \in \Delta^{n-1}} \mathcal{L}(\mathbf{m}; T_\star)$$

where $\Delta^{n-1} = \{\mathbf{m} \in \mathbb{R}_{>0}^n : \sum_{i=1}^n m_i = 1\}$ is the probability simplex and $\mathcal{L}(\mathbf{m}; T_\star)$ is the average cross-entropy loss on held-out validation data.

Two Experimental Conditions

Without Repetition Control (Scaling Laws):

• Identify optimal mixture at each of 5 training horizons (each ~doubling in length)
• Predict target-horizon mixture via direct extrapolation (1-horizon) or linear regression over 2, 3, or 4 smallest horizons
• Sweep mixing ratios in increments of 0.05 until a U-shaped validation loss curve emerges

With Repetition Control (Repeat-aware):

• Apply subsampling at document level using fractions $\frac{1}{16}, \frac{1}{8}, \frac{1}{4}, \frac{1}{2}$
• Key property: for target horizon $T_\star$, dataset length $n_D$, mixture proportion $h$, and subsample fraction $\frac{1}{S}$:

$$\frac{T_\star \times \frac{1}{S} \times h}{n_D \times \frac{1}{S}} = \frac{T_\star \times h}{n_D}$$

This preserves the same number of repetitions as the full scenario while using only $\frac{1}{S}$ of total tokens.

Three-Source Setting

Adds both WikiText and PubMed as high-quality sources alongside FineWeb, with evaluation averaging losses on both validation sets. Baselines are derived from two-source optima (using target-horizon information, making comparisons conservative).

Empirical Validation / Results

Two-Source Results

Key finding: Without repetition control, the optimal proportion of high-quality data decreases as training budget increases, reflecting diminishing returns from excessive repetition. With repetition control, optimal mixing ratios cluster tightly across horizons.

Table 1: Prediction Errors (distance from optimal mixture)

Single-horizon predictions: Repetition control recovers mixtures within 0.05–0.10 of optimum for 757M model (vs. 0.65–0.75 without), using only ~232M tokens vs. 3.74B at target.

Model capacity effect: Benefit of repetition control strengthens monotonically from 124M to 757M; at 30M, scaling-law extrapolation performs better, establishing a lower bound. The optimal repetition count at target ranges from ~5 (757M) to ~24 (30M).

Three-Source Results

Table 2: Key Three-Source Results at Full Training Horizon

Key results:

• Single horizon is insufficient with three sources (loss gap ~0.08–0.10 from optimum)
• Two horizons suffice at 757M: recovers target optimum at sweep granularity, outperforming baselines
• At 124M, multiple horizons progressively close the gap; four-horizon prediction effectively matches optimum (2.91950 vs. 2.91820)

Compute Cost

• Two-source: repetition control achieves error of 0.05 with ~6% of target token budget; without it, comparable accuracy requires 44–94% of the budget
• Three-source: two repetition-controlled horizons (~19% of target budget) recover optimum at 757M

Theoretical and Practical Implications

Theoretical Contributions

1. Identifies repetition mismatch as a distinct confounder in data mixing research, previously unrecognized in the literature
2. Shows repetition rate is model-size-dependent: the ~4-repetition threshold from Muennighoff et al. (2023) varies with model capacity
3. Demonstrates that scale alone doesn't determine transferability of mixture experiments—repetition dynamics are equally important

Practical Implications

1. Simple, orthogonal intervention: Repetition control operates at the dataset level, requiring no parametric modeling or proxy training runs, and can be incorporated into any existing mixture prediction method
2. Significant compute savings: Reduces experimental budgets needed for mixture selection, particularly valuable as data mixing studies become standard for billion-parameter pre-training
3. Growing importance with scale: The advantage of repetition control strengthens with model size, making it increasingly relevant for modern LLM training

Conclusion

The paper demonstrates that repetition mismatch is the primary culprit behind the failure of small-scale mixture experiments to transfer to larger training budgets in data-constrained settings. Controlling for repetition via subsampling that matches target repetition rates resolves the extrapolation problem:

• A single repetition-controlled horizon using ~1/16 of target tokens recovers near-optimal mixtures at 757M scale
• The advantage strengthens monotonically with model capacity
• In three-source settings, as few as two horizons suffice at larger scales

The authors argue that data repetition deserves treatment as a primary variable in mixture optimization, not an inconvenient side effect of limited data. Future directions include:

• Empirical confirmation at billion-parameter scale
• Extension to more diverse mixtures (7+ sources typical of real corpora)
• Investigation of multilingual settings where data scarcity is more acute
• Combining repetition-aware subsampling with parametric mixture prediction methods
• Validation on downstream benchmarks rather than just high-quality domain loss