# Repetition Mismatch: Why Data Mixture Experiments Don’t Scale and How to Fix Them

> Repetition mismatch, not scale, causes data-mixture extrapolation failures; repetition-controlled subsampling using 1/16 of tokens recovers near-optimal mixtures within 0.05 error at 757M scale.

- **Source:** [arXiv](https://arxiv.org/abs/2606.07597)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/GiafjZ
- **Whiteboard:** https://picx.dev/p/GiafjZ/image

## Summary

## Summary (Overview)

- **Core finding**: The paper identifies **repetition mismatch** as a primary cause of failure when extrapolating data mixture experiments from small-scale to target training budgets in data-constrained settings.
- **Key mechanism**: When high-quality data is scarce, its repetition rate changes as the training budget grows, shifting the optimal mixture in ways that small-scale proxy experiments fail to anticipate.
- **Proposed solution**: A **repetition-aware subsampling procedure** that matches the target repetition rate while using only a fraction of the total tokens recovers near-optimal mixtures from minimal compute.
- **Main results**: A single repetition-controlled experiment using ~1/16 of target tokens recovers mixtures within 0.05 of the optimum for a 757M parameter model (vs. error of 0.75 without repetition control); with three data sources, two repetition-controlled horizons suffice at 757M scale.
- **Implication**: Data repetition should be treated as a **first-class variable** in mixture optimization rather than an incidental side effect of limited data.

## Introduction and Theoretical Foundation

### Background

Pre-training data mixtures significantly impact language model performance (Miranda et al., 2025). Typical corpora combine noisy web crawl data with cleaner, high-quality sources, and finding the optimal balance is crucial yet costly (Shukor et al., 2025). A common strategy is running smaller-scale experiments and extrapolating results to the target budget, but this frequently fails (Kang et al., 2025).

### The Repetition Mismatch Problem

The paper identifies **repetition mismatch** as the key confounder:

- When high-quality data is limited, it must be repeated many times during training.
- The number of repetitions changes as the training budget grows.
- Standard scaling-based mixture selection ignores this effect, imposing a fundamentally different repetition regime on high-quality data in proxy experiments versus target runs.

### Theoretical Foundations

The work builds on two strands of prior research:

1. **Data mixing methods**: Scaling law-based approaches (Ge et al., 2025; Shukor et al., 2025; Ye et al., 2025), proxy model methods (Xie et al., 2023; Fan et al., 2024), and regression-based approaches (Liu et al., 2025a,b) that predict mixtures from small-scale experiments.

2. **Data repetition effects**: Muennighoff et al. (2023) showed that up to ~4 repetitions of a dataset are as effective as new data, with diminishing returns beyond; Xue et al. (2023) found that larger models are more susceptible to overfitting from excessive repetition.

The paper builds on a repetition-aware subsampling procedure introduced by Li et al. (2025), but isolates repetition mismatch as a distinct phenomenon and characterizes when it matters.

## Methodology

### Datasets

| Dataset | Type | Size | Role |
|---------|------|------|------|
| WikiText | High-quality Wikipedia articles | 116.9M tokens | Limited high-quality source |
| PubMed | Biomedical abstracts | 120.0M tokens | Limited high-quality source |
| FineWeb-10BT | Web crawl | ~10B tokens | Abundant noisy source |

Validation is performed exclusively on high-quality domains to isolate repetition and mixture effects.

### Models

Modified NanoGPT (GPT-2 architecture) with Muon optimizer and RoPE embeddings, at four scales:
- 30M, 124M, 345M, and 757M parameters

### Data Mixing Objective

Let $\mathcal{D} = \{\mathcal{D}_1, ..., \mathcal{D}_n\}$ be a set of datasets and $T_\star$ the target training horizon. The goal is to find the optimal target mixture vector $\mathbf{m}^*(T_\star) = [m_1^*, ..., m_n^*]$ with $\sum m_i^* = 1$ and $0 \leq m_i^* \leq 1$.

$$\mathbf{m}^*(T_\star) \in \arg\min_{\mathbf{m} \in \Delta^{n-1}} \mathcal{L}(\mathbf{m}; T_\star)$$

where $\Delta^{n-1} = \{\mathbf{m} \in \mathbb{R}_{>0}^n : \sum_{i=1}^n m_i = 1\}$ is the probability simplex and $\mathcal{L}(\mathbf{m}; T_\star)$ is the average cross-entropy loss on held-out validation data.

### Two Experimental Conditions

**Without Repetition Control (Scaling Laws):**
- Identify optimal mixture at each of 5 training horizons (each ~doubling in length)
- Predict target-horizon mixture via direct extrapolation (1-horizon) or linear regression over 2, 3, or 4 smallest horizons
- Sweep mixing ratios in increments of 0.05 until a U-shaped validation loss curve emerges

**With Repetition Control (Repeat-aware):**
- Apply subsampling at document level using fractions $\frac{1}{16}, \frac{1}{8}, \frac{1}{4}, \frac{1}{2}$
- Key property: for target horizon $T_\star$, dataset length $n_D$, mixture proportion $h$, and subsample fraction $\frac{1}{S}$:

$$\frac{T_\star \times \frac{1}{S} \times h}{n_D \times \frac{1}{S}} = \frac{T_\star \times h}{n_D}$$

This preserves the same number of repetitions as the full scenario while using only $\frac{1}{S}$ of total tokens.

### Three-Source Setting

Adds both WikiText and PubMed as high-quality sources alongside FineWeb, with evaluation averaging losses on both validation sets. Baselines are derived from two-source optima (using target-horizon information, making comparisons conservative).

## Empirical Validation / Results

### Two-Source Results

**Key finding**: Without repetition control, the optimal proportion of high-quality data decreases as training budget increases, reflecting diminishing returns from excessive repetition. With repetition control, optimal mixing ratios cluster tightly across horizons.

**Table 1: Prediction Errors (distance from optimal mixture)**

| High-Quality Dataset | Model Size | Scaling Laws 1-H | Scaling Laws 4-H | Repeat-aware 1-H | Repeat-aware 4-H |
|---------------------|------------|-------------------|-------------------|--------------------|--------------------|
| WikiText | 30M | 0.250 | 0.039 | 0.550 | 0.044 |
| WikiText | 124M | 0.650 | 0.001 | 0.200 | 0.062 |
| WikiText | 345M | 0.750 | ≤0.05 | 0.100 | 0.017 |
| WikiText | 757M | 0.750 | 0.006 | **0.050** | ≤0.05 |
| PubMed | 30M | 0.300 | 0.059 | 0.500 | 0.079 |
| PubMed | 124M | 0.650 | 0.032 | 0.200 | 0.061 |
| PubMed | 345M | 0.750 | 0.012 | 0.100 | 0.016 |
| PubMed | 757M | 0.650 | 0.029 | **0.100** | 0.050 |

**Single-horizon predictions**: Repetition control recovers mixtures within 0.05–0.10 of optimum for 757M model (vs. 0.65–0.75 without), using only ~232M tokens vs. 3.74B at target.

**Model capacity effect**: Benefit of repetition control strengthens monotonically from 124M to 757M; at 30M, scaling-law extrapolation performs better, establishing a lower bound. The optimal repetition count at target ranges from ~5 (757M) to ~24 (30M).

### Three-Source Results

**Table 2: Key Three-Source Results at Full Training Horizon**

| Model Size | Mixing Ratio (FineWeb, WikiText, PubMed) | Avg. Validation Loss | Experiment Type |
|------------|------------------------------------------|---------------------|-----------------|
| 124M | 0.45, 0.25, 0.3 | **2.91820** | Optimal Mixture |
| 124M | 0.51, 0.245, 0.245 | 2.91950 | Four-Horizon Prediction |
| 124M | 0.56, 0.22, 0.22 | 2.92830 | Three-Horizon Prediction |
| 124M | 0.57, 0.215, 0.215 | 2.92965 | Two-Horizon Prediction |
| 757M | 0.65, 0.175, 0.175 | **2.7699** | Optimal / Two-Horizon Prediction |
| 757M | 0.65, 0.15, 0.20 | 2.7751 | Baseline 1 |
| 757M | 0.825, 0.075, 0.100 | 2.8337 | Baseline 2 |

Key results:
- **Single horizon** is insufficient with three sources (loss gap ~0.08–0.10 from optimum)
- **Two horizons suffice at 757M**: recovers target optimum at sweep granularity, outperforming baselines
- **At 124M**, multiple horizons progressively close the gap; four-horizon prediction effectively matches optimum (2.91950 vs. 2.91820)

### Compute Cost

- Two-source: repetition control achieves error of 0.05 with ~6% of target token budget; without it, comparable accuracy requires 44–94% of the budget
- Three-source: two repetition-controlled horizons (~19% of target budget) recover optimum at 757M

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Identifies repetition mismatch as a distinct confounder** in data mixing research, previously unrecognized in the literature
2. **Shows repetition rate is model-size-dependent**: the ~4-repetition threshold from Muennighoff et al. (2023) varies with model capacity
3. **Demonstrates that scale alone doesn't determine transferability** of mixture experiments—repetition dynamics are equally important

### Practical Implications

1. **Simple, orthogonal intervention**: Repetition control operates at the dataset level, requiring no parametric modeling or proxy training runs, and can be incorporated into any existing mixture prediction method
2. **Significant compute savings**: Reduces experimental budgets needed for mixture selection, particularly valuable as data mixing studies become standard for billion-parameter pre-training
3. **Growing importance with scale**: The advantage of repetition control strengthens with model size, making it increasingly relevant for modern LLM training

## Conclusion

The paper demonstrates that **repetition mismatch** is the primary culprit behind the failure of small-scale mixture experiments to transfer to larger training budgets in data-constrained settings. Controlling for repetition via subsampling that matches target repetition rates resolves the extrapolation problem:

- A single repetition-controlled horizon using ~1/16 of target tokens recovers near-optimal mixtures at 757M scale
- The advantage strengthens monotonically with model capacity
- In three-source settings, as few as two horizons suffice at larger scales

The authors argue that **data repetition deserves treatment as a primary variable in mixture optimization**, not an inconvenient side effect of limited data. Future directions include:

- Empirical confirmation at billion-parameter scale
- Extension to more diverse mixtures (7+ sources typical of real corpora)
- Investigation of multilingual settings where data scarcity is more acute
- Combining repetition-aware subsampling with parametric mixture prediction methods
- Validation on downstream benchmarks rather than just high-quality domain loss

---

_Markdown view of https://picx.dev/p/GiafjZ, served by PicX — AI-generated visual whiteboard summaries of research papers._