# Internal Data Repetition Destroys Language Models

> Language model evaluation loss peaks at intermediate per-document repeat counts, wasting up to 33% of training compute, with the most damaging repeat count scaling as N to the power of negative 0.96.

- **Source:** [arXiv](https://arxiv.org/abs/2606.24998)
- **Published:** 2026-08-18
- **Permalink:** https://picx.dev/p/EETT3k
- **Whiteboard:** https://picx.dev/p/EETT3k/image

## Summary

# Summary of "Internal Data Repetition Destroys Language Models"

## Summary (Overview)

- **Core finding**: At fixed compute and fixed repeated-token fraction (f = 0.1), language model eval loss is **non-monotonic** in the per-document repeat count R, peaking at an **intermediate** repeat count rather than at extremes.
- **Quantified damage via compute-equivalent metrics**: Using a fitted no-repetition Chinchilla scaling law, the most damaging repeat setting at 344M parameters and OT = 1 yields **Compute-Equivalent Loss (CEL) ≈ 0.33**, meaning the run reaches the loss of a no-repetition run using only ~67% of the FLOPs.
- **Power-law scaling of peak location**: The most damaging repeat count follows $R^{\text{peak}} \propto N^{-0.96}$, meaning larger models peak at *fewer* repeats of *larger* repeated pools.
- **Statistical universality**: The same non-monotonic peak is reproduced in closed form in a misspecified linear regression with verbatim duplicates, suggesting the phenomenon is a generic statistical effect of sample duplication, not a transformer-specific artifact.
- **Practical implication**: Merely reporting the fraction of duplicated tokens is insufficient—the *concentration structure* of repetition (how many times each duplicate appears) substantially affects compute waste.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Pretraining has entered a **data-constrained regime**: high-quality public text corpora are exhausted, forcing multi-epoch training. Even aggressively deduplicated corpora (FineWeb-Edu, DataComp-LM, Dolma, RedPajama-v2) retain near-duplicates, paraphrased templates, and semantically redundant content.

The key theoretical foundation is the **Chinchilla budget identity**:

$$C \approx 6NT = 120 \cdot \mathrm{OT} \cdot N^2 \tag{1}$$

where:
- $C$ = total compute (FLOPs)
- $N$ = model parameter count
- $T$ = total training tokens
- $\mathrm{OT}$ = overtraining multiplier (OT = 1 corresponds to 20 tokens per parameter)

This identity allows the authors to vary repetition structure *inside* an otherwise fixed training budget, separating repetition concentration effects from total compute.

### Prior Work and Its Limitations

The closest prior study (Hernandez et al. [11]) established non-monotonic degradation from repetition but:
- Predated Chinchilla-style scaling laws
- Framed damage as "reduction in effective parameter count" (outdated)
- Trained all models on 300B tokens regardless of parameter count, leaving small models overtrained and large models undertrained

### Key Definitions

**Compute-Equivalent Gain (CEG)** for a repeated-data run with loss $L$ and actual compute $C_{\text{actual}}$:

$$\mathrm{CEG} = \frac{C^{\star}(L)}{C_{\text{actual}}}$$

where $C^{\star}(L)$ is the compute a no-repetition run would need to reach loss $L$.

**Compute-Equivalent Loss**: $\mathrm{CEL} = 1 - \mathrm{CEG}$

- $\mathrm{CEG} = 1$: matches no-repetition baseline
- $\mathrm{CEG} < 1$: indicates compute-equivalent loss

---

## Methodology

### Experimental Setup

- **Models**: Qwen3-style decoder-only transformers with $N \in \{34, 48, 63, 93, 153, 344\}$M parameters
- **Data**: FineWeb-Edu-Dedup
- **Overtraining multipliers**: $\mathrm{OT} \in \{0.25, 0.5, 1, 2, 4\}$
- **Repeat counts**: logarithmic grid from no repeats up to R = 20,000
- **Repeated-token fraction**: fixed at $f = 0.1$ (10% of training tokens from repeated pool)

### Repeated-Pool Construction

For each run, $(1-f)T$ tokens come from non-repeated documents. The repeated-pool size $D_r$ and repeat count $R$ satisfy:

$$fT \approx R D_r \quad \Rightarrow \quad D_r \approx \frac{fT}{R} = \frac{2 \cdot \mathrm{OT} \cdot N}{R} \tag{2}$$

Increasing $R$ concentrates the same 10% repeated-token budget onto a smaller pool. Repeated documents are:
- Sampled at document granularity
- Disjoint from non-repeated training documents
- Randomly interleaved with the non-repeated stream

### Evaluation

- Fixed held-out split of ~150M tokens, excluded from all training data and repeated pools
- No-repetition baselines trained for each (N, OT) sweep
- Six OT = 1 no-repetition baselines calibrate the Chinchilla scaling law

### Scaling Law Fitting

Peak locations extracted via log-Gaussian fits:

$$L(x) = b + A \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right), \quad R^{\text{peak}} = 10^{\mu} \tag{3}$$

---

## Empirical Validation / Results

### 4.1 Non-Monotonic Eval Loss

- **Raw maximum**: 1.0–4.2% above no-repetition baseline (median 3.1%)
- **Peak prominence** (relative to endpoints): 0.7–2.7% (median 1.8%)
- The intermediate-repeat regime is most damaging; both extremes (few repeats of large pool, many repeats of tiny pool) produce smaller loss increases

### 4.2 Power-Law Scaling of Peak Location

Fitted power laws:

$$R^{\text{peak}} = 2.31 \times 10^{10} N^{-0.96}, \quad D_r^{\text{peak}} = 7.58 \times 10^{-10} N^{1.84} \tag{4}$$

$$R^{\text{peak}} = 1.47 \times 10^{7} C^{-0.25}, \quad D_r^{\text{peak}} = 5.49 \times 10^{-12} C^{0.93} \tag{5}$$

**Observed trend**: The 34M model peaks at $R \approx 1400$ with $D_r \approx 5 \times 10^4$ tokens; the 344M model peaks at $R \approx 155$ with $D_r \approx 4.5 \times 10^6$ tokens. The peak location is approximately **OT-independent**.

### 4.3 Compute-Equivalent Loss

Fitted no-repetition Chinchilla scaling law:

$$L(C) = 2.365 + 6.647 \times 10^{5} C^{-0.317} \tag{6}$$

**Key results at OT = 1** (worst-case CEL by model size):

| Model Size (N) | 34M | 48M | 63M | 93M | 153M | 344M |
|---|---|---|---|---|---|---|
| **CEL** | 0.19 | 0.19 | 0.21 | 0.21 | 0.26 | **0.33** |

**Critical insights**:
- The 2–4% loss bump translates to CEL of 0.19–0.33 because the scaling law is **shallow**—small loss gaps map to large compute gaps
- Loss-space views systematically **understate** practical cost
- Varying OT shifts CEG levels but leaves peak location in R approximately unchanged

**Sensitivity analysis**: With $\gamma \approx 0.32$, a 1% shift in $(L-E)$ produces ~3% relative shift in CEG. The 344M peak run sits at $L - E \approx 0.65$ nats, making the headline 33% robust.

### 4.4–4.5 Statistical Model: Misspecified Linear Regression

**Setup**: High-dimensional linear model with isotropic Gaussian inputs $\mathbf{x} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_p)$, noiseless labels $y = x^\top \beta$. The learner observes only the first $m < p$ coordinates (misspecification). Training set: $n$ unique examples plus $d$ examples each repeated $r$ times.

**Key insight—block-diagonal noise covariance**: The $r$ copies of each repeated example share a single unobserved-feature realization, giving:

$$\Sigma_r = I_n \oplus \bigoplus_{i=1}^{d} \mathbf{1}_r \mathbf{1}_r^{\top} \in \mathbb{R}^{(n+rd) \times (n+rd)} \tag{8}$$

This yields $X_{\text{in}}^{\top}\Sigma_r X_{\text{in}} = C_u + r^2 C_d$—the extra factor of $r$ makes duplication qualitatively different from adding independent samples.

**Closed-form conditional risks**:

$$\mathbb{E}[L_{\text{train}} \mid X_{\text{in}}] = \frac{\|\beta_{\text{out}}\|_2^2}{n+rd}\left[n + rd - \mathrm{tr}\left((C_u + r^2 C_d)(C_u + rC_d)^{-1}\right)\right] \tag{9}$$

$$\mathbb{E}[L_{\text{test}} \mid X_{\text{in}}] = \|\beta_{\text{out}}\|_2^2\left[1 + \mathrm{tr}\left((C_u + rC_d)^{-1}(C_u + r^2 C_d)(C_u + rC_d)^{-1}\right)\right] \tag{10}$$

**Simulation results**:
- Closed-form and simulation agree to numerical precision
- Excess loss is **non-monotonic** in repeated-pool size $d$ at fixed $m$ and $r$
- Peak shifts to larger $d$ as $m$ (capacity) grows—consistent with empirical $D_r^{\text{peak}}(N)$ trend
- Sample-efficiency (SE) analogue of CEG falls sharply at intermediate $r$ and partially recovers at extremes, mirroring the language-model CEG curve

**Mechanism**: When $r$ is small, repeated examples carry little extra weight. When $r$ is too large, the repeated block saturates the rank of $C_u + r^2 C_d$ relative to $C_u + rC_d$, and test loss returns toward a "memorize-and-isolate" fixed point. The harmful middle regime appears when the repeated pool is both influential and too large to be harmlessly absorbed.

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Modernized measurement framework**: Replaces "effective parameter count" with compute-equivalent metrics (CEG/CEL), enabling fair comparison across model sizes and training durations in the Chinchilla era.

2. **Statistical universality**: Demonstrates that repetition damage peaks are not transformer-specific but arise from a general statistical tradeoff between memorization and generalization in misspecified models with duplicated samples.

3. **Scaling law for worst-case configuration**: The power-law relationship $R^{\text{peak}} \propto N^{-0.96}$ makes the most damaging repetition structure predictable from model size alone.

### Practical Implications

1. **Corpus curation guidance**: Aggressive deduplication is insufficient—the *structure* of residual repetition matters. A moderate number of repeats of a moderately sized pool is more damaging than extreme configurations.

2. **Compute accounting**: Practitioners should report repetition structure (repeat count and pool size), not just duplicated-token fractions, when documenting training runs.

3. **Risk assessment**: At 344M parameters, a 10% repeated-token budget at the worst repeat count wastes ~33% of training compute—a substantial cost that scales with model size.

4. **Caveat for scaling extrapolation**: The authors note that since peak repeated-token count grows faster than compute, the peak will eventually cross $0.1T$ for very large models, implying the scaling law should be interpreted as evidence that "memorization capacity grows faster than compute" rather than an exact predictive tool.

---

## Conclusion

This paper provides a precise, compute-based quantification of repetition damage in language model pretraining. Key takeaways:

1. **Intermediate repeats are worst**: Eval loss peaks at intermediate repeat counts, not at extremes—a moderately sized repeated pool replayed many times is more damaging than either a tiny pool replayed many times or a large pool replayed a few times.

2. **Larger models are more sensitive**: Peak damage occurs at fewer repeats of larger pools as model size grows, following $R^{\text{peak}} \propto N^{-0.96}$.

3. **Compute waste is substantial**: The most damaging configuration at 344M parameters wastes ~33% of training compute (CEL ≈ 0.33).

4. **The phenomenon is generic**: A misspecified linear regression with verbatim duplicates reproduces the same qualitative peak, indicating a fundamental statistical tradeoff between memorization and generalization.

**Future directions** implied by this work include:
- Extending the analysis to near-duplicate and semantic duplication (beyond exact document replay)
- Studying repetition damage at frontier model scales
- Developing explicit overfitting penalties for data-constrained scaling regimes
- Investigating whether the power-law scaling of peak damage extends beyond the studied range

---

_Markdown view of https://picx.dev/p/EETT3k, served by PicX — AI-generated visual whiteboard summaries of research papers._