Summary of "Internal Data Repetition Destroys Language Models"

Summary (Overview)

• Core finding: At fixed compute and fixed repeated-token fraction (f = 0.1), language model eval loss is non-monotonic in the per-document repeat count R, peaking at an intermediate repeat count rather than at extremes.
• Quantified damage via compute-equivalent metrics: Using a fitted no-repetition Chinchilla scaling law, the most damaging repeat setting at 344M parameters and OT = 1 yields Compute-Equivalent Loss (CEL) ≈ 0.33, meaning the run reaches the loss of a no-repetition run using only ~67% of the FLOPs.
• Power-law scaling of peak location: The most damaging repeat count follows $R^{\text{peak}} \propto N^{-0.96}$, meaning larger models peak at fewer repeats of larger repeated pools.
• Statistical universality: The same non-monotonic peak is reproduced in closed form in a misspecified linear regression with verbatim duplicates, suggesting the phenomenon is a generic statistical effect of sample duplication, not a transformer-specific artifact.
• Practical implication: Merely reporting the fraction of duplicated tokens is insufficient—the concentration structure of repetition (how many times each duplicate appears) substantially affects compute waste.

---

Introduction and Theoretical Foundation

Background and Motivation

Pretraining has entered a data-constrained regime: high-quality public text corpora are exhausted, forcing multi-epoch training. Even aggressively deduplicated corpora (FineWeb-Edu, DataComp-LM, Dolma, RedPajama-v2) retain near-duplicates, paraphrased templates, and semantically redundant content.

The key theoretical foundation is the Chinchilla budget identity:

$$C \approx 6NT = 120 \cdot \mathrm{OT} \cdot N^2 \tag{1}$$

where:

• $C$ = total compute (FLOPs)
• $N$ = model parameter count
• $T$ = total training tokens
• $\mathrm{OT}$ = overtraining multiplier (OT = 1 corresponds to 20 tokens per parameter)

This identity allows the authors to vary repetition structure inside an otherwise fixed training budget, separating repetition concentration effects from total compute.

Prior Work and Its Limitations

The closest prior study (Hernandez et al. [11]) established non-monotonic degradation from repetition but:

• Predated Chinchilla-style scaling laws
• Framed damage as "reduction in effective parameter count" (outdated)
• Trained all models on 300B tokens regardless of parameter count, leaving small models overtrained and large models undertrained

Key Definitions

Compute-Equivalent Gain (CEG) for a repeated-data run with loss $L$ and actual compute $C_{\text{actual}}$:

$$\mathrm{CEG} = \frac{C^{\star}(L)}{C_{\text{actual}}}$$

where $C^{\star}(L)$ is the compute a no-repetition run would need to reach loss $L$.

Compute-Equivalent Loss: $\mathrm{CEL} = 1 - \mathrm{CEG}$

• $\mathrm{CEG} = 1$: matches no-repetition baseline
• $\mathrm{CEG} < 1$: indicates compute-equivalent loss

---

Methodology

Experimental Setup

• Models: Qwen3-style decoder-only transformers with $N \in \{34, 48, 63, 93, 153, 344\}$M parameters
• Data: FineWeb-Edu-Dedup
• Overtraining multipliers: $\mathrm{OT} \in \{0.25, 0.5, 1, 2, 4\}$
• Repeat counts: logarithmic grid from no repeats up to R = 20,000
• Repeated-token fraction: fixed at $f = 0.1$ (10% of training tokens from repeated pool)

Repeated-Pool Construction

For each run, $(1-f)T$ tokens come from non-repeated documents. The repeated-pool size $D_r$ and repeat count $R$ satisfy:

$$fT \approx R D_r \quad \Rightarrow \quad D_r \approx \frac{fT}{R} = \frac{2 \cdot \mathrm{OT} \cdot N}{R} \tag{2}$$

Increasing $R$ concentrates the same 10% repeated-token budget onto a smaller pool. Repeated documents are:

• Sampled at document granularity
• Disjoint from non-repeated training documents
• Randomly interleaved with the non-repeated stream

Evaluation

• Fixed held-out split of ~150M tokens, excluded from all training data and repeated pools
• No-repetition baselines trained for each (N, OT) sweep
• Six OT = 1 no-repetition baselines calibrate the Chinchilla scaling law

Scaling Law Fitting

Peak locations extracted via log-Gaussian fits:

$$L(x) = b + A \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right), \quad R^{\text{peak}} = 10^{\mu} \tag{3}$$

---

Empirical Validation / Results

4.1 Non-Monotonic Eval Loss

• Raw maximum: 1.0–4.2% above no-repetition baseline (median 3.1%)
• Peak prominence (relative to endpoints): 0.7–2.7% (median 1.8%)
• The intermediate-repeat regime is most damaging; both extremes (few repeats of large pool, many repeats of tiny pool) produce smaller loss increases

4.2 Power-Law Scaling of Peak Location

Fitted power laws:

$$R^{\text{peak}} = 2.31 \times 10^{10} N^{-0.96}, \quad D_r^{\text{peak}} = 7.58 \times 10^{-10} N^{1.84} \tag{4}$$ $$R^{\text{peak}} = 1.47 \times 10^{7} C^{-0.25}, \quad D_r^{\text{peak}} = 5.49 \times 10^{-12} C^{0.93} \tag{5}$$

Observed trend: The 34M model peaks at $R \approx 1400$ with $D_r \approx 5 \times 10^4$ tokens; the 344M model peaks at $R \approx 155$ with $D_r \approx 4.5 \times 10^6$ tokens. The peak location is approximately OT-independent.

4.3 Compute-Equivalent Loss

Fitted no-repetition Chinchilla scaling law:

$$L(C) = 2.365 + 6.647 \times 10^{5} C^{-0.317} \tag{6}$$

Key results at OT = 1 (worst-case CEL by model size):

Critical insights:

• The 2–4% loss bump translates to CEL of 0.19–0.33 because the scaling law is shallow—small loss gaps map to large compute gaps
• Loss-space views systematically understate practical cost
• Varying OT shifts CEG levels but leaves peak location in R approximately unchanged

Sensitivity analysis: With $\gamma \approx 0.32$, a 1% shift in $(L-E)$ produces ~3% relative shift in CEG. The 344M peak run sits at $L - E \approx 0.65$ nats, making the headline 33% robust.

4.4–4.5 Statistical Model: Misspecified Linear Regression

Setup: High-dimensional linear model with isotropic Gaussian inputs $\mathbf{x} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_p)$, noiseless labels $y = x^\top \beta$. The learner observes only the first $m < p$ coordinates (misspecification). Training set: $n$ unique examples plus $d$ examples each repeated $r$ times.

Key insight—block-diagonal noise covariance: The $r$ copies of each repeated example share a single unobserved-feature realization, giving:

$$\Sigma_r = I_n \oplus \bigoplus_{i=1}^{d} \mathbf{1}_r \mathbf{1}_r^{\top} \in \mathbb{R}^{(n+rd) \times (n+rd)} \tag{8}$$

This yields $X_{\text{in}}^{\top}\Sigma_r X_{\text{in}} = C_u + r^2 C_d$—the extra factor of $r$ makes duplication qualitatively different from adding independent samples.

Closed-form conditional risks:

$$\mathbb{E}[L_{\text{train}} \mid X_{\text{in}}] = \frac{\|\beta_{\text{out}}\|_2^2}{n+rd}\left[n + rd - \mathrm{tr}\left((C_u + r^2 C_d)(C_u + rC_d)^{-1}\right)\right] \tag{9}$$ $$\mathbb{E}[L_{\text{test}} \mid X_{\text{in}}] = \|\beta_{\text{out}}\|_2^2\left[1 + \mathrm{tr}\left((C_u + rC_d)^{-1}(C_u + r^2 C_d)(C_u + rC_d)^{-1}\right)\right] \tag{10}$$

Simulation results:

• Closed-form and simulation agree to numerical precision
• Excess loss is non-monotonic in repeated-pool size $d$ at fixed $m$ and $r$
• Peak shifts to larger $d$ as $m$ (capacity) grows—consistent with empirical $D_r^{\text{peak}}(N)$ trend
• Sample-efficiency (SE) analogue of CEG falls sharply at intermediate $r$ and partially recovers at extremes, mirroring the language-model CEG curve

Mechanism: When $r$ is small, repeated examples carry little extra weight. When $r$ is too large, the repeated block saturates the rank of $C_u + r^2 C_d$ relative to $C_u + rC_d$, and test loss returns toward a "memorize-and-isolate" fixed point. The harmful middle regime appears when the repeated pool is both influential and too large to be harmlessly absorbed.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Modernized measurement framework: Replaces "effective parameter count" with compute-equivalent metrics (CEG/CEL), enabling fair comparison across model sizes and training durations in the Chinchilla era.

2. Statistical universality: Demonstrates that repetition damage peaks are not transformer-specific but arise from a general statistical tradeoff between memorization and generalization in misspecified models with duplicated samples.

3. Scaling law for worst-case configuration: The power-law relationship $R^{\text{peak}} \propto N^{-0.96}$ makes the most damaging repetition structure predictable from model size alone.

Practical Implications

1. Corpus curation guidance: Aggressive deduplication is insufficient—the structure of residual repetition matters. A moderate number of repeats of a moderately sized pool is more damaging than extreme configurations.

2. Compute accounting: Practitioners should report repetition structure (repeat count and pool size), not just duplicated-token fractions, when documenting training runs.

3. Risk assessment: At 344M parameters, a 10% repeated-token budget at the worst repeat count wastes ~33% of training compute—a substantial cost that scales with model size.

4. Caveat for scaling extrapolation: The authors note that since peak repeated-token count grows faster than compute, the peak will eventually cross $0.1T$ for very large models, implying the scaling law should be interpreted as evidence that "memorization capacity grows faster than compute" rather than an exact predictive tool.

---

Conclusion

This paper provides a precise, compute-based quantification of repetition damage in language model pretraining. Key takeaways:

1. Intermediate repeats are worst: Eval loss peaks at intermediate repeat counts, not at extremes—a moderately sized repeated pool replayed many times is more damaging than either a tiny pool replayed many times or a large pool replayed a few times.

2. Larger models are more sensitive: Peak damage occurs at fewer repeats of larger pools as model size grows, following $R^{\text{peak}} \propto N^{-0.96}$.

3. Compute waste is substantial: The most damaging configuration at 344M parameters wastes ~33% of training compute (CEL ≈ 0.33).

4. The phenomenon is generic: A misspecified linear regression with verbatim duplicates reproduces the same qualitative peak, indicating a fundamental statistical tradeoff between memorization and generalization.

Future directions implied by this work include:

• Extending the analysis to near-duplicate and semantic duplication (beyond exact document replay)
• Studying repetition damage at frontier model scales
• Developing explicit overfitting penalties for data-constrained scaling regimes
• Investigating whether the power-law scaling of peak damage extends beyond the studied range