Summary (Overview)

• Core finding: With sufficient compute, the best data filter for large model pretraining is no filter at all—models trained on unfiltered Common Crawl (CC) outperform those trained on heavily filtered subsets.
• Key mechanism: Sufficiently large models are highly robust to low-quality data; they can absorb "junk" data without performance degradation and even benefit from nominally "poor" data.
• Scaling prediction: The 240-trillion-token CC pool may become optimal over filtered versions at approximately $1 \times 10^{30}$ FLOPs, a compute level predicted to be reachable by ~2030.
• Robustness to junk data: Models trained on data augmented with randomly generated strings or shuffled-word documents close the performance gap with large enough model size and training steps.
• Theoretical support: A low-rank matrix factorization model demonstrates that sufficient model capacity allows noise to be absorbed without penalty, mirroring experimental observations.

---

Introduction and Theoretical Foundation

The paper challenges the prevailing assumption that data filtering is essential for large-scale language model pretraining. The standard approach filters text from sources like Common Crawl (CC), keeping only "high-quality" data—for example, the DCLM-Baseline dataset retains only ~1% of original CC (~3.8T tokens). However, this conflicts with scaling trends that prescribe ever-increasing data amounts.

The authors formalize the goal as finding the best achievable performance:

$$\mathcal{L}^{\star}(D) := \min_{M, N} \ell(\mathcal{A}(D, M, N))$$

where $M$ is parameter count, $N$ is training steps, and $\mathcal{A}$ is the training algorithm. The key hypothesis tested is whether $\mathcal{L}^{\star}(D_{cc,m}) < \mathcal{L}^{\star}(f(D_{cc,m}))$ holds, where $f$ is a filtering function and $D_{cc,m}$ is an $m$-token subset of CC.

Related theoretical work: Cheng et al. [2024] developed theoretical models suggesting that with sufficient model fidelity, not cleaning data is better—a prediction this paper empirically confirms.

---

Methodology

Experimental Setup

• Data: DCLM-Pool (all CC before 2023, 240T GPT-NeoX tokens), with subsets ranging from 670M to 10B tokens.
• Models: Llama-style dense transformers from 15M to 7B parameters, trained with Meta Lingua.
• Filters tested: English filter (28.2% retained), repetition filter (45.3%), stop word filter (50.4%), RefinedWeb (13%), DCLM-Baseline (2.1%).

Evaluation

• Primary metrics: Negative log-likelihood on C4, Fineweb-Edu, and Cosmopedia (averaged).
• Secondary metrics: ARC-Easy and PIQA benchmarks.
• Junk data injection: Randomly generated strings (10,000-word vocabulary of 3–8 character strings) and shuffled-word documents (word order randomized in additional CC documents).

Scaling Law Construction

Two scaling law approaches were used to predict compute thresholds:

1. Token-to-parameter ratio: Fixed at 600:1 (following DeepSeek V4).
2. Epoch constraint: Fixed at 4 epochs (based on Muennighoff et al. [2025]).

---

Empirical Validation / Results

Data Filtering Experiments

• On a 670M-token pool, the unfiltered pool outperforms all five filters for models ≥330M parameters after sufficient training steps (Figure 1).
• The pool transitions from worst to best on the compute-performance Pareto frontier as compute increases (Figure 2).
• Crossing points appear earlier as model size increases.

Data Injection Robustness

• Random strings: Models close the performance gap with sufficient scale, with some gains (+20% random) possibly due to regularization effects.
• Shuffled words: Surprisingly, models ≥330M parameters benefit from shuffled-word documents after ~11B training tokens, despite only the unigram distribution remaining intact.
• Performance does not degrade to random levels ($-\log(1/V) \approx 10.8$ with the tokenizer used).

Pool Size Scaling

• The training steps needed for the pool to beat RefinedWeb ($N^{\star}(M,m)$) grow super-linearly with pool size (Table 1 in paper):
  • 670M pool: ~1 epoch needed
  • 2B pool: ~3 epochs
  • 10B pool: ~10 epochs
• Larger models dramatically reduce the epoch count needed for the pool to win.

Compute Scaling Law Prediction

Both scaling laws give $R^2 > 0.99$ and predict similar budgets (~$1 \times 10^{30}$ FLOPs), compared to current frontier compute near $5 \times 10^{26}$ FLOPs.

Factuality Analysis (Table 1 in paper)

Average GPT5-mini judgements on keyword-matched CC data for MMLU categories:

Supporting documents are at least an order of magnitude more common than refuting ones, suggesting harmful content is relatively rare in CC.

---

Theoretical and Practical Implications

Theoretical Model (Proposition 7.1)

The paper presents a low-rank matrix factorization model where $k$ tasks generate data with orthogonal inputs. For any model rank $r$:

$$\min_{\substack{U\in \mathbb{R}^{m\times r}\\ V\in \mathbb{R}^{d\times r}}}\mathbb{E}\big[ \| Y - UV^{\top}X\|^{2}\big] = \sum_{j = r + 1}^{\rho}\sigma_{j}^{2} + \mathbb{E}\big[ \| \xi \|^{2}\big]$$

where $\sigma_j$ are singular values of $M_{\star}\Sigma^{1/2}$. This shows that:

• With enough capacity ($r \geq \rho$), noise is absorbed without penalty.
• Without enough capacity, performance necessarily degrades due to task interference.
• Gradient-based methods converge to optimal solutions (no non-strict saddle points when $r \geq k$).

Practical Implications

1. Data filtering may suffer from the "bitter lesson": Human-designed filters that work at small scale may be replaced by simple no-filter approaches that scale more gracefully with compute.
2. Compute thresholds matter: Filtering remains important when compute is constrained; the benefits of no-filtering only manifest at very high compute levels.
3. Distribution shift robustness: Language models are highly resistant to covariate shifts; it is "incorrectly labeled" data or conditional distribution shifts that are detrimental.

---

Conclusion

The paper provides strong evidence that data filtering may become unnecessary at sufficient compute scales. Key takeaways:

1. No filter is best with enough compute: Sufficiently trained large models benefit from nominally "poor" data in Common Crawl.
2. Predictable scaling: The compute threshold for no-filter optimality follows clean scaling laws ($R^2 > 0.99$), predicting ~$1 \times 10^{30}$ FLOPs for the full CC pool.
3. Robustness is capacity-dependent: Large models can extract signal even from shuffled-word data; small models cannot.

Future Directions

• Deviations from vanilla pretraining: MoE architectures, synthetic data, and post-training may alter conclusions.
• Duplicate documents: Larger pools have more duplicates, though epoching mitigates this.
• AI-generated content: Increasing AI content in CC may change the calculus.
• Factuality edge cases: Rare instances of harmful content may still warrant targeted filtering.

The authors conclude that while it is possible to construct harmful data (e.g., non-factual content that looks like high-quality data), such content is not prevalent in CC, suggesting that data filtering may eventually follow the bitter lesson: simple, no-filter approaches that scale gracefully with compute will prevail over human-designed filters.