# A Bitter Lesson for Data Filtering

> At sufficient compute scales, no data filtering outperforms all existing filters for pretraining large language models, with unfiltered Common Crawl becoming optimal around 1e30 FLOPs.

- **Source:** [arXiv](https://arxiv.org/abs/2605.19407)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/vB2BqW
- **Whiteboard:** https://picx.dev/p/vB2BqW/image

## Summary

## Summary (Overview)

- **Core finding**: With sufficient compute, the best data filter for large model pretraining is **no filter at all**—models trained on unfiltered Common Crawl (CC) outperform those trained on heavily filtered subsets.
- **Key mechanism**: Sufficiently large models are highly robust to low-quality data; they can absorb "junk" data without performance degradation and even benefit from nominally "poor" data.
- **Scaling prediction**: The 240-trillion-token CC pool may become optimal over filtered versions at approximately $1 \times 10^{30}$ FLOPs, a compute level predicted to be reachable by ~2030.
- **Robustness to junk data**: Models trained on data augmented with randomly generated strings or shuffled-word documents close the performance gap with large enough model size and training steps.
- **Theoretical support**: A low-rank matrix factorization model demonstrates that sufficient model capacity allows noise to be absorbed without penalty, mirroring experimental observations.

---

## Introduction and Theoretical Foundation

The paper challenges the prevailing assumption that data filtering is essential for large-scale language model pretraining. The standard approach filters text from sources like Common Crawl (CC), keeping only "high-quality" data—for example, the DCLM-Baseline dataset retains only ~1% of original CC (~3.8T tokens). However, this conflicts with scaling trends that prescribe ever-increasing data amounts.

The authors formalize the goal as finding the best achievable performance:

$$
\mathcal{L}^{\star}(D) := \min_{M, N} \ell(\mathcal{A}(D, M, N))
$$

where $M$ is parameter count, $N$ is training steps, and $\mathcal{A}$ is the training algorithm. The key hypothesis tested is whether $\mathcal{L}^{\star}(D_{cc,m}) < \mathcal{L}^{\star}(f(D_{cc,m}))$ holds, where $f$ is a filtering function and $D_{cc,m}$ is an $m$-token subset of CC.

**Related theoretical work**: Cheng et al. [2024] developed theoretical models suggesting that with sufficient model fidelity, not cleaning data is better—a prediction this paper empirically confirms.

---

## Methodology

### Experimental Setup
- **Data**: DCLM-Pool (all CC before 2023, 240T GPT-NeoX tokens), with subsets ranging from 670M to 10B tokens.
- **Models**: Llama-style dense transformers from 15M to 7B parameters, trained with Meta Lingua.
- **Filters tested**: English filter (28.2% retained), repetition filter (45.3%), stop word filter (50.4%), RefinedWeb (13%), DCLM-Baseline (2.1%).

### Evaluation
- **Primary metrics**: Negative log-likelihood on C4, Fineweb-Edu, and Cosmopedia (averaged).
- **Secondary metrics**: ARC-Easy and PIQA benchmarks.
- **Junk data injection**: Randomly generated strings (10,000-word vocabulary of 3–8 character strings) and shuffled-word documents (word order randomized in additional CC documents).

### Scaling Law Construction
Two scaling law approaches were used to predict compute thresholds:
1. **Token-to-parameter ratio**: Fixed at 600:1 (following DeepSeek V4).
2. **Epoch constraint**: Fixed at 4 epochs (based on Muennighoff et al. [2025]).

---

## Empirical Validation / Results

### Data Filtering Experiments
- On a 670M-token pool, the **unfiltered pool outperforms all five filters** for models ≥330M parameters after sufficient training steps (Figure 1).
- The pool transitions from worst to best on the compute-performance Pareto frontier as compute increases (Figure 2).
- Crossing points appear earlier as model size increases.

### Data Injection Robustness
- **Random strings**: Models close the performance gap with sufficient scale, with some gains (+20% random) possibly due to regularization effects.
- **Shuffled words**: Surprisingly, models ≥330M parameters *benefit* from shuffled-word documents after ~11B training tokens, despite only the unigram distribution remaining intact.
- Performance does not degrade to random levels ($-\log(1/V) \approx 10.8$ with the tokenizer used).

### Pool Size Scaling
- The training steps needed for the pool to beat RefinedWeb ($N^{\star}(M,m)$) grow **super-linearly** with pool size (Table 1 in paper):
  - 670M pool: ~1 epoch needed
  - 2B pool: ~3 epochs
  - 10B pool: ~10 epochs
- Larger models dramatically reduce the epoch count needed for the pool to win.

### Compute Scaling Law Prediction

| Scaling Law Type | Predicted Compute Threshold |
|---|---|
| Token-per-parameter (600:1) | ~$1 \times 10^{30}$ FLOPs |
| Epoch constraint (4 epochs) | ~$1 \times 10^{30}$ FLOPs |

Both scaling laws give $R^2 > 0.99$ and predict similar budgets (~$1 \times 10^{30}$ FLOPs), compared to current frontier compute near $5 \times 10^{26}$ FLOPs.

### Factuality Analysis (Table 1 in paper)

Average GPT5-mini judgements on keyword-matched CC data for MMLU categories:

| Dataset | Support | Refute | Related | Unrelated |
|---|---|---|---|---|
| MMLU/world_religions | 5.89 | 0.00 | 13.22 | 7.50 |
| MMLU/astronomy | 2.03 | 0.14 | 10.14 | 17.41 |
| MMLU/college_biology | 2.67 | 0.17 | 11.07 | 13.40 |
| MMLU/medical_genetics | 2.80 | 0.23 | 14.30 | 11.23 |

Supporting documents are at least an order of magnitude more common than refuting ones, suggesting harmful content is relatively rare in CC.

---

## Theoretical and Practical Implications

### Theoretical Model (Proposition 7.1)

The paper presents a low-rank matrix factorization model where $k$ tasks generate data with orthogonal inputs. For any model rank $r$:

$$
\min_{\substack{U\in \mathbb{R}^{m\times r}\\ V\in \mathbb{R}^{d\times r}}}\mathbb{E}\big[ \| Y - UV^{\top}X\|^{2}\big] = \sum_{j = r + 1}^{\rho}\sigma_{j}^{2} + \mathbb{E}\big[ \| \xi \|^{2}\big]
$$

where $\sigma_j$ are singular values of $M_{\star}\Sigma^{1/2}$. This shows that:
- With enough capacity ($r \geq \rho$), noise is absorbed without penalty.
- Without enough capacity, performance necessarily degrades due to task interference.
- Gradient-based methods converge to optimal solutions (no non-strict saddle points when $r \geq k$).

### Practical Implications
1. **Data filtering may suffer from the "bitter lesson"**: Human-designed filters that work at small scale may be replaced by simple no-filter approaches that scale more gracefully with compute.
2. **Compute thresholds matter**: Filtering remains important when compute is constrained; the benefits of no-filtering only manifest at very high compute levels.
3. **Distribution shift robustness**: Language models are highly resistant to covariate shifts; it is "incorrectly labeled" data or conditional distribution shifts that are detrimental.

---

## Conclusion

The paper provides strong evidence that data filtering may become unnecessary at sufficient compute scales. Key takeaways:

1. **No filter is best with enough compute**: Sufficiently trained large models benefit from nominally "poor" data in Common Crawl.
2. **Predictable scaling**: The compute threshold for no-filter optimality follows clean scaling laws ($R^2 > 0.99$), predicting ~$1 \times 10^{30}$ FLOPs for the full CC pool.
3. **Robustness is capacity-dependent**: Large models can extract signal even from shuffled-word data; small models cannot.

### Future Directions
- **Deviations from vanilla pretraining**: MoE architectures, synthetic data, and post-training may alter conclusions.
- **Duplicate documents**: Larger pools have more duplicates, though epoching mitigates this.
- **AI-generated content**: Increasing AI content in CC may change the calculus.
- **Factuality edge cases**: Rare instances of harmful content may still warrant targeted filtering.

The authors conclude that while it is possible to construct harmful data (e.g., non-factual content that looks like high-quality data), such content is not prevalent in CC, suggesting that **data filtering may eventually follow the bitter lesson**: simple, no-filter approaches that scale gracefully with compute will prevail over human-designed filters.

---

_Markdown view of https://picx.dev/p/vB2BqW, served by PicX — AI-generated visual whiteboard summaries of research papers._