# CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data

> CURAWEB's unified curation framework, integrating domain-specific cleaning, soft semantic deduplication, and value-aware Power Sampling, yields a 2T-token corpus that boosts 3B-parameter LLM performance by 1.82% over DCLM, with gains persisting at 1T-token scale.

- **Source:** [arXiv](https://arxiv.org/abs/2607.22662)
- **Published:** 2026-08-29
- **Permalink:** https://picx.dev/p/cleNm1
- **Whiteboard:** https://picx.dev/p/cleNm1/image

## Summary

# Summary of CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data

## Summary (Overview)

- **Novel curation paradigm**: CuraWeb shifts from traditional linear, stage-wise filtering (cleaning → deduplication → sampling) to a unified, multi-dimensional signal-driven framework that jointly optimizes quality, redundancy, and diversity across the entire data curation pipeline.

- **Key technical innovations**: (1) Domain-specific heuristic thresholds with priority bypass for STEM content, (2) Hybrid deduplication combining n-gram (MinHash-LSH) with soft semantic deduplication using a weighted voting mechanism, and (3) A multi-objective Power Sampling strategy that jointly optimizes content value and diversity.

- **Dataset scale**: Constructs CURAWEB, a 2T-token English corpus from 12 years of Common Crawl snapshots (2013–2024), compressing 184.9B raw documents to a final curated subset through a multi-stage funnel.

- **Empirical results**: At 3B parameter scale with 200B training tokens, CURAWEB achieves 48.07% average score across 10 benchmarks, outperforming DCLM (46.25%) by 1.82% absolute points, with the advantage maintained (1.95%) when scaling to 1T tokens.

- **Balanced performance**: Unlike specialized baselines that trade off capabilities across domains, CURAWEB ranks best or second-best on nearly all benchmarks, demonstrating superior knowledge-intensive and reasoning performance without sacrificing general capabilities.

## Introduction and Theoretical Foundation

### Background and Motivation

The quality and diversity of web-scale corpora (e.g., Common Crawl) are recognized as core variables determining the upper bound of LLM emergent capabilities and reasoning performance. However, prevailing data curation frameworks (FineWeb-Edu, DCLM) rely on **singular optimization approaches**, deploying monolithic quality classifiers that inevitably narrow distributional diversity and marginalize long-tail knowledge.

### Key Limitations of Existing Pipelines

1. **Structural isolation**: Cleaning, deduplication, and sampling are treated as disjoint, sequential stages, each driven by the same uni-dimensional quality signal, causing distributional bias to compound across the pipeline.

2. **Uniform thresholds**: Rule-based filters apply identical constraints across all domains, inadvertently discarding high-value content in specialized fields (mathematics, science, code).

3. **Hard-threshold deduplication**: Fixed similarity thresholds cause over-filtering of domain-specific documents that naturally exhibit dense terminological overlap.

4. **Quality-diversity trade-off**: Quality and diversity are treated as disjoint objectives optimized in isolation, leaving the inherent trade-off systematically unresolved.

### Theoretical Foundation

The framework is built on the principle that **multi-dimensional signal-driven global governance**—rather than disjointed, stage-wise filtering—enables well-calibrated balance between data quality, domain coverage, and token diversity. This requires:

- Fine-grained annotation of all data with multi-aspect quality scores and domain tags
- These multi-dimensional attributes serving as core dependencies across the entire pipeline
- Consistent funnel monitoring and strategy optimization

## Methodology

### 1. Text Extraction

- **Source**: 12 years of Common Crawl snapshots (2013–2024)
- **Extraction**: Resiliparse for main-body text extraction
- **Language filtering**: FastText classifier (English, confidence > 0.5)
- **Exact deduplication**: SHA-256 document hashing (184.9B → 41.4B documents, 22.4% retention)
- **URL filtering**: Extended blacklist combining public lists and custom rules (→ 39.1B documents)

### 2. Data Understanding System

A lightweight **300M-parameter multi-task Transformer** with two linear heads performs quality assessment and domain classification in a single forward pass:

- **Quality metrics**: Nine fine-grained dimensions—writing_score, coherence_score, completeness_score, logicality_score, safety_score, knowledge_score, education_score, helpfulness_score, reasoning_score. Pearson correlation analysis confirms low collinearity (feature independence).

- **Domain taxonomies**: Two-level hierarchy adapted from Google AdSense classification—26 coarse-grained themes and 105 fine-grained categories.

- **Training**: GPT-4o teacher model generates supervision labels for 500K stratified-sampled documents; knowledge distillation trains the student model with uncertainty weighting [41] to balance multi-task gradients.

### 3. Two-Stage Data Filtering

**Rule-based filtering** (Table 1): Removed 5 low-precision rules (e.g., median_word_length, fraction_bullet_points) and recalibrated 3 thresholds to protect STEM content:

| Rule Name | Original | Optimized | Motivation |
|-----------|----------|-----------|------------|
| word_count | ≤ 50 | ≤ 25 | Relaxes word limit to retain concise knowledge snippets |
| symbol_to_word_ratio | > 0.1 | > 0.38 | Tolerates symbol-dense text to protect formulaic content |
| alphabetic_char_ratio | < 0.8 | < 0.26 | Relaxes alphabetic constraint to prevent STEM data loss |

**Priority bypass**: Documents tagged as Mathematics, STEM, or Code skip heuristic filtering. This stage retains 87.3% of documents (34.2B) vs. only 32.6% with traditional uniform rules.

**Model-based filtering**: Conservative intersection mechanism—documents failing any of four dimensions (writing, coherence, completeness, safety) are eliminated, reducing corpus from 34.2B to 23.2B documents.

### 4. Hybrid Deduplication

**Stage 1—N-gram fuzzy deduplication**: MinHash-LSH removes near-duplicates with high surface-text overlap.

**Stage 2—Semantic deduplication**: Identifies two hidden redundancy types:
- **Templatized Clusters**: Same structural skeleton, different localized parameters (e.g., tax calculators, salary tools)—up to 99.96% intra-cluster duplication
- **Redundant Crawling**: Same page crawled at different times, falling below LSH threshold

**Soft deduplication algorithm**: Weighted voting mechanism accumulates penalty scores across multiple similarity edges:

$$D_{total} = \sum_{k} w_k \cdot \mathbb{1}[\text{edge in interval } k]$$

A document is classified as redundant only if the aggregated score exceeds a Target Score ($T_{target} = 10$). Interval weights are empirically determined (e.g., 8.0 for [0.99, 1.00], 7.0 for [0.98, 0.99)).

**Embedding model selection**: embeddinggemma-300m [44] selected after benchmarking against jina-v3, Qwen3-0.6B, and Qwen3-4B for linear monotonicity and discrimination accuracy.

### 5. Data Sampling

**Diversity score** (K-Means clustering, K ≈ 49,000):

$$d_i = d_{\text{intra},j} \times d_{\text{inter},j} \tag{1}$$

**Content value score**: Linear summation of knowledge_score, education_score, helpfulness_score, reasoning_score. Documents with $q_i \leq 1$ are filtered out.

**Unified sampling weight**:

$$s_i = \alpha \cdot q_i + (1 - \alpha) \cdot d_i \tag{2}$$

with $\alpha = 0.6$.

**Power Sampling** (replacing Softmax in SampleMix):

$$p_i = \frac{\tilde{x}_i^n \cdot \sigma(k(\tilde{x}_i - a))}{\sum_j \tilde{x}_j^n \cdot \sigma(k(\tilde{x}_j - a))}\tag{3}$$

where $\tilde{x}_i \in [0,1]$ is min-max normalized score, $n$ is power exponent, $\sigma(\cdot)$ is sigmoid soft threshold gate, $k$ controls gate steepness, $a$ is gate center.

## Empirical Validation / Results

### Main Results (200B tokens, 3B parameters)

| Dataset | GSM8K | MathQA | MMLU | MMLU-Pro | RACE | HellaSwag | PIQA | SCIQ | Wino. | OBQA | Avg. |
|---------|-------|--------|------|----------|------|-----------|------|------|-------|------|------|
| DCLM | 3.72 | 24.02 | 39.15 | 11.11 | 36.94 | 71.63 | 77.37 | 90.30 | 66.46 | 41.80 | 46.25 |
| FineWeb_Edu | 3.64 | 25.53 | 28.02 | 9.20 | 35.02 | 67.16 | 76.28 | 90.20 | 63.30 | 44.20 | 44.26 |
| Nemotron_CC | 2.56 | 24.96 | 36.25 | 10.04 | 36.75 | 72.76 | 78.13 | 92.10 | 63.46 | 40.30 | 45.73 |
| Dolma3 | 3.80 | 24.92 | 40.56 | 12.08 | 36.65 | 66.94 | 75.84 | 88.70 | 62.90 | 40.60 | 45.30 |
| **CuraWeb** | **8.19** | **26.00** | **47.16** | **14.90** | 36.84 | 71.09 | 77.86 | **90.90** | 63.85 | **43.90** | **48.07** |

### Key Findings

- **Reasoning advantage**: +4.39% on GSM8K over best baseline (Dolma3); +6.61% on MMLU; +2.82% on MMLU-Pro
- **Balanced performance**: Mitigates domain bias seen in specialized baselines (e.g., FineWeb-Edu's 44.20% on OBQA but only 28.02% on MMLU)
- **STEM recovery**: Consistent improvements on MathQA (+0.47%) and SCIQ (+0.70%) vs. FineWeb-Edu

### Scaling Behavior

CURAWEB maintains a stable 1.95% performance margin over DCLM when scaling from 200B to 1T tokens, demonstrating sustained training efficacy without early saturation.

### Ablation Studies

- **Model-based filtering**: Doubles retained high-quality data volume without compromising training efficiency
- **Semantic deduplication**: Removal shifts training curve downward, confirming templated redundancy causes substantial performance loss
- **Power Sampling**: Replacement with uniform sampling causes visible performance drop

### Deduplication Effectiveness

| Similarity Interval | Weight $w_k$ | Baseline FPR | Ours (Multi-path) FPR |
|---------------------|-------------|--------------|----------------------|
| [0.99, 1.00] | 8.0 | 37.5% | 28.03% |
| [0.98, 0.99) | 7.0 | 34.8% | 21.89% |
| [0.96, 0.98) | 6.0 | 42.2% | 23.13% |
| [0.95, 0.96) | 5.0 | 54.5% | 29.70% |
| [0.94, 0.95) | 4.0 | 65.8% | 35.49% |
| [0.92, 0.94) | 3.0 | 71.6% | 34.45% |
| [0.90, 0.92) | 2.0 | 85.0% | 44.37% |

### Diversity Metrics

**Domain entropy**: CURAWEB achieves $H_{L1} = 4.295$ bits and $H_{L2} = 6.075$ bits, substantially higher than FineWeb-Edu (3.733/5.477) and Dolma3 (4.086/6.034), comparable to DCLM (4.308/6.122).

**Lexical diversity**: Unique n-gram count of 5.13M (highest among compared baselines except Dolma3), with cumulative deduplication rate of 72.4%.

### Quality Analysis

- **STEM retention**: CURAWEB achieves average retention rate of 0.432 for high-quality STEM documents, outperforming DCLM (0.364), Dolma3 (0.376), Nemotron-CC (0.261)
- **Value scores**: Best in Knowledge (1.164) and PracticalHelpfulness (0.751) among all external datasets
- **Sampling-driven optimization**: Upsampled domains concentrate in knowledge-intensive categories (Education 6.53%, Health 5.18%, Legal 3.68%, Programming 2.59%)

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Paradigm shift**: Demonstrates that joint optimization across quality, redundancy, and diversity outperforms singular-objective filtering, challenging the dominant monolithic-classifier approach in data curation.

2. **Soft decision boundaries**: The weighted voting mechanism for semantic deduplication provides theoretical grounding for how multi-path evidence aggregation can bound false positive rates compared to single-edge decisions.

3. **Quality-diversity reconciliation**: Shows that quality and diversity are not necessarily opposing objectives—when properly integrated via multi-dimensional signals, they can be jointly optimized to improve both token efficiency and knowledge coverage.

### Practical Implications

1. **Industrial-grade standards**: Establishes a reference pipeline for large-scale data curation with continuous monitoring and strategy optimization based on multi-dimensional attributes.

2. **STEM data preservation**: Domain-specific thresholds and priority bypass mechanisms provide a template for protecting high-value technical content that traditional pipelines systematically discard.

3. **Scalable annotation**: The lightweight 300M-parameter multi-task model with knowledge distillation demonstrates that fine-grained annotation of billions of documents is computationally feasible.

4. **Token efficiency**: At 1T-token scale, CURAWEB's sustained performance advantage validates that precise data selection yields better returns than simply accumulating more data.

## Conclusion

CURAWEB introduces a unified data curation framework that shifts from disjointed, stage-wise filtering to multi-dimensional signal-driven global governance. Three key innovations—domain-specific heuristic cleaning, soft semantic deduplication with weighted voting, and content value-centric Power Sampling—enable the construction of a 2T-token corpus with enhanced diversity, minimal redundancy, and broader long-tail knowledge coverage.

At 3B scale, models trained on CURAWEB consistently outperform strong baselines (FineWeb-Edu, Dolma3, DCLM), particularly on knowledge-intensive and reasoning tasks, with the advantage maintained through 1T-token scaling. The work provides strong empirical evidence that meticulous data refinement and distribution reshaping are critical paths for advancing LLM intelligence, and serves as a practical guide for future open-source data curation efforts.

**Future directions** include applying the framework to multilingual corpora, extending the multi-dimensional signal system to additional quality dimensions, and further optimizing the trade-off between sampling-induced diversity reduction and quality enhancement.

---

_Markdown view of https://picx.dev/p/cleNm1, served by PicX — AI-generated visual whiteboard summaries of research papers._