Summary of CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data

Summary (Overview)

• Novel curation paradigm: CuraWeb shifts from traditional linear, stage-wise filtering (cleaning → deduplication → sampling) to a unified, multi-dimensional signal-driven framework that jointly optimizes quality, redundancy, and diversity across the entire data curation pipeline.

• Key technical innovations: (1) Domain-specific heuristic thresholds with priority bypass for STEM content, (2) Hybrid deduplication combining n-gram (MinHash-LSH) with soft semantic deduplication using a weighted voting mechanism, and (3) A multi-objective Power Sampling strategy that jointly optimizes content value and diversity.

• Dataset scale: Constructs CURAWEB, a 2T-token English corpus from 12 years of Common Crawl snapshots (2013–2024), compressing 184.9B raw documents to a final curated subset through a multi-stage funnel.

• Empirical results: At 3B parameter scale with 200B training tokens, CURAWEB achieves 48.07% average score across 10 benchmarks, outperforming DCLM (46.25%) by 1.82% absolute points, with the advantage maintained (1.95%) when scaling to 1T tokens.

• Balanced performance: Unlike specialized baselines that trade off capabilities across domains, CURAWEB ranks best or second-best on nearly all benchmarks, demonstrating superior knowledge-intensive and reasoning performance without sacrificing general capabilities.

Introduction and Theoretical Foundation

Background and Motivation

The quality and diversity of web-scale corpora (e.g., Common Crawl) are recognized as core variables determining the upper bound of LLM emergent capabilities and reasoning performance. However, prevailing data curation frameworks (FineWeb-Edu, DCLM) rely on singular optimization approaches, deploying monolithic quality classifiers that inevitably narrow distributional diversity and marginalize long-tail knowledge.

Key Limitations of Existing Pipelines

1. Structural isolation: Cleaning, deduplication, and sampling are treated as disjoint, sequential stages, each driven by the same uni-dimensional quality signal, causing distributional bias to compound across the pipeline.

2. Uniform thresholds: Rule-based filters apply identical constraints across all domains, inadvertently discarding high-value content in specialized fields (mathematics, science, code).

3. Hard-threshold deduplication: Fixed similarity thresholds cause over-filtering of domain-specific documents that naturally exhibit dense terminological overlap.

4. Quality-diversity trade-off: Quality and diversity are treated as disjoint objectives optimized in isolation, leaving the inherent trade-off systematically unresolved.

Theoretical Foundation

The framework is built on the principle that multi-dimensional signal-driven global governance—rather than disjointed, stage-wise filtering—enables well-calibrated balance between data quality, domain coverage, and token diversity. This requires:

• Fine-grained annotation of all data with multi-aspect quality scores and domain tags
• These multi-dimensional attributes serving as core dependencies across the entire pipeline
• Consistent funnel monitoring and strategy optimization

Methodology

1. Text Extraction

• Source: 12 years of Common Crawl snapshots (2013–2024)
• Extraction: Resiliparse for main-body text extraction
• Language filtering: FastText classifier (English, confidence > 0.5)
• Exact deduplication: SHA-256 document hashing (184.9B → 41.4B documents, 22.4% retention)
• URL filtering: Extended blacklist combining public lists and custom rules (→ 39.1B documents)

2. Data Understanding System

A lightweight 300M-parameter multi-task Transformer with two linear heads performs quality assessment and domain classification in a single forward pass:

• Quality metrics: Nine fine-grained dimensions—writing_score, coherence_score, completeness_score, logicality_score, safety_score, knowledge_score, education_score, helpfulness_score, reasoning_score. Pearson correlation analysis confirms low collinearity (feature independence).

• Domain taxonomies: Two-level hierarchy adapted from Google AdSense classification—26 coarse-grained themes and 105 fine-grained categories.

• Training: GPT-4o teacher model generates supervision labels for 500K stratified-sampled documents; knowledge distillation trains the student model with uncertainty weighting [41] to balance multi-task gradients.

3. Two-Stage Data Filtering

Rule-based filtering (Table 1): Removed 5 low-precision rules (e.g., median_word_length, fraction_bullet_points) and recalibrated 3 thresholds to protect STEM content:

Priority bypass: Documents tagged as Mathematics, STEM, or Code skip heuristic filtering. This stage retains 87.3% of documents (34.2B) vs. only 32.6% with traditional uniform rules.

Model-based filtering: Conservative intersection mechanism—documents failing any of four dimensions (writing, coherence, completeness, safety) are eliminated, reducing corpus from 34.2B to 23.2B documents.

4. Hybrid Deduplication

Stage 1—N-gram fuzzy deduplication: MinHash-LSH removes near-duplicates with high surface-text overlap.

Stage 2—Semantic deduplication: Identifies two hidden redundancy types:

• Templatized Clusters: Same structural skeleton, different localized parameters (e.g., tax calculators, salary tools)—up to 99.96% intra-cluster duplication
• Redundant Crawling: Same page crawled at different times, falling below LSH threshold

Soft deduplication algorithm: Weighted voting mechanism accumulates penalty scores across multiple similarity edges:

$$D_{total} = \sum_{k} w_k \cdot \mathbb{1}[\text{edge in interval } k]$$

A document is classified as redundant only if the aggregated score exceeds a Target Score ($T_{target} = 10$). Interval weights are empirically determined (e.g., 8.0 for [0.99, 1.00], 7.0 for [0.98, 0.99)).

Embedding model selection: embeddinggemma-300m [44] selected after benchmarking against jina-v3, Qwen3-0.6B, and Qwen3-4B for linear monotonicity and discrimination accuracy.

5. Data Sampling

Diversity score (K-Means clustering, K ≈ 49,000):

$$d_i = d_{\text{intra},j} \times d_{\text{inter},j} \tag{1}$$

Content value score: Linear summation of knowledge_score, education_score, helpfulness_score, reasoning_score. Documents with $q_i \leq 1$ are filtered out.

Unified sampling weight:

$$s_i = \alpha \cdot q_i + (1 - \alpha) \cdot d_i \tag{2}$$

with $\alpha = 0.6$.

Power Sampling (replacing Softmax in SampleMix):

$$p_i = \frac{\tilde{x}_i^n \cdot \sigma(k(\tilde{x}_i - a))}{\sum_j \tilde{x}_j^n \cdot \sigma(k(\tilde{x}_j - a))}\tag{3}$$

where $\tilde{x}_i \in [0,1]$ is min-max normalized score, $n$ is power exponent, $\sigma(\cdot)$ is sigmoid soft threshold gate, $k$ controls gate steepness, $a$ is gate center.

Empirical Validation / Results

Main Results (200B tokens, 3B parameters)

Key Findings

• Reasoning advantage: +4.39% on GSM8K over best baseline (Dolma3); +6.61% on MMLU; +2.82% on MMLU-Pro
• Balanced performance: Mitigates domain bias seen in specialized baselines (e.g., FineWeb-Edu's 44.20% on OBQA but only 28.02% on MMLU)
• STEM recovery: Consistent improvements on MathQA (+0.47%) and SCIQ (+0.70%) vs. FineWeb-Edu

Scaling Behavior

CURAWEB maintains a stable 1.95% performance margin over DCLM when scaling from 200B to 1T tokens, demonstrating sustained training efficacy without early saturation.

Ablation Studies

• Model-based filtering: Doubles retained high-quality data volume without compromising training efficiency
• Semantic deduplication: Removal shifts training curve downward, confirming templated redundancy causes substantial performance loss
• Power Sampling: Replacement with uniform sampling causes visible performance drop

Deduplication Effectiveness

Diversity Metrics

Domain entropy: CURAWEB achieves $H_{L1} = 4.295$ bits and $H_{L2} = 6.075$ bits, substantially higher than FineWeb-Edu (3.733/5.477) and Dolma3 (4.086/6.034), comparable to DCLM (4.308/6.122).

Lexical diversity: Unique n-gram count of 5.13M (highest among compared baselines except Dolma3), with cumulative deduplication rate of 72.4%.

Quality Analysis

• STEM retention: CURAWEB achieves average retention rate of 0.432 for high-quality STEM documents, outperforming DCLM (0.364), Dolma3 (0.376), Nemotron-CC (0.261)
• Value scores: Best in Knowledge (1.164) and PracticalHelpfulness (0.751) among all external datasets
• Sampling-driven optimization: Upsampled domains concentrate in knowledge-intensive categories (Education 6.53%, Health 5.18%, Legal 3.68%, Programming 2.59%)

Theoretical and Practical Implications

Theoretical Contributions

1. Paradigm shift: Demonstrates that joint optimization across quality, redundancy, and diversity outperforms singular-objective filtering, challenging the dominant monolithic-classifier approach in data curation.

2. Soft decision boundaries: The weighted voting mechanism for semantic deduplication provides theoretical grounding for how multi-path evidence aggregation can bound false positive rates compared to single-edge decisions.

3. Quality-diversity reconciliation: Shows that quality and diversity are not necessarily opposing objectives—when properly integrated via multi-dimensional signals, they can be jointly optimized to improve both token efficiency and knowledge coverage.

Practical Implications

1. Industrial-grade standards: Establishes a reference pipeline for large-scale data curation with continuous monitoring and strategy optimization based on multi-dimensional attributes.

2. STEM data preservation: Domain-specific thresholds and priority bypass mechanisms provide a template for protecting high-value technical content that traditional pipelines systematically discard.

3. Scalable annotation: The lightweight 300M-parameter multi-task model with knowledge distillation demonstrates that fine-grained annotation of billions of documents is computationally feasible.

4. Token efficiency: At 1T-token scale, CURAWEB's sustained performance advantage validates that precise data selection yields better returns than simply accumulating more data.

Conclusion

CURAWEB introduces a unified data curation framework that shifts from disjointed, stage-wise filtering to multi-dimensional signal-driven global governance. Three key innovations—domain-specific heuristic cleaning, soft semantic deduplication with weighted voting, and content value-centric Power Sampling—enable the construction of a 2T-token corpus with enhanced diversity, minimal redundancy, and broader long-tail knowledge coverage.

At 3B scale, models trained on CURAWEB consistently outperform strong baselines (FineWeb-Edu, Dolma3, DCLM), particularly on knowledge-intensive and reasoning tasks, with the advantage maintained through 1T-token scaling. The work provides strong empirical evidence that meticulous data refinement and distribution reshaping are critical paths for advancing LLM intelligence, and serves as a practical guide for future open-source data curation efforts.

Future directions include applying the framework to multilingual corpora, extending the multi-dimensional signal system to additional quality dimensions, and further optimizing the trade-off between sampling-induced diversity reduction and quality enhancement.