Issue 1: The Bitter Fruit of Filtering — Mixing, Duplication, and Extraction Quality Under Data Constraints
Highlights of This Issue
This week's leads pull "data quality" back to the first-principles level, away from just cleaning pipelines. Document extraction quality is being systematically established as an independent data quality axis: one line of work turns scientific literature into a 900B token corpus via OCR and layout analysis, then validates it through controlled continued pretraining on 3B/7B base models trained from scratch (SciPedia). Another line treats OCR noise as a first-class citizen, with benchmarks specifically evaluating its impact on downstream RAG robustness (StruNRAG, When Good OCR Is Not Enough), and even makes OCR itself an open tool based on VLMs (olmOCR). These efforts converge on a judgment: as web corpora near depletion, tokens extracted from PDFs, scans, and rich-format documents are becoming the marginal supply, and 'extraction quality' itself is starting to determine whether these tokens are usable.
Another thread is the emergence of a unified modeling framework for mixing and duplication under 'data-constrained' conditions. Some works explicitly model the interaction between mixing ratios and total data budget, finding that the optimal ratio shifts as data becomes scarcer (Scaling Laws for Mixture Pretraining Under Data Constraints). Concurrently, evaluation criteria are shifting from base loss to 'post-training ceiling', with some attempts to apply reinforcement learning directly on pretraining data to improve downstream trainability (RL on Pre-Training Data). Together, these imply that data mixing decisions can no longer rely solely on validation loss but must also consider the headroom left for subsequent RL and post-training.
Overall, the consensus this week is that 'filtering and cleaning are neither free nor always beneficial' — but most evidence remains at the level of base loss and small-scale benchmarks, with validation from the post-training ceiling perspective still scarce. This carves out a clear gap for the next phase of data research.
Community and Updates
Several notable releases this week center on 'document extraction quality as a data quality axis'. SciPedia creates a 900B token scientific corpus from academic literature via OCR and layout analysis, then performs continued pretraining on 3B/7B base models trained from scratch for 600B tokens. They find that directly feeding raw literature yields almost no benefit (a 'learnability gap'), requiring content cleaning and instructional augmentation to achieve gains of +2.12/+2.95 — providing controlled evidence for the distinction between 'extraction quality' and 'content learnability'. olmOCR turns OCR itself into an open VLM-based tool, directly serving the corpus construction step of converting PDFs into clean, linearized text. Meanwhile, two works evaluate OCR noise as a first-class data quality axis for RAG (StruNRAG, When Good OCR Is Not Enough), shifting evaluation from 'OCR character accuracy' to 'downstream retrieval-augmented generation robustness'.
In the direction of mixing and post-training ceilings, Scaling Laws for Mixture Pretraining Under Data Constraints explicitly models the interaction between mixing ratios and total data budget, suggesting the optimal ratio shifts with data scarcity. RL on Pre-Training Data attempts to apply reinforcement learning directly on pretraining data, shifting the objective of data selection from base loss to RL trainability.
Open Questions
- To what extent do conclusions about 'filtering being useless' transfer to the post-training ceiling? Existing evidence is primarily based on base loss and small-scale benchmarks. If judged by pass@k or RL trainability, it remains unclear whether no-filtering or lenient filtering still holds an advantage.
- Where is the boundary between duplication harm and synthetic rewriting? Duplication is most harmful at intermediate repetition counts, while synthetic rewriting also essentially repeats information — how do they differ information-theoretically, and is there a 'sweet spot for rewriting ratio' that shifts with model scale?
- Does document extraction quality as a data axis also have a compute threshold for its marginal benefit? When compute is abundant, does extraction quality, like filtering, get 'washed out' by compute, or is structured information like layout and tables an irreplaceable input for post-training, especially RL and long-context models?
Papers in this issue
At sufficient compute scales, no data filtering outperforms all existing filters for pretraining large language models, with unfiltered Common Crawl becoming optimal around 1e30 FLOPs.
Editor's noteProvides systematic scaling evidence in the high-compute, data-scarce regime, arguing that 'the best filter is no filter': with sufficiently large models and training steps, training directly on raw Common Crawl can surpass heavily filtered DCLM-Baseline/RefinedWeb, and even shuffled word documents provide benefits. It advances the filtering question from 'threshold tightness' to 'whether any filter is superior' and provides a crossover prediction at ~1e30 FLOPs. Limitation: only evaluates base loss and small benchmarks, not post-training ceiling; however, it's enough to challenge the default assumption that 'more filtering is better'.
InfoLaw predicts LLM loss under data repetition and quality mixtures with under 1% error, enabling efficient data recipe selection without expensive hyperparameter sweeps.
Editor's noteProposes an 'information accumulation' scaling law that explicitly models quality-weighted mixing and duplication jointly. It extrapolates loss to unseen recipes and larger scales (252M–7B, up to 425B tokens) and releases the recipe search. Compared to Data Mixing Laws/RegMix/CMR, the increment is treating duplication as a term with diminishing returns with scale, rather than handling it separately from quality mixing. Limitation: evaluation metric remains validation loss, not post-training ceiling, but it is one of the few empirical frameworks that can guide both mixing and overtraining decisions.
Language model evaluation loss peaks at intermediate per-document repeat counts, wasting up to 33% of training compute, with the most damaging repeat count scaling as N to the power of negative 0.96.
Editor's noteModernizes duplication harm research using compute-equivalent units (CEG/CEL), finding that at a fixed 10% duplicated token ratio, harm peaks at intermediate repetition counts, and the peak location shifts with model scale according to a power law (R_peak ∝ N^-0.96). It provides a closed-form statistical analogy (misspecified linear regression with block-diagonal noise covariance) showing this effect is not unique to transformers. Limitation: small scale (max 344M) and single architecture/corpus, but the controlled iso-FLOP design and theoretical analogy make the conclusions robust enough to influence deduplication and mixing decisions.
The CD scaling law unifies compute- and data-optimal pretraining, showing derived-token effectiveness decays with model size and data availability, making classic Chinchilla allocation suboptimal in practice.
Editor's noteProposes a unified Compute-Data scaling law using a token effectiveness function η that varies with (N, TPP) and its saturation ceiling R* to bridge compute-optimal and data-optimal regimes. It can predict when 'duplication vs. rewriting' will dominate and when compute investment saturates. Relative to Muennighoff et al.'s data-constrained scaling laws, the increment is an explicit saturation function form rather than a constant assumption. Cross-scale validation (14M–600M) yields actionable conclusions (e.g., the 4-epoch rule holds only for ~3B models; rewriting fails for N≥7B); the main caveat is the maximum scale is below 1B.
SYNPRO's model-aware synthetic data generation, grounded in organic text via RL-optimized rephrasing and reformatting, matches or exceeds training on unique data in the data-bound scaling regime.
Editor's noteProposes a model-aware synthetic data generation framework that adds a 'reformat' operation and data influence reward on top of rewriting, steering generation towards content not yet absorbed by the current model, and iteratively updates the generator during pretraining plateaus. At 400M and 1.1B scales, faithful synthesis can unlock effective tokens equivalent to 3.7–5.2x repetition, even surpassing the unique data oracle. The cost is ~2.6–5x compute compared to repetition, but this trade-off is worthwhile in data-constrained scenarios; recipes and ablations are public.
PureDocBench, a source-rendered benchmark with triple-track degradation, reveals document parsing is far from solved, with formula recognition as the field's shared bottleneck and clean-only evaluation misleading real-world deployment choices.
Editor's noteProposes a source-traceable, contamination-resistant document parsing benchmark, rendering 1,475 pages × 3 tracks (clean/digital degradation/real degradation) for a total of 4,425 images from fresh HTML/CSS, covering 66 subcategories across 10 domains, with annotations generated from the same source for verifiability. Evaluation of 40 models shows general VLMs are more robust to degradation than pipeline-specific models. It directly addresses OmniDocBench's annotation errors, score saturation, narrow coverage, and contamination risks, serving as a reliable evaluation resource for the document extraction quality axis.
SEFD reconstructs the SEC EDGAR archive into layout-faithful MultiMarkdown, achieving over 99 percent structural accuracy and enabling token-efficient financial pretraining with less than 0.1 percent Common Crawl overlap.
Editor's noteReconstructs the entire SEC EDGAR archive (estimated 550B tokens, 152B public) into layout-preserving, token-efficient MultiMarkdown, explicitly preserving tables, indentation, and visual hierarchy that previous financial corpora like BeanCounter discarded. Its 'visual-first' parsing method reverse-engineers EDGAR's Three-Column Hack and fragmented headers, with <0.1% overlap with Common Crawl corpora. Two downstream benchmarks (EDGAR-OCR, EDGAR-Forecast) shift evaluation towards post-training usability; main caveat is the paper does not report actual pretraining results using SEFD.
Masked-Input Regularization improves data-constrained language model pretraining, and the SoftQ scaling law better predicts performance in this regime than existing models.
Editor's noteProposes the SoftQ scaling law that explicitly couples model scale and unique data volume, showing that additive Chinchilla-style laws are misspecified in the data-constrained regime. It also introduces a MIR masked input auxiliary loss, providing complementary regularization on top of strong weight decay, with benefits increasing with scale. The increment over Muennighoff/Kim et al. is the coupled form and isolating mask benefits from weight decay. Evidence up to 1.4B parameters directly informs scaling and budget decisions under data constraints.
