Pretraining Data

This direction focuses on the construction, mixing, and utilization of pretraining corpora for large language models, covering corpus quality and sourcing, the benefit limits of synthetic data, data-constrained reuse, and data selection across training stages. The emphasis is on evidence from real-scale experiments with ablations and controlled comparisons to inform data-mixing decisions that affect downstream training performance.