Pretraining DataIssue 7Sep 19 – 26, 2026

Data Selection Begins to Use RL Coverage as Criterion

Highlights of This Issue

The clearest signal this issue is that the "post-training upper bound" criterion has moved from diagnosis to actionable intervention. TailSFT shifts the goal of data filtering in the SFT stage from "improving single-sample accuracy" to "preserving coverage for subsequent RL": using the initial policy as a reference, it filters already-fitted sequences based on loss reduction, and provides a coverage ratio diagnostic to predict when filtering will be effective. At the 7B scale, TailSFT checkpoints actually have lower pass@1 before GRPO, but after GRPO, pass@1 is higher in every comparison group (up to +3.9%), directly echoing the conclusion from Issue 5, "Good Pretraining, Bad SFT," that "optimal base loss does not equal optimal post-training," and advancing it into an actionable filtering intervention (TailSFT).

Data mixing continues to converge toward "low-cost dynamics" this issue. TiKMiX observes that the ranking of data domain influence remains stable over long training periods, and accordingly models mixing as a periodic, low-overhead influence maximization problem, achieving comparable or better downstream performance with about 20% of RegMix's computational cost (TiKMiX). This aligns with the trend in Issue 6 where FastMix/ByDoRe compressed mixing search to single-agent or near-zero cost, but TiKMiX's increment is the explicit use of the empirical regularity of "influence ranking invariance" to avoid the overhead of online dynamic updates, with experiments scaled to 1T tokens.

Multilingual data mixing yields a counterintuitive sweet spot conclusion. Building Multilingual Bridges systematically studies the impact of data composition and scheduling on the generalization of "reasoning in the prompt language" (L2 reasoning) during the SFT stage at the 3.35B scale, finding that multilingual non-reasoning data is a cheap lever for cross-lingual transfer, with a sweet spot of 20–30% for the proportion of non-reasoning data, and that data mixing outperforms sequential adaptation and weight merging (Building Multilingual Bridges). It treats "reasoning language" itself as a trainable behavior, providing a new dimension for synthetic data benefit boundaries and ratio decisions.

Community and Updates

Infinity-Parser released a layout-aware RL framework for document parsing and a high-quality 400K-scale dataset, Infinity-Doc-400K, replacing weak supervision pipelines with verifiable multi-faceted rewards (edit distance, paragraph count, reading order), achieving new SOTA on OmniDocBench, olmOCR-Bench, PubTabNet, and FinTabNet, explicitly treating document extraction quality as a data axis (Infinity-Parser). This complements the residual diagnostics of WeVisDoc from Issue 6, advancing document parsing to RL trainability criteria.

Two commentary articles are worth noting. One questions the unmeasured impact of aggressive quality filtering recipes (e.g., FineWeb-Edu cutting 92% of tokens) on informal/conversational/creative abilities, and points out the risk of correlated errors in synthetic bucketing (Why frontier models throw away 90% of their web data); the other, titled "No-filter pretraining outperforms curated data at scale," directly challenges the curation conclusions of DCLM/FineWeb, emphasizing the regime dependence of these conclusions and the lack of independent replication (No-filter pretraining outperforms curated data at scale). Both echo the skepticism of "more filtering is better" from Issue 1's "A Bitter Lesson for Data Filtering," but neither provides reproducible new empirical evidence, so they should be treated as hypotheses to be tested rather than conclusions.

Open Questions

  1. Can TailSFT's coverage ratio diagnostic be generalized to pretraining corpus composition? Does the criterion of "preserving coverage for RL" also apply to data selection in mid-training and pretraining stages, or is it only valid for SFT?
  2. To what scale and training phase does TiKMiX's "influence ranking invariance" hold? If the ranking drifts over longer training intervals or larger models, how should the periodic re-ranking interval for semi-dynamic mixing be set?
  3. Does the non-reasoning data sweet spot (20–30%) in Building Multilingual Bridges vary with model scale and language coverage? Can the conclusion that "reasoning language is trainable" transfer to corpus composition in pretraining, or is it limited to SFT?
  4. The debate between no-filter and aggressive filtering (Issue 1 and this issue's commentaries) still lacks controlled comparisons at realistic scales—can the hypothesis that "filtering benefits diminish with compute" be tested at ≥1B scale with post-training upper bound as the criterion?

Papers in this issue

  1. TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

    TailSFT filters already-fit SFT examples to improve response coverage, yielding up to 16.8% pass@16 gains and consistently better post-RL performance.

    Editor's note

    Shifts SFT data filtering from "improving single-sample accuracy" to "preserving coverage for subsequent RL": using the initial policy as a reference, it filters already-fitted sequences based on loss reduction, and provides a coverage ratio diagnostic to predict when filtering will be effective. Relative to Issue 5's "Good Pretraining, Bad SFT," which treated "optimal base loss does not equal optimal post-training" as a diagnostic conclusion, the increment here is advancing that criterion into an actionable SFT intervention, validated at the 7B scale with GRPO-matched controls: filtered checkpoints have lower pass@1 before RL but higher pass@1 in every comparison after RL (up to +3.9%), and 10 out of 11 settings with ρ16>1 have non-negative coverage gains. Limitations: does not address pretraining corpus composition itself, and filtering ratios are not systematically scanned.

  2. Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning

    Tiny Aya L2-Thinker achieves over 93% in-language reasoning across 60 languages at 3.35B scale, proving reasoning language is transferable without sacrificing task accuracy.

    Editor's note

    First systematic study of the impact of data mixing and scheduling in the SFT stage on the generalization of "reasoning in the prompt language," providing evidence at the 3.35B scale that multilingual non-reasoning data is a cheap lever for cross-lingual transfer: there is a sweet spot of 20–30% for the proportion of non-reasoning data (0%→10% raises reasoning rate on unseen languages in MGSM from 46% to 89%, and accuracy from 49% to 67%), and data mixing outperforms sequential adaptation and weight merging. Relative to Issue 6's MiST synthetic mid-training corpus design, the increment is treating "reasoning language" itself as a trainable behavior and providing a ratio sweet spot; public model weights and multilingual reasoning data are released. Limitations: scale is 3.35B and the criterion is post-SFT benchmarks.

  3. When Data Is Scarce: Scaling Sparse Language Models with Repeated Training

    Dynamic sparse training delays data saturation and achieves dense-equivalent loss with 8-10x fewer FLOPs, making moderate sparsity optimal when data is scarce.

    Editor's note

    First to explicitly couple dynamic sparse training (DST) with data-constrained repeated training, providing sparse-aware data-constrained scaling laws, finding that moderate sparsity (~50%) delays the repeated data saturation point from R_d*=4.4 to about 6.9, and giving a separation between loss-optimal (~50%) and compute-optimal (higher and increasing with data scale) sparsity. Relative to Issue 5's "Data Scarcity and Model Sparsity," which focused on mechanistic analysis of MoE sparsity and repetition harm, the increment is providing extrapolable scaling law functional forms and resource allocation prescriptions for DST; code is public with sensitivity analysis. Limitations: scale stops at 1.92B fitting/7.68B extrapolation, criterion is validation loss, and DST is not the default pretraining paradigm.

  4. Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training

    Data-DPO treats data value as target-model-dependent, using activation probing and DPO-style preference learning to select SFT subsets that outperform full-data training with only 5-15% of data.

    Editor's note

    Advances SFT data selection from static quality/diversity to "target model conditioning": constructs pairwise preferences between samples using the target model's own one-step update activation gain, and trains a lightweight reward model in data space with a DPO objective. Relative to Issue 4's "Beyond Solvability," which used learnability as a prior, the increment is explicitly converting "local training feedback from the target model" into a learnable preference signal, and providing direct evidence of proxy signal mismatch (using only target preferences gives ARP 92.33, below the random baseline of 95.30; fusing quality and diversity raises it to 102.63), directly addressing the "proxy→target extrapolation" concern in this direction. Limitations: only covers VLM SFT settings, criterion is post-SFT benchmarks, and code is not public.