Leverage for Repeated-Data Training Expands from Epochs to Batch Size and Training Recipe
Highlights of This Issue
The clearest signal this issue is that the optimization leverage for "repeated-data pretraining" has expanded from epochs to batch size and the training recipe itself. Scaling Repeated-Data Pretraining is the first to treat batch size as an independent optimization variable in repeated-data pretraining, discovering an internal optimum (U-shaped curve)—small-batch gradient noise is typically harmful in single-pass training but provides useful implicit regularization under repeated data; it also provides a repeated-aware recipe (Repack, weight tying, positive embedding decay) and a joint frontier under two resource constraints: GPU-time and data (Scaling Repeated-Data Pretraining). This directly challenges the common practice of maximizing data parallelism under a fixed deadline, advancing the repeated-value geometry framework from Issue 9, "What Is a Repeated Token Worth?", into actionable training recipe decisions.
Data mixture scaling laws have been advanced this issue into two new settings: multimodal and code. Scaling Laws for Multimodal Data Mixtures is the first to provide a unified mixture scaling law under a three-modal MoE, explicitly modeling directional cross-modal transfer, mixture-dependent capacity sharing, and repetition saturation, and finding that non-text modality repetition beyond 2 epochs yields diminishing returns, with audio being model-heavy (Scaling Laws for Multimodal Data Mixtures). On the code front, Scaling Laws for Code treats the mixture ratio itself as a scaling-law variable, showing that the optimal programming language token allocation shifts with model scale (Scaling Laws for Code).
On synthetic data, two new design axes emerge this issue. AFCS advances the driving unit of synthetic generation from "source text chunks" to "atomic facts"—first extracting source-supported atomic facts, validating them via source grounding, then using them as generation targets; under matched budgets of 7B/32B and 20B tokens, fact-level conditioning outperforms chunk/paragraph/entity levels (AFCS). Effective Synthetic Data Curation, on the other hand, proves that individual-level selection signals are blind to inter-group interactions, and that group-level signals avoid collapse on generative tasks and require stronger relational weights as the synthetic ratio increases (Effective Synthetic Data Curation). Meanwhile, Conditional Transfer provides controlled evidence of non-monotonic dose-response for synthetic data: 25% share yields gains, 100% share falls below the zero-synthetic baseline, and teachability (loss response to its own token budget) is inconsistent with downstream transfer ranking (Conditional Transfer).
Community and Developments
This issue introduces a previously unmeasured statistical dimension in document extraction quality. Counting Documents Is Not Counting Text recharacterizes web-PDF corpora using token weighting rather than document counts, revealing extreme length skew (Gini 0.807, 3.02% of documents account for 50% of tokens) that leads to drastically different corpus descriptions under the two units, and quantifies for the first time the unrecoverable loss from Common Crawl truncation: 1 MiB truncation affects 23% of documents but 63% of tokens, with two text-layer parsers recovering only 1.4–11.4% (Counting Documents Is Not Counting Text). This is a direct warning for mixture decisions relying on Common Crawl PDF corpora.
Progress has also been made on the theoretical boundaries of synthetic data collapse and data-side levers. Escaping Collapse provides a formal threshold (β>0) for the minimum proportion of real data required to avoid model collapse, advancing empirical observations into a provable minimal curation bound (Escaping Collapse); while Curating the Collapse offers a counter-warning—sample-level quality filtering itself may accelerate model collapse in tabular synthesis (Curating the Collapse). In data-constrained settings, Mix, Don't Tune provides comparative evidence at 1.43B scale that "data-side intervention of mixing auxiliary language outperforms hyperparameter tuning" (Mix, Don't Tune); on mid-training mixtures, MergeMix proposes using model merge weights as a proxy for mixture ratios, transferring across scales from 8B to 16B (MergeMix); Scaling data mixing advances mixture sources from topic taxonomies to behavior-derived sources, achieving 4x gains with nano-scale per-document proxy training (Scaling data mixing for language and DNA pretraining).
Open Questions
- To what extent do the U-shaped batch optimum and repeated-aware recipe of Scaling Repeated-Data Pretraining hold at larger scales? Under its 153M–1.5B scale and base loss criterion, can the implicit regularization of batch gradient noise be reproduced at ≥1B with downstream-capability-based criteria?
- Does the "non-text repetition beyond 2 epochs yields diminishing returns" finding of the multimodal mixture scaling law hold at larger scales and on real multimodal corpora? Can directional transfer coefficients and capacity dilution coefficients guide practical text-vision-audio mixture ratios?
- Whether the "atomic fact" granularity of AFCS outperforms entity/paragraph granularity depends on fact extraction quality—does the gain of fact-level conditioning saturate with corpus scale in longer mid-training and more tasks?
- To what extent does the conclusion that group-level signals outperform individual-level signals depend on gradient diversity diagnostics—how do the benefit boundaries and computational costs of group-level curation trade off as synthetic ratios and task types vary?
Papers in this issue
Small batches act as implicit regularizers in repeated-data pretraining, yielding a U-shaped loss curve with optima far below standard recipes, plus three recipe tweaks that further improve performance.
Editor's noteFirst to treat batch size as an independent optimization variable in repeated-data pretraining, discovering an internal optimum (U-shaped curve), where small-batch gradient noise provides useful implicit regularization under repeated data; also provides a repeated-aware recipe (Repack, weight tying, positive embedding decay) and a joint frontier under two resource constraints: GPU-time and data. Relative to Issue 9's "What Is a Repeated Token Worth?" repeated-value geometry framework, the increment is expanding optimization leverage from epochs to the training recipe itself, directly challenging the common practice of maximizing data parallelism under a fixed deadline. Limitation: scale up to 1.5B and base loss criterion, but controlled ablations and resource trade-off analyses suffice to change training decisions under data constraints.
A unified multimodal scaling law reveals audio favors parameters over data, vision and text mutually interfere, and non-text repetition saturates after roughly two epochs.
Editor's noteFirst to provide a unified data mixture scaling law for text-vision-audio three modalities under MoE architecture, explicitly modeling directional cross-modal transfer, mixture-dependent capacity sharing, and repetition saturation, and finding that non-text modality repetition beyond 2 epochs yields diminishing returns, with audio being model-heavy. Relative to Issue 3's "Scaling Laws for Mixture Pretraining Under Data Constraints", the increment is coupling repetition and mixture into the multimodal MoE setting, and providing directional transfer coefficients and capacity dilution coefficients. Controlled ablations at 457M–8.3B scale and 150B tokens align with preferences; limitation: criterion remains base loss and code not released.
AFCS makes fact selection explicit in synthetic pretraining data, achieving 85.1% source-fact coverage and outperforming WRAP and Nemotron-CC on factual QA benchmarks.
Editor's noteFirst to advance the driving unit of synthetic data generation from "source text chunks" to "atomic facts": first extracting source-supported atomic facts, validating them via source grounding, then using them as generation targets, moving fact selection from the generator's implicit preferences to the extraction and validation stages. Under matched budgets of 7B/32B and 20B tokens, fact-level conditioning outperforms chunk/paragraph/entity levels, and in 100B mid-training, replacing 5B general tokens improves 12-task average by +1.09. Relative to Issue 6's QVAC Genesis III student failure signals, the increment is explicitly targeting fine-grained fact coverage as the generation objective, providing a new design axis of "synthetic granularity" for synthetic data.
Loss-based data quality metrics mislead mixture selection: target-aligned sources dominate, with curated synthetic data optimal at a bounded 25% share, beating pure or adaptive loss-reduced mixtures.
Editor's noteFirst to explicitly separate the three signals of "diagnosticity", "teachability", and "transferability" in a controlled pretraining setting, proving that teachability (loss response to its own token budget) is inconsistent with downstream transfer ranking, and providing controlled evidence of non-monotonic dose-response for synthetic data (25% share yields gains, 100% share falls below the zero-synthetic baseline). Relative to Issue 9's Fisher-Guided and Issue 8's Everything in Moderation, the increment is making the "teachability vs. transfer" disconnect an explicit research object, serving as a strong warning for studies relying on loss decreases for mixture decisions. Limitation: small scale and single downstream (HumanEval), but controlled ablations and Ado comparisons make conclusions transferable.
Counting documents instead of tokens misrepresents web-PDF corpora by up to 11x, with 63% of text lost to truncation that recovers only 1-11% via standard parsers.
Editor's noteFirst to recharacterize web-PDF corpus statistics using token weighting rather than document counts, revealing extreme length skew (Gini 0.807, 3.02% of documents account for 50% of tokens) that leads to drastically different corpus descriptions under the two units, and quantifying for the first time the unrecoverable loss from Common Crawl truncation (1 MiB truncation affects 23% of documents but 63% of tokens). Relative to Issue 8's ReScraper and Issue 6's WeVisDoc extraction quality axes, the increment is treating "corpus statistical unit bias" as a previously unmeasured dimension, directly challenging the convention of reporting coverage/routing rates in document counts. Limitation: only a single crawl snapshot analyzed, no model training validation, but as a corpus statistics methodology contribution it can change how PDF corpora are reported and interpreted.
Group-level influence signals, not individual sample scores, are essential for curating synthetic LLM training data, with gains concentrated in generative tasks and growing as pipelines become more synthetic.
Editor's noteFirst to systematically prove that individual-level signals in synthetic data curation are blind to inter-group interactions, and propose group-level signals as a necessary complement, providing an upper bound on the gap between group-level oracle and individual proxy oracle (bounded by intra-group gradient diversity). In both pretraining and RLVR post-training settings, individual-level selection collapses on generative tasks, group-level is better, and relational weights need strengthening as synthetic ratio increases. Relative to Issue 9's LESSER gradient feature cost study, the increment is making "group-level interactions" an explicit research object in synthetic curation, and providing actionable gradient diversity diagnostics. Limitation: small scale (400M/1.5B), but controlled ablations and theoretical bounds make conclusions transferable.
