Issue 2: The Interval Where Repetition Has Negative Effects - New Criteria for Data-Constrained Mixing
Highlights of This Issue
The most noteworthy clue this week advances "repetition under data constraints" from "diminishing returns" to "the existence of a negative-effect interval." One work proposes a scaling law with an additive overfitting penalty, pointing out that after a data-dependent compute threshold, continuing to repeat is actually harmful and compute should be redirected to model capacity. It also reanalyzes and finds that the popular advice of "smaller models, more epochs" stems from a mis-specification of the Chinchilla baseline (Prescriptive Scaling Laws for Data Constrained Training). Complementarily, another work identifies a major cause of the common failure of small-scale mixing experiments - "repetition mismatch": high-quality datasets are small, and their repetition rate drifts as the training budget grows, causing proxy experiments to predict the wrong optimal mixing ratio; a subsampling method that preserves the repetition rate can reduce prediction error from 0.75 to 0.05 under a 1/16 token budget (Repetition Mismatch). Together, these imply that the repetition rate should be treated as a first-class variable in mixing optimization, not a side effect of data scarcity.
Data mixing scaling laws are also gaining theoretical foundations. One work extends quantized models and linear regression models to multi-domain settings, explaining why mixing laws hold through two mechanisms - "capacity competition" and "noise suppression" - and provides a functional form with fewer parameters that can extrapolate across scales (Explaining Data Mixing Scaling Laws). On the document extraction quality axis, this week introduces a variable never measured before - "notation" (how document layout is written into the corpus) - and provides deterministic corpus statistics like clean-window survival and the de-structured "pure frame" format (The Announcement Carries the Cue).
Data selection during the mid-training phase is also being refined. One work constructs the first open mid-training corpus for general tool use, demonstrating at 4B/8B scale that two types of synthetic trajectories - grounding and execution - are complementary (MidTool); another work organizes synthetic textbooks into book-level documents, isolating the independent gain of "document packaging" itself for mid-training through controlled comparisons (Beyond Rephrasing).
Community and Developments
Around the shift of "evaluation criteria from base loss to post-training ceiling," several directional advances emerged this week. One work proposes Learning-Zone Energy based on pass rate as a data selection signal for RL post-training, retaining only 40% of data per step to match the full baseline (Learning-Zone Energy); another work advances data selection in the annealing phase from perplexity/random to a curvature-aware criterion based on spectral geometry (DiReCT). Another analysis points out that the optimal pre-RL checkpoint should be selected by post-training ceiling (pass@64) rather than peak pass@1, and it appears earlier (Understanding SFT dynamics around the best pre-RL checkpoint). On the document extraction side, an industrial study decomposes the extraction of long scanned financial documents and finds that page-level retrieval, rather than OCR quality itself, is the key driver of downstream accuracy (A Multistage Extraction Pipeline for Long Scanned Financial Documents).
Open Questions
- Does the additive overfitting penalty hold at ≥1B scale and under post-training metrics? Evidence from Prescriptive Scaling Laws stops at 1B and uses validation loss as the criterion; if measured by pass@k or RL trainability, whether "repetition beyond the threshold is harmful" still prevails is undetermined.
- Can clean-window survival, as a deterministic statistic of notation, predict downstream performance (especially long-context and post-training)? The judgment that "the announcement is the real cue" currently lacks end-to-end validation.
- Can the subsampling correction for repetition mismatch be combined with existing scaling laws (e.g., InfoLaw, Prescriptive Scaling Laws) and maintain predictive power under the post-training ceiling perspective? This determines whether it can be upgraded from a diagnostic tool to a routine step in mixing decisions.
Papers in this issue
A new scaling law with an additive overfitting penalty reveals that beyond a compute threshold, repeating data becomes counterproductive, favoring larger models over more epochs.
Editor's noteProposes an additive overfitting penalty, modeling the benefits of repetition and the superlinear cost separately, and for the first time characterizes the interval where "repetition beyond a data-dependent threshold is harmful and compute should be redirected to model capacity" - a direct increment over Issue #1's "Bridging Compute- and Data-Optimal Pretraining" and Muennighoff et al.'s "diminishing returns" formulation. Its reanalysis shows that the popular advice of "smaller models, more epochs" stems from a mis-specification of the Chinchilla baseline, and provides cross-configuration prescriptions validated under both perplexity and downstream OLMES; the case where strong weight decay reduces the overfitting coefficient by about 70% explicitly links regularization strength to repetition tolerance. The limitation is that evidence stops at 1B and uses validation loss, but it is sufficient to change decisions on scale and budget allocation under data constraints.
Repetition mismatch, not scale, causes data-mixture extrapolation failures; repetition-controlled subsampling using 1/16 of tokens recovers near-optimal mixtures within 0.05 error at 757M scale.
Editor's noteIdentifies "repetition mismatch" as an independent confounder in small-scale mixing extrapolation: high-quality datasets are small, and their repetition rate drifts as the training budget grows, causing proxy experiments to predict the wrong optimal mixing ratio. Relative to InfoLaw and Internal Data Repetition in Issue #1, which treat repetition as a modeling term, this work's increment is to treat the repetition rate as a controllable variable; a subsampling method that preserves the repetition rate can reduce the mixing prediction error of a 757M model from 0.75 to 0.05 under a 1/16 token budget. The method is orthogonal to existing scaling laws, can be absorbed into any mixing pipeline, and is a low-cost, highly operable correction; the limitation is scale ≤757M and validation loss as the criterion.
A unified theory of data mixing scaling laws, grounded in capacity competition and noise reduction, predicts optimal mixtures and extrapolates across scales with fewer parameters than empirical baselines.
Editor's noteProvides the first theoretical derivation of data mixing scaling laws, extending quantized models and linear regression models to multi-domain settings, explaining why mixing laws hold through two mechanisms - "capacity competition" and "noise suppression" - and pointing out that the optimal mixing ratio deviates from target weights due to data-dependent noise terms. Relative to empirical laws like Additive Law/RegMix, its increment is a functional form with fewer parameters that can extrapolate across scales using only small-scale proxies, and the code is public. The limitation is validation at 200M–1B and base loss as the criterion, but the theoretical contribution is sufficient to become a new benchmark for mixing research.
Optimal data repetition in LLM pretraining is primarily determined by domain validation loss, increasing with model size under fixed tokens-per-parameter scaling.
Editor's noteAdvances repetition scaling laws from global to domain-level: under a fixed TPP, the optimal repetition count for scarce high-quality domain data rises mildly with model scale and is determined by domain validation loss (rather than unique data volume), thus supporting proxy-to-target extrapolation. Relative to the global repetition perspective in Issue #1's Internal Data Repetition and Bridging Compute- and Data-Optimal Pretraining, its increment is domain-level granularity and the operational criterion of "validation loss as a predictor," and it notes that conclusions on repetition are opposite under fixed data budget versus fixed TPP settings. Evidence comes from ByteDance Seed/Tsinghua at real scale; the abstract is thin and lacks ablation numbers, which is the main reservation.
Pre-training corpus markup is an unmeasured variable that affects model behavior: deleting structural announcements harms prediction, while swapping notation alone has zero effect.
Editor's noteEstablishes "notation" (how document layout is written into the corpus) as a training variable never measured before, defines the deterministic corpus statistic clean-window survival, and surveys it across 13 corpora (vision-converted PDF slices drop to 0.153, C4 is 0.889). The core finding is that the cue models truly rely on is the "announcement" (short lines marking boundaries) rather than the marker symbols themselves, and it provides a "pure frame" format that removes all structural cues and is reversibly side-loaded, directly targeting the long-context training phase. It advances the document extraction quality axis from "character/layout accuracy" to "encoding of structural cues," providing a new operational lever for corpus construction; downstream end-to-end validation is still lacking.
MidTool introduces the first open mid-training pipeline and 20.3B-token corpus for general tool use, nearly doubling downstream agentic task performance across benchmarks and model scales.
Editor's noteThe first open mid-training corpus pipeline for general tool use, combining raw web/PDF/code corpora with two types of synthetic trajectories (context-grounded augmentation and native agentic trajectory) into MidTool-Mix of 20.3B tokens, and demonstrating at Qwen3-4B/8B with a fixed post-training recipe that it improves downstream SFT/RL tool-use performance. Relative to general or SWE-specific mid-training corpora like Dolmino/daVinci-Dev, its increment is a specific focus on general tool capabilities, and it distinguishes the complementary contributions of grounding and execution synthetic branches through ablations. The limitation is no matched-budget sweep of synthetic ratios, but as pioneering evidence in this direction, it is worth reading.
Book-level organization of synthetic training data, not content alone, improves LLM mid-training by over one point across 28 benchmarks.
Editor's noteProposes "book-level organization" as an independent design axis for synthetic mid-training data beyond "rephrasing": using a retrievable pipeline to organize synthetic textbooks into complete documents with hierarchical tables of contents (686K books, 32B tokens), and isolating the gain of document packaging itself through controlled comparisons of Full/Split/RandomConcat/Rephrase (+1.02/+1.17). Relative to Issue #1's Generating Pretraining Tokens from Organic Data, which focuses on "what content to generate," this work's increment is "how to organize generated content," and it validates across two architectures. The scale (3B MoE/8B) is slightly below the preference threshold, but the controlled ablations make its conclusions transferable to mixing decisions.
