Issue 3: Repetition Tolerance under Mixed Ratios and Post-Training Ceiling
Highlights of This Issue
Under data constraints, a core conclusion that may rewrite mixing ratio decisions emerged this week: mixed training has a much higher tolerance for repetition than single-source training. Some work explicitly couples target domain weight with repetition factor into scaling laws, finding that general-purpose data acts as a regularizer, allowing the target domain to be repeated 15–20 times without performance degradation—far exceeding the common "<4 times" rule of thumb in single-source training (Scaling Laws for Mixture Pretraining Under Data Constraints). This creates tension with last issue's conclusion that "repetition beyond a threshold is harmful"—the difference lies in that under mixed settings, general tokens never saturate and continuously push up the effective data volume. On the same topic, the design efficiency of proxy experiments itself is beginning to be studied as a research object (Data Mixing as Mixture Experiment), while dynamic mixing advances along RegMix's trajectory signals (RegMix-D).
This week, evidence continued to strengthen for shifting evaluation criteria from base loss to "post-training ceiling." Some work directly runs RL on intermediate checkpoints during pretraining, finding that pretraining corpus composition (proportion of math tokens) is a stronger lever for RL effectiveness than model scale—a 1B model with 60B math-heavy tokens achieves greater RL gains than a 4B model with the original ratio (RL Excursions during Pretraining); another work provides joint scaling laws for pretraining and RL on a controlled chess testbed (Understanding Reasoning from Pretraining to Post-Training). Together, these imply that when RL trainability is the criterion, corpus ratio may deserve optimization priority over compute allocation.
On the corpus construction side, joint optimization of quality-redundancy-diversity (CuraWeb) and source-aware rubric discovery in mid-training (MIRA) push filtering and selection from single thresholds to multi-objective and source-customized approaches; annealing-stage language mixing also gains controlled scan evidence (KletterMix).
Community and Developments
Around "corpus composition determines post-training ceiling," several directional advances emerged this week. Some work uses perplexity landscape as a data quality predictor for continual pretraining, proposing a perplexity-aware scaling law that goes beyond "scaling by data volume" (Perplexity-Aware Data Scaling Law); another work uses reinforcement learning to train a small agent that dynamically re-weights domain ratios during continual pretraining, achieving an average 3.02% improvement on math reasoning target domains and transferring to unseen source domains (Learning to Re-weight Domains for Continual Pre-training). Both push ratio decisions from static/heuristic toward model-aware dynamic decision-making.
Open Questions
- Does the 15–20 times repetition tolerance in mixed training vs. the <4 rule in single-source hold at ≥1B scale and under post-training metrics? Current evidence stops at ≤805M and uses validation loss as the criterion; if measured by pass@k or RL trainability, whether the "general data as regularizer" mechanism still dominates remains undetermined.
- Where is the sweet spot for corpus composition as a lever for RL effectiveness? "Math token proportion" provides directional evidence, but the exact ratio, when to mix, and interaction with model scale still lack quantitative prescriptions.
- Can source-aware rubrics (MIRA) and spectral geometry (DiReCT) maintain predictive power at ≥1B scale with post-training ceiling as the criterion? This determines whether they can be upgraded from methodological references to basis for ratio decisions.
Papers in this issue
Mixture training with generic data tolerates 15-20 repetitions of scarce target data, far exceeding single-source limits, and a repetition-aware scaling law predicts optimal mixtures saving up to 74% compute.
Editor's noteFirst to explicitly couple target domain weight h and repetition factor r into data-constrained mixture scaling laws. Core finding: general data as a regularizer allows mixed training to tolerate 15–20 repetitions (far exceeding the single-source <4 rule of thumb), and provides functional forms to predict optimal target weight from data volume and compute. Relative to #2 issue Scaling Domain Data Repetition (only studies domain-level repetition, not mixing weights) and #1 issue InfoLaw (models repetition as an independent term rather than a function of mixing ratio), the increment is the coupling of r and h and the mechanistic explanation of "general tokens never saturate." Evidence covers multilingual/domain/quality-filtered mixtures across 2000+ runs, but stops at ≤805M and uses validation loss as the criterion; needs cross-reading with larger-scale validation.
CURAWEB's unified curation framework, integrating domain-specific cleaning, soft semantic deduplication, and value-aware Power Sampling, yields a 2T-token corpus that boosts 3B-parameter LLM performance by 1.82% over DCLM, with gains persisting at 1T-token scale.
Editor's noteAdvances web-scale corpus construction from single quality filtering to joint optimization of quality-redundancy-diversity: domain-aware thresholds (STEM/code priority bypass) respond to #1 issue A Bitter Lesson's "more filtering is better" challenge, but implement via in-domain recalibration rather than global removal; soft semantic deduplication reduces the false deletion rate in the highest similarity interval from 37.5% to 28.03%. At 3B scale, 200B–1T tokens, with per-component ablations, evidence strength exceeds most corpus papers; limitation is that it still uses base benchmarks as the criterion and the corpus is not public.
MIRA discovers source-specific quality rubrics via a frontier teacher and distills them into student scorers, matching full-corpus performance at half the token budget.
Editor's noteAdvances mid-training data selection from fixed rubrics to "source-aware rubric discovery": automatically induces quality criteria per source group, distills them into scalable scorers, and pairs with source-conditional reliability masks and group-wise retention thresholds. Relative to #2 issue MidTool's content heuristics, the increment is making "what to evaluate" itself an empirical product, with evidence at 14B scale and half token budget; limitation is that the code-oriented setting limits generality.
REGMIX-D learns dynamic data mixture schedules from proxy loss trajectories, outperforming static baselines on 13 tasks while using only 25% of RegMix's compute budget.
Editor's noteAdvances RegMix from static to dynamic mixing: trains a trajectory-conditional regression model on proxy loss trajectories, supporting both offline schedule generation and online adaptation, and with 128 proxies (25% of RegMix budget) exceeds RegMix at 1B/25B scale across 13 downstream tasks. Relative to dynamic methods like Aioli/TiKMiX, the increment is reusing RegMix's existing proxy signals without introducing additional optimization mechanisms; limitation is single-domain (pile-cc) optimization signal and lack of direct comparison with other dynamic methods.
This paper reframes LLM data mixing as a mixture experiment, showing that sparse Scheffé models with I-optimal designs recover optimal domain proportions using 25% fewer proxy runs.
Editor's noteFormally models data mixing as classical mixture experiments, introducing sparse Scheffé response surfaces and model-robust I-optimal designs, treating proxy mixture selection as an experimental design problem rather than a pure prediction problem; simulations show ~25% savings in proxy runs. Relative to RegMix/Data Mixing Laws, the increment is the statistical efficiency perspective on proxy experiments, complementary to #2 issue Repetition Mismatch's "keep repetition rate subsampling"; limitation is that simulations are based on fitted surfaces at small scale, requiring caution in extrapolation.
Pretraining loss predicts post-RL performance, while pretraining compute predicts RL improvement rate, with optimal compute allocation shifting toward RL as total budget grows.
Editor's noteEstablishes joint scaling laws for pretraining and RL on a controlled chess testbed (up to 1B): pretraining loss predicts post-RL performance levels, pretraining token count predicts RL gain slopes, and provides difficulty-stratified policy evolution mechanisms (easy problems amplify preferred correct actions, hard problems expose tail correct actions). Relative to data-constrained scaling law works, the increment is explicitly modeling the pretraining→RL compute allocation trade-off, directly addressing the "post-training ceiling" criterion; limitation is transfer from toy domain to natural language, partially mitigated by math cases.
Translated English pretraining mixtures into German yield 4.5-point downstream gains at 7B scale, preserving reasoning capabilities without degrading English performance.
Editor's noteConstructs a 725B token German corpus by translating ClimbMix, and runs controlled ablations at 0.6B and 7B scale: 7B annealing scans show KletterMix proportions of 0-20% monotonically improve German four-task averages (+4.5pt) while English remains stable. Relative to translation-pretraining works like TransWebEdu, the increment is systematically preserving document boundaries/metadata and controlled language-mixing annealing scans; limitation is single run per condition, 12B token budget, and German gains must be weighed against 10,080 GPU-hour costs.
