Fragility of Mid-Training Priors
Highlights of This Issue
The most notable signal this issue comes from stress tests on mid-training itself. One work systematically examines the robustness of alignment mid-training (AMT) across 10B–110B scales and 10M–1B tokens, finding that a tiny fraction (2%) of elicitation fine-tuning data carrying competing motives can override a 190M-token mid-training prior (a token ratio of about 1:4200), and that mid-training generalizes poorly to rules not demonstrated in the data, with effects varying significantly by post-training method (SFT vs RL) (Stress-testing Alignment Midtraining). This directly challenges the assumption in this direction that treats mid-training data selection as a mixing lever—it makes mid-training itself the object of study, providing negative evidence that "priors can be overridden by small amounts of downstream data," contrasting with the optimistic view in Issue 5's Knowledge Distillation During Mid-Training, which treats mid-training as a source of supervision. The authors conclude that there is currently insufficient public evidence to support mid-training solving the core difficulties of alignment, serving as an important warning for researchers who rely on mid-training as a data mixing tool.
This issue also sees a batch of data mixing optimization works that push search costs to extremely low levels, converging on "single-agent/near-zero-cost" mixing searches. FastMix reparameterizes mixing ratios as per-source loss weights under uniform sampling, using a single proxy model to jointly optimize mixing coefficients and model parameters via gradient descent, reducing search costs by 55–550x compared to RegMix/CLIMB, while covering both pretraining and SFT stages (FastMix); ByDoRe treats domain weights as a Dirichlet distribution for Bayesian variational inference, using uncertainty to smooth the jitter of deterministic weight optimization, achieving RegMix-level performance at about 0.8% of the search cost (ByDoRe); AutoData goes further, advancing data selection from "optimizing weights over fixed domains" to "searching over the program space of executable selection algorithms," where an agent automatically discovers composite scoring and diversity-preservation mechanisms on document features (AutoData). Relative to Issue 3's RegMix-D dynamic mixing and Issue 4's CausalMix causal estimation, these three works collectively push the cost-efficiency frontier of mixing optimization much further, but all still rely on base loss or post-SFT benchmarks as criteria, without addressing RL trainability.
Document extraction quality and mid-training synthetic data each see progress. WeVisDoc proposes a two-stage data-centric document parsing framework that uses held-out probes disjoint from training to diagnose residual errors within visual-structural clusters, deciding between "adding data vs. increasing exposure" under a fixed token budget, achieving the largest gains on degraded tracks at 2B/4B scales (WeVisDoc); MiST shifts mid-training corpus construction in the cybersecurity domain from token-scale expansion to "synthetic transformation of compact expert seed corpora," outperforming large-scale raw CPT with fewer tokens at 8B/32B, and providing stronger initializations for downstream RL/SFT (MiST); QVAC Genesis III uses a weak student model as a signal, converting student failures and successes into corrective explanations and contrastive multiple-choice reasoning, respectively, building a 191B-token open STEM synthetic corpus (QVAC Genesis III).
Community and Updates
FineBooks releases a public OCR evaluation leaderboard for "historical corpus construction": based on 2,165 pages of expert-transcribed historical book pages, it provides accuracy/cost frontiers for 14 open OCR models and explicitly answers whether "open OCR models are sufficient to unlock historical knowledge at scale"—for LLM training corpora, most leading models are already sufficient, and reprocessing the entire BHL public domain collection is practically feasible (FineBooks). This provides reproducible evidence for decisions on "whether to re-OCR and which model to use" when building corpora from historical PDFs/scans, serving as an evaluation resource on the document extraction quality axis that directly changes corpus reprocessing decisions.
Open Questions
- Is the "1:4200 override ratio" from Stress-testing AMT robust at larger scales and with different alignment targets? Is mid-training fundamentally unable to establish priors that persist against post-training data, or does the current setup (single seed, synthetic data) underestimate its potential?
- Do FastMix/ByDoRe's reductions of mixing search to single-agent or near-zero cost come at the expense of adapting to "post-training upper bounds"? Can these fast searches targeting base loss directly transfer to mixing ratios judged by RL trainability?
- Can the composite selection algorithms discovered by AutoData on ClimbMix transfer across corpora and domains? Is the choice of proxy objective for agentic search (val-bpb vs CORE) the key factor determining its upper bound?
- Can WeVisDoc's "residual-error-driven data allocation" generalize beyond document parsing to other data axes? Can formalizing the "add data vs. increase exposure" decision apply to mid-training and post-training data selection?
Papers in this issue
Alignment midtraining's effects are fragile: 2% conflicting fine-tuning data overrides 190M midtraining tokens, and gains vanish under RL post-training.
Editor's noteFirst systematic stress test of alignment midtraining robustness at realistic scales (10B–110B parameters, 10M–1B tokens): 2% conflicting fine-tuning data can override a 190M-token midtraining prior (about 1:4200 token ratio), with weak generalization to rules not demonstrated and effects varying significantly by post-training method (SFT vs RL). Relative to Issue 5's *Knowledge Distillation During Mid-Training*, which treats mid-training as a supervision source, its increment is making mid-training itself the object of study, providing strong negative evidence that "priors can be overridden by small amounts of downstream data," directly warning against assumptions that rely on mid-training as a data mixing lever.
FASTMIX reformulates data mixture optimization as a differentiable weighted bilevel problem, achieving state-of-the-art LLM performance up to 550x faster than prior methods using only one proxy model.
Editor's noteReparameterizes data mixing optimization as a differentiable bi-level optimization: mixing ratios are equivalent to per-source loss weights under uniform sampling, requiring only a single proxy model to jointly optimize mixing coefficients and model parameters via gradient descent, reducing search costs by 55–550x compared to RegMix/CLIMB, while covering both pretraining and SFT stages. Relative to Issue 3's RegMix-D dynamic mixing and Issue 4's CausalMix causal estimation, its increment is embedding mixing ratios directly into a differentiable training objective for end-to-end gradient optimization; limitations include small proxy scale and reliance on base loss and post-SFT benchmarks as criteria.
AutoData frames pretraining data selection as agentic search over executable algorithms, discovering recipes that outperform human-designed pipelines and transfer across model scales up to 1.3B parameters.
Editor's noteAdvances pretraining data selection from "optimizing weights over fixed domains" to "searching over the program space of executable selection algorithms": an agent iteratively proposes and refines scoring/layering/random selection rules on document features (lexical statistics, category labels, perplexity), with recipes discovered on a 125M proxy transferring without retuning to 1.3B and improving downstream CORE. Relative to Issue 3's RegMix-D and Issue 4's CausalMix, its increment is moving from weight optimization to algorithm space search, automatically discovering feature interactions; limitations include a single corpus (ClimbMix) and CORE gains not reaching statistical significance.
ByDoRe's Bayesian domain reweighting achieves state-of-the-art LLM pretraining performance with only 0.8% of RegMix's compute, without relying on fragile scaling-law assumptions.
Editor's noteUses a Gamma-Dirichlet hierarchical Bayesian model for variational inference on domain weights, treating weights as probability distributions rather than deterministic vectors, smoothing the jitter of DoReMi-type direct optimization methods with uncertainty, and adaptively estimating Gamma hyperparameters with a prior prediction network. At 1B/150B token scale, it achieves RegMix-level performance at about 0.8% of the search cost, with a 5.2pt improvement on specialized objectives. Relative to Issue 3's RegMix-D and Issue 4's CausalMix, its increment is the stability and efficiency from Bayesian inference; limitations include no public code and reliance on base benchmarks as criteria.
WeVisDoc's two-stage data-centric framework—broad coverage then capability-aware refinement under a fixed token budget—achieves state-of-the-art document parsing, ranking first on OmniDocBench and PureDocBench.
Editor's noteProposes a two-stage data-centric document parsing framework: Stage I broadens semantic/structural/appearance coverage, and Stage II uses held-out probes disjoint from training to diagnose residual errors within visual-structural clusters, deciding between "adding data vs. increasing exposure" under a fixed token budget. At 2B/4B scales, it achieves the largest gains on degraded tracks (4B +4.03pt on Real Degraded). Relative to Issue 5's OCR-EDR closed-loop repair and Issue 1's PureDocBench evaluation, its increment is explicitly coupling residual diagnosis with data construction and exposure reallocation under a fixed budget; limitations include a lack of per-component ablations.
QVAC Genesis III, a 191B-token STEM corpus built from model failures and contrastive reasoning, boosts 1.7B models by up to 28.57% on ARC-E over baselines.
Editor's noteBuilds a 191B-token open STEM synthetic corpus with a dual generation strategy using a weak edge-scale student model as a signal: converting student failures into corrective explanations and successes into contrastive multiple-choice reasoning, and introducing an LLM-as-parser Valid Answer Rate protocol to decouple answer extractability from domain knowledge. In 1.7B from-scratch pretraining ablations, it consistently outperforms Cosmopedia-v2 and Cosmo-1B. Relative to Issue 1's *Generating Pretraining Tokens from Organic Data* model-aware generation, its increment is systematically using student errors as teaching signals; limitations include only 1.7B scale and reliance on base benchmarks as criteria.
MiST shows curated mid-training with synthetic data from expert-vetted seeds outperforms raw continual pre-training, boosting cybersecurity accuracy by up to 13.1 points while preserving general capabilities.
Editor's noteShifts mid-training corpus construction from "token-scale expansion" to "synthetic transformation of compact expert seed corpora," and provides the first systematic comparison of synthetic mid-training vs. large-scale raw CPT in the cybersecurity domain: the former is better with fewer tokens (8B/32B +13.1/+8.6pt over Qwen baselines) and provides stronger initializations for downstream RL/SFT. Relative to Issue 4's *Closing Cost-Quality Gap* difficulty-aware curation, its increment is explicitly defining mid-training as a corpus design problem and providing ablations of the synthetic pipeline; limitations include no public recipe and no systematic scan of synthetic ratios.
