Base Loss Optimal ≠ Post-Training Optimal: An Empirical Moment of Criterion Reversal

Highlights of This Issue

This issue brings the most direct empirical endorsement of the "post-training upper bound" criterion. One work, on a full 30B MoE training pipeline, found that checkpoints selected as "optimal" by base loss or base benchmarks may actually perform worse after full downstream SFT, and that what truly determines post-training performance is the checkpoint's "solution density"—the ability to maintain downstream performance under local weight perturbations (Good Pretraining, Bad SFT). This provides real-scale validation for the pretraining→RL scaling law established in toy board-game domains by Understanding Reasoning from Pretraining to Post-Training (Issue 3), and advances "evaluating data recipes by post-training upper bound rather than base loss" from a methodological claim to an actionable checkpoint selection criterion. Under the same criterion, RLVR data selection is shown to be "regime-dependent" rather than reducible to a static quality ranking—selection signals based on reasoning-circuit activation strength outperform at low participation quantiles rather than high ones (CircuitLens), suggesting that post-training data value depends heavily on the specific state of the model and data pool.

Mid-training data selection gains a new lever this issue: treating the distillation objective itself as a supervision source. One work systematically characterizes the starkly different behavior of knowledge distillation in mid-training versus pretraining—standard forward-KL distillation improves both reasoning and factual recall during pretraining, but during mid-training it improves reasoning at the cost of factual recall, with the mechanism lying in asymmetric teacher entropy across procedural/knowledge-dense domains and students learning low-entropy facts first. The proposed Switch Distillation uses teacher predictive entropy as a token-level routing signal, applying reverse-KL distillation only to low-entropy tokens, and in controlled experiments with a 1B student, 7B/13B teachers, and 60B tokens, it significantly improves the reasoning-recall trade-off, with gains persisting after full post-training (SFT+DPO+RLVR) (Knowledge Distillation During Mid-Training). Relative to Beyond Solvability (Issue 4), which uses learnability as a prior for RL data selection, and MIRA (Issue 3), which uses source-aware rubrics for mid-training data selection, the increment here is using "teacher entropy" as a token-level routing signal for mid-training supervision. On the synthetic data side, a theoretical work provides exact finite-sample bounds in high-dimensional linear regression, showing that mixed training induces strong model collapse, while a "synthetic-first, real-later" two-stage training can avoid the collapse lower bound, and that larger models amplify synthetic degradation under mixed training (Learning with Synthetic Data via SGD)—offering a theoretical complement to empirical findings on training-protocol choices for synthetic data.

In data-scarcity and repetition, this issue adds a previously overlooked moderating variable: model sparsity. One work explicitly couples MoE sparsity with repetition harm for the first time, finding that MoEs overfit repeated data more severely than compute-matched dense models, that the optimal repetition count decreases with the number of experts, and that harm is driven mainly by total parameter count rather than activated parameters (Data Scarcity and Model Sparsity). This directly responds to the dense-model repetition scaling law in Internal Data Repetition (Issue 1), introducing sparsity into data-scarcity decisions. The document-extraction quality axis also advances: a rendering-aware, closed-loop OCR diagnosis-repair framework uses "rendering equivalence" as the criterion for preserving valid predictions, significantly improving formula recognition on a 9B model (OCR-EDR); a multidimensional educational-quality scorer decomposes QuRating's single scalar quality into actionable 20-dimensional educational axes, used simultaneously for pretraining filtering and GRPO rewards (Edu-QuRating).

Community and Dynamics

A noteworthy evaluation perspective emerges on the document-extraction quality axis: practitioners separate "layout fidelity" from character accuracy as an independent data-quality dimension, orthogonal to CER, and argue it independently affects downstream extraction (OCR Benchmarks & Real-World Documents). This provides a second actionable axis for evaluating parser outputs when building corpora from PDFs/scans. At the community level, a practitioner commentary observes that "pretraining progress is mostly coming from data" (Pretraining progress is mostly coming from data); while offering no new empirics, it reflects that data curation and synthetic-data boundaries are becoming a community focus.

Open Questions

1. How do the mechanism and transferability of "solution density" as a checkpoint selection criterion (Good Pretraining, Bad SFT) hold? Why would base loss and post-training performance be inversely correlated, and is solution density still a reliable predictor at larger scales and under different architectures?
2. To what extent does Switch Distillation's teacher-entropy routing hold? How does the optimal distillation ratio vary with teacher scale, domain, and training stage, and can the routing signal generalize to other distillation objectives?
3. Is the conclusion that MoEs overfit repeated data more severely robust at larger scales and with more regularization? Can the sparsity-repetition-capacity coupling be incorporated into existing data-scarcity scaling laws?
4. Does CircuitLens's finding that "low participation quantiles are better" and its regime dependence imply that RLVR data selection is fundamentally unsolvable with a single static signal? Under what conditions are mechanism-level signals and difficulty/quality signals complementary?