Summary

• This paper challenges the common assumption that the best pretraining checkpoint (by loss or benchmark scores) will remain the best starting point for subsequent training stages in a full LLM pipeline.
• Through a 30B-parameter mixture-of-experts (MoE) training pipeline, the authors demonstrate a ranking reversal: a checkpoint (COOLDOWN) that dominates on all pretraining signals performs significantly worse after supervised fine-tuning (SFT) than a checkpoint (CONSTANT) with worse pretraining metrics.
• The paper introduces solution density (fraction of Gaussian weight perturbations retaining task performance above a threshold) as a local parameter-space geometry signal that better predicts downstream trainability than unperturbed scores.
• Intermediate aggregates (mid-training, long-context) do not reliably anticipate the final ranking reversal, though rankings within a learning-rate sweep stabilize after long-context adaptation.
• The study shows that COOLDOWN's largest failure is a severe HumanEval+ collapse due to repetitive response generation (stopping failure), which persists across SFT learning-rate sweeps, indicating latent code capability is present but unstable.

Introduction and Theoretical Foundation

Language-model checkpoints are typically compared by their current performance: lower loss or higher benchmark scores. This is sufficient when training ends at that checkpoint. However, modern LLMs are trained in sequential stages including pretraining, mid-training, long-context adaptation, and SFT. The relevant question becomes:

Which checkpoint selection criteria will produce the best final model?

Using intermediate evaluations as selection criteria implicitly assumes checkpoint rankings are preserved by later training. The paper shows this assumption can fail.

Key Concepts:

• Solution density (Gan et al., 2026): the fraction of Gaussian perturbations that retain task performance above a threshold. Previously shown to increase with model scale; this paper shows it also varies across training trajectories of the same model.
• Checkpoint merging: a weighted average of checkpoints that induces an effective decay over updates.

The paper's central hypothesis: solution density varies not only with scale but also across training trajectories, and these differences help explain downstream adaptability.

Methodology

Model and Training Pipeline:

• 30B-parameter MoE with 3B active parameters
• 7.5T-token pretraining → 100B tokens mid-training (8k context) → 100B tokens long-context adaptation (64k) → 10B tokens conversational SFT (64k)
• Optimizer state reset with repeated LR warmup at each stage

Pretraining Sources:

• CONSTANT and COOLDOWN: share first 6.7T tokens and total budget; COOLDOWN decays LR to 10% of max during final 800B tokens
• MERGE: weighted average of 20 equally spaced CONSTANT checkpoints over a 600B-token trailing window:

$$\theta_{\mathrm{merge}} = \sum_{i=1}^{20} \frac{i}{210} \theta_{i}$$

Learning-Rate Sweep:

• Nine combinations of mid/long LR factors: $\eta_{\mathrm{mid}}, \eta_{\mathrm{long}} \in \{1, \frac{1}{3}, \frac{1}{9}\}$
• Each factor scales a stage's peak LR relative to the preceding stage

Solution Density Definition: For checkpoint $\theta$, benchmark $b$, perturbation $\epsilon$, and evaluation score $s_b(\theta)$, the relative-threshold solution-density profile is:

$$\delta_{\theta,b}(\tau) = \mathrm{Pr}_{\epsilon}\left[s_b(\theta + \epsilon) \geq \tau s_b(\theta)\right]\tag{1}$$

Each source checkpoint evaluated under 100 Gaussian perturbations on GSM8K and MBPP at standard deviations $\sigma_\epsilon \in \{0.005, 0.001\}$.

Empirical Validation / Results

Table 1: Pretraining vs. Post-SFT performance

COOLDOWN improves every pretraining signal yet finishes significantly worse after SFT.

Table 2: Post-SFT benchmark scores

The aggregate gap is amplified by COOLDOWN's severe HumanEval+ failure (4.9% vs 64.6% and 66.5%); without HE+, the gap shrinks to 0.016.

Ranking Correlations:

• Mid-training aggregate vs. post-SFT: Pearson $r = 0.473$ (p = 0.142), Spearman $\rho = 0.482$ (p = 0.133) — weak
• Long-context aggregate vs. post-SFT: Pearson $r = 0.884$ (p < 0.001), Spearman $\rho = 0.964$ (p < 0.001) — strong
• Restricting to nine COOLDOWN trajectories: mid-training $r = 0.448$, long-context $r = 0.935$ — still fails to predict CONSTANT vs COOLDOWN reversal

Solution Density Results:

• On GSM8K at $\tau = 0.90$: CONSTANT = 27%, MERGE = 13%, COOLDOWN = 0%
• Separation persists at $\sigma_\epsilon = 0.001$
• Perturbation ordering matches trainability: CONSTANT and MERGE finish nearly tied, COOLDOWN lags

Retuning SFT:

• Changing SFT factor from 1/3 to 1 leaves 100% of AIME and 99% of GPQA responses at the 32k-token cap
• No tested learning rate resolves the stopping failure
• Evaluating only the first generated code block from COOLDOWN raises HumanEval+ from 7.3% to 61.0%, showing latent capability but still trailing CONSTANT (64.6%) and MERGE (66.5%)

Theoretical and Practical Implications

Theoretical Implications:

• Checkpoint quality is pipeline-conditional: a checkpoint's value depends on what training comes next
• Conventional pretraining metrics (loss, benchmarks) can actively mislead checkpoint selection for downstream training
• Solution density provides a local parameter-space geometry signal that complements unperturbed scores: greater local robustness helps checkpoints tolerate parameter displacement from subsequent optimization
• Intermediate evaluation rankings are not stable predictors; predictiveness emerges only after long-context adaptation, and even then fails across different source checkpoints

Practical Implications:

• Checkpoint selection should target performance after the remaining pipeline, not current scores alone
• Solution density audits can identify trainable checkpoints before committing expensive downstream compute
• Eval scores can miss critical properties (e.g., stopping behavior) that dominate downstream performance
• MERGE (checkpoint averaging) offers a compute-free way to improve trainability without additional pretraining

Conclusion

The paper's central finding is that checkpoint quality depends on what comes next. COOLDOWN has better pretraining metrics than CONSTANT but performs worse after the downstream pipeline. Rankings within a learning-rate sweep stabilize after long-context adaptation, yet neither intermediate aggregate anticipates the reversal between CONSTANT and COOLDOWN. Eval scores can miss properties relevant to continued training.

Future directions implied by this work:

• Developing selection criteria that predict post-pipeline performance
• Understanding when solution density becomes predictive and how to measure it efficiently
• Investigating stopping failures and their interaction with learning-rate schedules
• Extending the analysis to more model families, seeds, and task distributions

Limitations acknowledged:

• Results from one 30B MoE family with one seed per setup
• Solution density measured on only two tasks, no causal establishment
• COOLDOWN's stopping failure enlarges the observed gap