# Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

> Pretraining loss and benchmark scores fail to predict post-SFT performance, as a checkpoint with worse pretraining metrics yields better final models after supervised fine-tuning.

- **Source:** [arXiv](https://arxiv.org/abs/2609.08966)
- **Published:** 2026-09-12
- **Permalink:** https://picx.dev/p/5bXiCA
- **Whiteboard:** https://picx.dev/p/5bXiCA/image

## Summary

## Summary

- This paper challenges the common assumption that the best pretraining checkpoint (by loss or benchmark scores) will remain the best starting point for subsequent training stages in a full LLM pipeline.
- Through a 30B-parameter mixture-of-experts (MoE) training pipeline, the authors demonstrate a **ranking reversal**: a checkpoint (COOLDOWN) that dominates on all pretraining signals performs significantly worse after supervised fine-tuning (SFT) than a checkpoint (CONSTANT) with worse pretraining metrics.
- The paper introduces **solution density** (fraction of Gaussian weight perturbations retaining task performance above a threshold) as a local parameter-space geometry signal that better predicts downstream trainability than unperturbed scores.
- Intermediate aggregates (mid-training, long-context) do not reliably anticipate the final ranking reversal, though rankings within a learning-rate sweep stabilize after long-context adaptation.
- The study shows that COOLDOWN's largest failure is a severe HumanEval+ collapse due to repetitive response generation (stopping failure), which persists across SFT learning-rate sweeps, indicating latent code capability is present but unstable.

## Introduction and Theoretical Foundation

Language-model checkpoints are typically compared by their current performance: lower loss or higher benchmark scores. This is sufficient when training ends at that checkpoint. However, modern LLMs are trained in sequential stages including pretraining, mid-training, long-context adaptation, and SFT. The relevant question becomes:

> Which checkpoint selection criteria will produce the **best final model**?

Using intermediate evaluations as selection criteria implicitly assumes checkpoint rankings are preserved by later training. The paper shows this assumption can fail.

**Key Concepts:**
- **Solution density** (Gan et al., 2026): the fraction of Gaussian perturbations that retain task performance above a threshold. Previously shown to increase with model scale; this paper shows it also varies across training trajectories of the same model.
- **Checkpoint merging**: a weighted average of checkpoints that induces an effective decay over updates.

The paper's central hypothesis: solution density varies not only with scale but also across training trajectories, and these differences help explain downstream adaptability.

## Methodology

**Model and Training Pipeline:**
- 30B-parameter MoE with 3B active parameters
- 7.5T-token pretraining → 100B tokens mid-training (8k context) → 100B tokens long-context adaptation (64k) → 10B tokens conversational SFT (64k)
- Optimizer state reset with repeated LR warmup at each stage

**Pretraining Sources:**
- **CONSTANT** and **COOLDOWN**: share first 6.7T tokens and total budget; COOLDOWN decays LR to 10% of max during final 800B tokens
- **MERGE**: weighted average of 20 equally spaced CONSTANT checkpoints over a 600B-token trailing window:
$$\theta_{\mathrm{merge}} = \sum_{i=1}^{20} \frac{i}{210} \theta_{i}$$

**Learning-Rate Sweep:**
- Nine combinations of mid/long LR factors: $\eta_{\mathrm{mid}}, \eta_{\mathrm{long}} \in \{1, \frac{1}{3}, \frac{1}{9}\}$
- Each factor scales a stage's peak LR relative to the preceding stage

**Solution Density Definition:**
For checkpoint $\theta$, benchmark $b$, perturbation $\epsilon$, and evaluation score $s_b(\theta)$, the relative-threshold solution-density profile is:

$$\delta_{\theta,b}(\tau) = \mathrm{Pr}_{\epsilon}\left[s_b(\theta + \epsilon) \geq \tau s_b(\theta)\right]\tag{1}$$

Each source checkpoint evaluated under 100 Gaussian perturbations on GSM8K and MBPP at standard deviations $\sigma_\epsilon \in \{0.005, 0.001\}$.

## Empirical Validation / Results

**Table 1: Pretraining vs. Post-SFT performance**

| Selection signal | CONSTANT | COOLDOWN | MERGE |
|---|---|---|---|
| Train loss ↓ | 1.717 | **1.627** | - |
| Validation loss ↓ | 1.734 | **1.648** | - |
| Pretraining aggregate ↑ | 0.415 | **0.440** | **0.460** |
| Post-SFT aggregate ↑ | **0.360** | 0.247 | **0.363** |

COOLDOWN improves every pretraining signal yet finishes significantly worse after SFT.

**Table 2: Post-SFT benchmark scores**

| Source | AIME | DAPO | Skywork | IFBench | HE+ | GPQA | Mean | w/o HE+ |
|---|---|---|---|---|---|---|---|---|
| CONSTANT | 0.221 | 0.430 | 0.390 | 0.238 | **0.646** | 0.237 | **0.360** | 0.303 |
| COOLDOWN | 0.171 | **0.450** | 0.355 | **0.250** | 0.049 | 0.207 | 0.247 | 0.287 |
| MERGE | 0.217 | **0.495** | 0.345 | 0.236 | **0.665** | 0.222 | **0.363** | 0.303 |

The aggregate gap is amplified by COOLDOWN's severe HumanEval+ failure (4.9% vs 64.6% and 66.5%); without HE+, the gap shrinks to 0.016.

**Ranking Correlations:**
- Mid-training aggregate vs. post-SFT: Pearson $r = 0.473$ (p = 0.142), Spearman $\rho = 0.482$ (p = 0.133) — weak
- Long-context aggregate vs. post-SFT: Pearson $r = 0.884$ (p < 0.001), Spearman $\rho = 0.964$ (p < 0.001) — strong
- Restricting to nine COOLDOWN trajectories: mid-training $r = 0.448$, long-context $r = 0.935$ — still fails to predict CONSTANT vs COOLDOWN reversal

**Solution Density Results:**
- On GSM8K at $\tau = 0.90$: CONSTANT = 27%, MERGE = 13%, COOLDOWN = 0%
- Separation persists at $\sigma_\epsilon = 0.001$
- Perturbation ordering matches trainability: CONSTANT and MERGE finish nearly tied, COOLDOWN lags

**Retuning SFT:**
- Changing SFT factor from 1/3 to 1 leaves 100% of AIME and 99% of GPQA responses at the 32k-token cap
- No tested learning rate resolves the stopping failure
- Evaluating only the first generated code block from COOLDOWN raises HumanEval+ from 7.3% to 61.0%, showing latent capability but still trailing CONSTANT (64.6%) and MERGE (66.5%)

## Theoretical and Practical Implications

**Theoretical Implications:**
- Checkpoint quality is **pipeline-conditional**: a checkpoint's value depends on what training comes next
- Conventional pretraining metrics (loss, benchmarks) can actively mislead checkpoint selection for downstream training
- Solution density provides a local parameter-space geometry signal that complements unperturbed scores: greater local robustness helps checkpoints tolerate parameter displacement from subsequent optimization
- Intermediate evaluation rankings are not stable predictors; predictiveness emerges only after long-context adaptation, and even then fails across different source checkpoints

**Practical Implications:**
- Checkpoint selection should target performance **after** the remaining pipeline, not current scores alone
- Solution density audits can identify trainable checkpoints before committing expensive downstream compute
- Eval scores can miss critical properties (e.g., stopping behavior) that dominate downstream performance
- MERGE (checkpoint averaging) offers a compute-free way to improve trainability without additional pretraining

## Conclusion

The paper's central finding is that **checkpoint quality depends on what comes next**. COOLDOWN has better pretraining metrics than CONSTANT but performs worse after the downstream pipeline. Rankings within a learning-rate sweep stabilize after long-context adaptation, yet neither intermediate aggregate anticipates the reversal between CONSTANT and COOLDOWN. Eval scores can miss properties relevant to continued training.

**Future directions** implied by this work:
- Developing selection criteria that predict post-pipeline performance
- Understanding when solution density becomes predictive and how to measure it efficiently
- Investigating stopping failures and their interaction with learning-rate schedules
- Extending the analysis to more model families, seeds, and task distributions

**Limitations acknowledged:**
- Results from one 30B MoE family with one seed per setup
- Solution density measured on only two tasks, no causal establishment
- COOLDOWN's stopping failure enlarges the observed gap

---

_Markdown view of https://picx.dev/p/5bXiCA, served by PicX — AI-generated visual whiteboard summaries of research papers._