Summary (Overview)

• TailSFT is a novel supervised fine-tuning (SFT) algorithm that filters out already-fit sequences during training, focusing learning on the under-modeled "tail" of the response distribution to improve coverage for subsequent reinforcement learning (RL).
• The method achieves up to 16.8% absolute improvement in pass@16 on coding benchmarks and 3.1% on math benchmarks compared to standard SFT, with minimal computational overhead.
• TailSFT checkpoints consistently translate to up to 3.9% absolute pass@1 gains in subsequent GRPO runs, demonstrating that higher-coverage initializations serve as better starting points for RL.
• The authors introduce a lightweight coverage-ratio diagnostic (Definition 4.1) that provides a sufficient condition for predicting when TailSFT will improve coverage.
• The work motivates a stage-aware approach to model development, where intermediate checkpoints are judged by how effectively they support subsequent training rather than by standalone performance metrics alone.

---

Introduction and Theoretical Foundation

Background and Motivation

Modern AI systems are typically trained in multiple phases:

1. Next-token prediction on general-purpose corpora
2. Supervised fine-tuning (SFT) for instruction following and domain adaptation
3. Reinforcement learning (RL) for complex reasoning tasks

A tacit assumption in such pipelines is that improving the objective at one stage produces a better initialization for the next. However, recent work suggests this assumption need not hold—models that perform better on the local objective of one stage can be worse starting points for subsequent training (Zhang et al., 2026a; Wang et al., 2025; Springer et al., 2025; Chen et al., 2025).

The Coverage Principle

The paper formalizes the coverage principle, which relates a model's coverage profile to what repeated sampling can recover during RL. The key insight is that for RL with verifiable rewards (RLVR), the usefulness of rollouts depends strongly on the initialization—if rewarding responses are difficult to sample from the SFT model, RL has little positive learning signal to reinforce.

Definition 2.1 (Coverage Profile). Given a data-generating policy $\pi_{\mathrm{D}}$ and model $\pi$, the coverage profile at scale N is:

$$\operatorname{Cov}_{N}\left(\pi_{\mathrm{D}} \| \pi\right) := \operatorname{Pr}_{x \sim \mu, y \sim \pi_{\mathrm{D}} (\cdot | x)} \left[ \frac{\pi_{\mathrm{D}} (y \mid x)}{\pi (y \mid x)} \geq N \right]$$

The coverage profile measures the probability mass under $\pi_{\mathrm{D}}$ that $\pi$ underweights by a factor of at least $N$; smaller values indicate better coverage. Chen et al. (2026a) show that if $\operatorname{Cov}_{N}(\pi_{\mathrm{D}} \| \pi) \le \frac{1}{2}$, then $K \ge 2N \log(1/\varepsilon)$ samples suffice for Best-of-K to achieve expected reward within $\operatorname{Cov}_{N}(\pi_{\mathrm{D}} \| \pi) + \varepsilon$ of $\pi_{\mathrm{D}}$.

Key Insight: Cross-Entropy vs. Coverage

Standard SFT is not designed to preserve or improve coverage. Cross-entropy rewards increases in likelihood of every demonstration, including responses the model can already produce readily. Further fitting these responses can shift probability mass away from potentially useful responses. Consequently:

A model with worse cross-entropy can have better coverage, and vice versa.

Since $\pi_{\mathrm{D}}(y \mid x)$ is generally unknown, the coverage profile cannot be measured directly. However, for binary rewards, pass@K (the expected reward of Best-of-K sampling) serves as an empirical measure of coverage:

$$\text{pass@}K(\pi) := \mathbb{E}_{x \sim \mu}\left[1 - (1 - p_{\pi}(x))^K\right]$$

---

Methodology

The TailSFT Algorithm

TailSFT combines two key ideas:

1. Stopping updates on responses that are already sufficiently well modeled (via filtering)
2. Determining which responses to filter relative to the initial policy $\pi_0$

Algorithm 1: TAILSFT

Require: Initial policy $\pi_0$, SFT data D, filtering schedule $\{\gamma_t\}$

1: Record $\ell_i^0 \leftarrow \ell_i(\pi_0)$ for every $(x_i, y_i) \in \mathcal{D}$

2: for each training step t do

3: Sample a batch $B_t$

4: Compute $\ell_i^t \leftarrow \ell_i(\pi_t)$ for each $(x_i, y_i) \in \mathcal{B}_t$

5: Let $F_t \subseteq B_t$ be the $\gamma_t$ fraction of sequences with the smallest $\ell_i^t - \ell_i^0$

6: Set $\pi_{t+1}$ by taking a gradient step from $\pi_t$ on

$\mathcal{L}_t(\pi) = \frac{\sum_{i \in \mathcal{B}_t \setminus \mathcal{F}_t} -\log \pi(y_i | x_i)}{\sum_{i \in \mathcal{B}_t \setminus \mathcal{F}_t} |y_i|}$

Filtering is applied at the sequence level. The length-normalized loss is used only to determine which sequences are filtered; optimization uses the standard token-averaged cross-entropy over target tokens in retained sequences.

Alternative Filtering Methods

The authors compare TailSFT against two alternatives:

• Absolute filtering: stops training on an example once its loss falls below a fixed threshold
• Quantile filtering: filters the lowest-loss fraction of each batch

These alternatives isolate the benefit of filtering itself from the benefit of measuring progress relative to the initial policy.

Theoretical Analysis

The target policy is assumed to preserve the relative preferences of the initial policy among rewarding responses:

$$\pi^{\star}(y) \propto \pi_0(y) \mathbf{1}\{y \in S\}$$

where S is the set of rewarding responses. The absolute filtering loss is:

$$\ell_{\mathrm{abs}}(\pi ; x, y) = \left[\ell_{\pi}(x, y) + \log(\alpha)\right]_{+}$$

The TailSFT offset loss is:

$$\ell_{\mathrm{off}}(\pi ; x, y) = \left[\ell_{\pi}(x, y) - \ell_{\pi_0}(x, y) + \log \beta\right]_{+}$$

which stops updating once $\pi(y \mid x) \geq \beta \pi_0(y \mid x)$.

Theorem 3.1 (Informal). For any initial policy, rewarding set, and SFT sample, offset filtering can be tuned to achieve coverage no worse than standard SFT or the best absolute threshold:

$$\inf_{\beta} \operatorname{Cov}_{N}(\pi^{\star} \| \pi_{\mathrm{OFF}, \beta}) \leq \min \left\{\operatorname{Cov}_{N}(\pi^{\star} \| \pi_{\mathrm{ERM}}), \inf_{\alpha} \operatorname{Cov}_{N}(\pi^{\star} \| \pi_{\mathrm{ABS}, \alpha}) \right\}$$

The inequality can be strict, and absolute filtering can be worse than standard SFT.

---

Empirical Validation / Results

Controlled Graph Navigation Task

The authors first validate TailSFT in a controlled setting where each prompt specifies a layered directed graph with exactly eight valid s→t paths. The SFT data deterministically selects and rewards a single path.

Key findings (Figure 3):

• Standard SFT achieves the lowest cross-entropy and highest pass@1
• All three filtering methods achieve substantially higher pass@8 (used as a coverage surrogate)
• Filtering trades single-sample accuracy for better coverage under repeated sampling

Language Model Experiments

Setup: OLMo-3 7B base model, fine-tuned on math (OpenMathInstruct-2) and code (Magicoder, BigCode Self-OSS-Instruct, OpenCodeInstruct) instruction data, evaluated on 18 dataset–benchmark pairs.

Key Results (Table 1):

TailSFT improves pass@16 in 15 of 18 cases, while changes in pass@1 are mixed—consistent with the coverage motivation.

Coverage Ratio Diagnostic

Definition 4.1 (Coverage ratio). The base-reachable set is $\mathcal{R}_0 := \{i : 0.05 < P_{i,16}(\pi_0) < 0.95\}$. Coverage lost and gained by standard SFT:

$$L := \sum_{i \in \mathcal{R}_0} \left[f_{16}\left(P_{i,1}(\pi_0)\right) - f_{16}\left(P_{i,1}(\pi_{\mathrm{SFT}})\right)\right]_{+}$$ $$G := \sum_{i \in \mathcal{R}_0} \left[f_{16}\left(P_{i,1}(\pi_{\mathrm{SFT}})\right) - f_{16}\left(P_{i,1}(\pi_0)\right)\right]_{+}$$

The coverage ratio is $\rho_{16} := L/G$ (set to $\infty$ if $G = 0$). When $\rho_{16} > 1$, standard SFT loses more base-reachable coverage than it gains. All 11 dataset–benchmark pairs with $\rho_{16} > 1$ have nonnegative coverage gains from TailSFT (10 improve, 1 unchanged).

GRPO Results (Table 2)

Key finding: Initializing from TailSFT improves post-RL pass@1 in every matched comparison. Notably, in coding experiments, each TailSFT checkpoint has lower pass@1 than its standard SFT counterpart before RL, but higher pass@1 afterward. TailSFT runs also show faster early learning (up to 2.5× faster reward increases).

---

Theoretical and Practical Implications

Theoretical Implications

1. Stage-aware optimization: The results demonstrate that the objective appropriate for producing a strong model at one stage may not produce the best initialization for the next. Cross-entropy and coverage can be in tension, and high large-k pass@k can be more valuable than lower cross-entropy for downstream RL.

2. Role of the initial policy: Theorem 3.1 formalizes that $\pi_0$ contains useful information about how probability should be distributed among rewarding responses. Standard SFT can overwrite this structure by fitting empirical frequencies of a finite sample, while absolute filtering pushes responses toward a common threshold regardless of their starting point.

3. Connection to direct coverage optimization: TailSFT relates to the direct coverage optimization (DCO) loss:

$$\ell_{\mathrm{DCO}}(\pi ; x, y) = -\log\left(1 - (1 - \pi(y \mid x))^K\right)$$

which is a soft analogue for filtering without reference to the initial policy. The theoretical analysis shows offset filtering can be strictly better.

Practical Implications

1. Drop-in replacement: TailSFT is a lightweight, drop-in objective for SFT that requires only recording initial losses and filtering batches—minimal computational overhead.

2. Diagnostic tool: The coverage-ratio diagnostic ($\rho_{16}$) provides a practical, cheap way to identify settings where TailSFT is most likely to help, requiring only the base model and a single standard SFT run.

3. Broader outlook: The results motivate a holistic approach to language modeling—targeting the full training trajectory rather than fitting each stage independently. This has implications for how intermediate checkpoints should be evaluated and selected in multi-stage training pipelines.

---

Conclusion

TailSFT is presented as a drop-in replacement for standard SFT that filters already-fit sequences to improve coverage after the SFT stage, producing improvements in post-RL performance. The method is derived using a combination of theoretical analysis and controlled empirics, and significantly outperforms standard SFT in language modeling experiments.

Main takeaways:

• Filtering already-fit examples during SFT improves coverage (high pass@K at large K) at the cost of cross-entropy and pass@1
• Measuring progress relative to the initial policy is theoretically justified and practically effective
• Higher-coverage checkpoints consistently translate to better post-RL performance
• The coverage-ratio diagnostic provides a sufficient condition for predicting TailSFT's benefit

Future directions suggested by the work include:

• Adopting a more holistic, stage-aware approach to language model optimization
• Further investigating the interaction between SFT and RL stages
• Exploring whether similar filtering principles apply to other multi-stage training transitions (e.g., pre-training to SFT)

The authors believe their results motivate a shift in how language models should be optimized, "toward targeting the full training trajectory rather than fitting each stage independently."