Summary (Overview)

• GapFT (Gap Fine-Tuning) is a novel data-selection method that identifies which verified responses should enter post-training when the training budget is fixed. Rather than selecting responses by correctness alone, GapFT conditions selection on whether the source policy fails in a single attempt but succeeds within K search samples.
• The paper introduces a formal partition of problems into three evidence states: $G_K$ (single-sample success), $S_K$ (failure recovered by search), and $U_K$ (failure missed by search), with GapFT selecting exclusively from $S_K$.
• An exact decomposition identity attributes any Pass@1 change to corrections on $S_K$ and $U_K$ problems minus regressions on $G_K$ problems: $A_{dep}(\pi) - A_{dep}(\pi_{src}) = p_S \kappa_S + p_U \kappa_U - p_G \rho_G$.
• On LogiQA 2.0 and ReClor across Llama-3.1-8B, Qwen2.5-7B, and Mistral-7B, GapFT outperforms budget-matched uniform RFT in every setting, recovering two-thirds to four-fifths of the full verified pool's gain with only 11–34% of its problems.
• Key controls reveal that gains come from covering reachable failures, that failures found only by deep search trade correction for regression, and that gold-label selection of unreachable failures lowers accuracy.

---

Introduction and Theoretical Foundation

Background and Motivation

Large language models exhibit a well-documented gap between Pass@K and Pass@1: a single response often fails on problems the model can still solve when multiple responses are sampled and verified (Chen et al., 2021; Li et al., 2022). This has made test-time search a standard approach (Wang et al., 2023; Snell et al., 2025), but search must be paid again for every query.

Post-training (e.g., STaR, ReST, rejection-sampling fine-tuning) can amortize this cost by searching a training pool once and using verified responses to update the policy (Zelikman et al., 2022; Gulcehre et al., 2023). However, existing recipes judge verified responses by properties of the response itself (correctness, reward, diversity), ignoring a critical question: whether the response can change single-attempt behavior depends on the source model, not the response alone.

The Data-Selection Gap

On problems the source model already answers correctly, a verified response repeats existing behavior—it is redundant. On problems missed in one attempt but solved by search, the verified response supplies the exact target that deployment lacks. Because problems already solved typically outnumber those recovered by search, allocation without this distinction wastes a fixed budget on redundant examples. The paper calls this the data-selection gap: not how to obtain correct responses, but how to allocate them relative to what the deployed model already does.

Theoretical Foundation

The paper formalizes the setting with an exact verifier $V(x,y) \in \{0,1\}$ and defines deployment accuracy and search success:

$$A_{dep}(\pi) = \mathbb{E}_{x \sim \mathcal{D}}[V(x, d_\pi(x))], \qquad A_{search}(\pi; K) = \mathbb{E}_{x \sim \mathcal{D}, y_{1:K} \sim \pi(\cdot|x)} \left[ \max_{k \leq K} V(x, y_k) \right] \tag{1}$$

The offline RFT objective is maximum likelihood:

$$\mathcal{L}(\theta) = -\sum_{(x,y) \in \mathcal{T}} \sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{<t}) \tag{2}$$

---

Methodology

Evidence States

Each problem is placed in exactly one of three states relative to the source policy and search budget $K \geq 1$:

$$G_K = \{x : V(x, d_{src}(x)) = 1\} \tag{3}$$ $$S_K = \{x : V(x, d_{src}(x)) = 0, \exists y \in \mathcal{Y}_K^{\pi_{src}}(x) : V(x,y) = 1\} \tag{4}$$ $$U_K = \{x : V(x, d_{src}(x)) = 0, \forall y \in \mathcal{Y}_K^{\pi_{src}}(x) : V(x,y) = 0\} \tag{5}$$

These are observations under finite search, not capability labels—$U_K$ does not mean the source assigns zero probability to a correct response.

Evidence Selection Under a Matched Budget

GapFT uses two evidence sets:

$$\mathcal{E}_G = \{(x, d_{src}(x)) : x \in \mathcal{D}_{cand} \cap G_K\}, \quad \mathcal{E}_{S,K} = \{(x, y_K^*(x)) : x \in \mathcal{D}_{cand} \cap S_K\} \tag{6}$$

where $y_K^*(x)$ is one verified response chosen by a frozen rule (shortest verified response). GapFT sets $\mathcal{T}_{gap} = \mathcal{E}_{S,K}$, while baselines draw the same $n$ examples: uniform RFT samples $\mathcal{E}_G \cup \mathcal{E}_{S,K}$ regardless of state; solved replay samples $\mathcal{E}_G$ only (the no-search $K=0$ condition). Budgets are enforced as:

$$N = R|\mathcal{T}|, \qquad R \cdot \text{Tok}(\mathcal{T}) \simeq L, \qquad \text{Step}(\mathcal{T}; R) = J \tag{7}$$

Decomposition Identity

For a post-training policy $\pi$, define correction and regression rates:

$$\kappa_B = \Pr[V(x, d_\pi(x)) = 1 \mid x \in B_K] \text{ for } B \in \{S, U\}, \quad \rho_G = \Pr[V(x, d_\pi(x)) = 0 \mid x \in G_K] \tag{8}$$

Since $G_K$, $S_K$, $U_K$ partition the audit set:

$$A_{dep}(\pi) - A_{dep}(\pi_{src}) = p_S \kappa_S + p_U \kappa_U - p_G \rho_G \tag{9}$$

This identity holds exactly, with weights $p_B = \Pr_{x \sim \mathcal{D}_{audit}}[x \in B_K]$. The weight bound $p_S \kappa_S \leq p_S$ means no post-training can gain more from correction than the share of search-recovered failures in the audit set.

---

Empirical Validation / Results

Main Results (Table 1)

Descriptive equal-weight mean of GapFT–Source over six cells: +6.37 points.

Key findings:

• GapFT is above uniform RFT in all six settings; the pooled effect is positive with mean +1.57 [0.77, 2.45] points.
• Uniform RFT beats replay only for Llama (whose pools are 1/3 recovered failures); for Mistral and Qwen, uniform draws contain too few recovered failures.
• Llama-3.1-8B and Mistral-7B recover ~2/3 to 4/5 of the full pool's gain with 11–34% of its problems.

Why GapFT Works: Evidence Controls (Table 2)

A. Search depth (replay replaced by recovered failures):

• K=1 vs. K=0: +5.46 pp (LogiQA), +5.73 pp (ReClor)
• K=8 vs. K=0: +6.75 pp (LogiQA), +9.40 pp (ReClor)
• Failures recovered by the first sample deliver 3/5 to 4/5 of the total gain.

B. Randomized inclusion of failures first found in samples 5–8:

• All intervals include zero; adding deep-search failures trades correction for regression with no net gain.

C. Gold-label selection of arbitrary failures (vs. GapFT):

• LogiQA: −9.49 pp [−11.29, −7.68]; ReClor: −6.87 pp [−12.20, −1.40]
• Gold selection corrects more missed failures but causes far more regression on solved problems.

MATH Results (Table 3)

At matched training exposures with Qwen2.5-7B on 600 MATH problems ($K=8$):

Favoring coverage over repetition: 188 distinct problems × 4 exposures yields +2.94 pp vs. +2.33 pp for 94 problems × 8 exposures.

---

Theoretical and Practical Implications

Theoretical Implications

1. Amplification vs. acquisition: The decomposition places long-standing debates into one identity—amplification is correction on $S_K$, acquisition beyond observed support is correction on $U_K$, and the cost of adaptation is regression on $G_K$.

2. Search reachability as a selection signal: Reachability itself matters more than gold labels. When the budget holds no replay, filling it with unreachable failures (even with correct gold labels) lowers accuracy because fitting targets the source assigns little probability to pulls the policy away from correct decisions.

3. Bounded gains: The contribution $p_S \kappa_S$ is bounded by the share of search-recovered failures in the audit set—no post-training can gain more from correction than the failures search exposes.

Practical Implications

1. Data allocation principle: The source checkpoint's single-sample outcome should be an explicit variable in verified-data allocation. A training slot is worth little unless it holds a problem the source still fails.

2. Budget efficiency: GapFT achieves most of the full-pool gain with a fraction of the problems (11–34%), making it valuable when training budgets are constrained.

3. Deep search diminishing returns: At fixed budgets without separate replay, failures found only by deep search (samples 5–8) add no net gain—the value of search concentrates in its first few samples.

4. Model-dependent limits: Qwen2.5-7B's near-zero gain traces to having almost no recoverable failures ($S_8 \approx 2\%$ of audit), making any selection recipe moot.

---

Conclusion

GapFT turns search gains into single-response accuracy by targeting the gap between Pass@K and Pass@1: problems where the source model can generate a correct response within K samples but fails in its direct answer. The method fills a fixed training budget with verified $S_K$ responses without changing the objective, measured by corrections gained versus regressions incurred on a disjoint audit set.

Main takeaways:

• GapFT consistently outperforms budget-matched uniform RFT on logic benchmarks across three model families.
• Gains come from covering distinct reachable failures, not from correctness alone or answer frequencies.
• Failures search cannot reach cost regressions when the budget holds no replay.
• The gain is fundamentally bounded by how many transferable failures search exposes.

Future directions suggested by the work include: using GapFT to decide what to train on when budgets are larger (replaying solved successes alongside), extending the framework to online methods like PPO/GRPO, and investigating whether the reachability principle transfers to code generation and agentic settings where the search space differs.

Note: Details such as model training settings (LoRA, lr=5×10⁻⁵, one epoch, 3 seeds, R=8 repetitions), statistical procedures (2,000 bootstrap replicates), and additional controls (answer reshuffling, four independent sample sets for S/U attribution) are documented in the appendices. Code is available at https://github.com/Xuanxuana1/GapFT.