GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR
Summary (Overview)
- Core problem: In offline RLVR (Reinforcement Learning with Verifiable Rewards), not all responses and tokens are equally informative, yet standard approaches supervise all tokens of retained responses equally.
- Key theoretical contribution: The authors derive data selection principles from the bidirectional KL divergence between the current policy and the Gibbs-optimal target policy of KL-regularized RLVR, showing that correct responses are governed by forward KL (coverage) while incorrect responses are governed by reverse KL (over-commitment).
- Proposed method: GIST (Gibbs-Targeted Hierarchical Data Selection), a two-level selection method that uses only three observable signals—sampling temperature, verifier outcome, and initial-policy token probabilities—to select responses and tokens.
- Empirical results: GIST improves accuracy across GSM8K, MATH-500, and MBPP for all three tested models under both BOLT and SFT objectives (29 of 30 settings improved), and outperforms seven selection baselines while supervising 66.6% fewer tokens.
- Key insight: The selection principles are independent of the regularization parameter β—only the balance between correct and incorrect response classes depends on β.
Introduction and Theoretical Foundation
Background
Offline RLVR trains language models on a fixed pool of responses whose outcomes are verified automatically. Common approaches supervise every token of retained responses, but responses and tokens differ in informativeness. Low-probability tokens in high-temperature samples may be noise, but in verified correct responses they likely represent valid steps the policy under-weights. Conversely, failures sampled at low temperature reveal errors near the policy's mode.
Theoretical Foundation
The paper builds on KL-regularized RLVR, which maximizes:
Proposition 2.1 (Gibbs target): The unique maximizer is:
with the objective gap:
Offline fitting: The forward KL can be rewritten as a weighted likelihood objective:
Token-level factorization: The Gibbs target factorizes over tokens as:
Methodology
Bidirectional Mismatch and Curvature Score
For a fixed prompt , define and . The bidirectional mismatch is:
Theorem 3.1 (KL-curvature bounds): Under bounded density ratios :
where and .
Corollary 3.3 (Informative correct and incorrect responses):
- If and : (forward term dominates) — favors under-represented correct responses
- If and : (reverse term dominates) — favors over-represented incorrect responses
Proposition 3.4 (Exact mismatch at reference policy): At , where . With the reference success probability: and .
Token-Level Mismatch
Corollary 3.6 (Informative tokens):
- In correct responses: low-probability tokens () carry the forward mismatch
- In incorrect responses: high-probability tokens () carry the reverse mismatch
Proposition 3.7: At : where and .
GIST Algorithm
Response-level selection (temperature pairing):
- Correct responses from on-policy pool ()
- Incorrect responses from near-mode pool ()
- Keep prompts with both classes (eligible prompts), sample correct and incorrect uniformly
Token-level selection (masks):
where is the mean token probability over the response.
Training: Combined with BOLT (weighted by , rescaled so prompt weights average to 1) or SFT (unit weights).
Empirical Validation / Results
Main Results (Table 2)
| Model | Method | GSM8K Strict | GSM8K Flex. | MATH-500 Strict | MATH-500 Flex. | MBPP pass@1 | Avg. |
|---|---|---|---|---|---|---|---|
| Qwen3-0.6B | BOLT | 43.29 | 47.46 | 44.60 | 50.40 | 26.60 | 41.49 |
| BOLT+GIST | 45.79 | 49.73 | 45.40 | 51.00 | 31.40 | 44.04 | |
| Qwen3.5-2B | BOLT | 54.13 | 54.51 | 46.40 | 54.40 | 42.40 | 50.44 |
| BOLT+GIST | 56.63 | 57.09 | 47.40 | 56.20 | 41.80 | 51.70 | |
| Gemma-4-E2B | BOLT | 12.89 | 26.31 | 40.80 | 52.80 | 56.40 | 45.17 |
| BOLT+GIST | 14.10 | 29.57 | 42.20 | 53.20 | 57.00 | 46.59 |
Comparison with Baselines (Table 3, Qwen3-0.6B/GSM8K)
| Metric | Random | PODS | RFT-style | RSI-S | ETS | ProFit | HES | GIST |
|---|---|---|---|---|---|---|---|---|
| Strict ↑ | 43.14 | 43.52 | 44.05 | 43.21 | 43.75 | 42.46 | 44.20 | 45.79 |
| Flex. ↑ | 48.60 | 48.22 | 48.14 | 47.84 | 47.54 | 46.47 | 47.31 | 49.73 |
Ablation Study (Table 4)
Removing any component hurts performance:
- Removing all selection: −1.97/−3.03 points
- Removing pairing: −2.20/−0.98 points
- Removing mask: −2.04/−2.12 points
- Removing Boltzmann weights: −1.97/−1.59 points
Key interaction: The mask adds 2.04/2.12 points with pairing but only 0.45/0.15 without it.
Efficiency (Table 6)
| All | Random | GIST | |
|---|---|---|---|
| Tokens | 13.60M | 6.64M | 4.55M |
| Reduction | - | 51.2% | 66.6% |
| Strict ↑ | 43.82 | 43.14 | 45.79 |
GIST also requires only 2,394 optimizer steps vs. 4,779 for all candidates (49.9% fewer).
Theoretical and Practical Implications
Theoretical Implications
-
Unified selection framework: The paper provides a principled, theory-driven framework for data selection in offline RLVR, unifying response-level and token-level selection under a single bidirectional KL-divergence analysis.
-
β-independence of selection principles: The four selection principles (Table 1) hold for every ; only the class balance depends on β through the ratio:
- Exact factorization at reference policy: The mismatch factorizes exactly at the reference policy, enabling principled selection before training begins.
Practical Implications
-
Efficiency: GIST achieves higher accuracy while supervising 66.6% fewer tokens and taking 49.9% fewer optimizer steps—significant compute savings.
-
Generality: GIST uses only observable signals (temperature, verifier outcome, initial-policy probabilities), works across model families (Qwen, Gemma), and pairs with different offline objectives (BOLT, SFT).
-
Contrary to existing practice: Unlike ProFit which keeps high-probability tokens, GIST keeps low-probability tokens of correct responses—a finding that challenges existing heuristics.
Conclusion
GIST addresses the central design question in offline RLVR: which responses and tokens to supervise. By analyzing the bidirectional KL mismatch between the policy and its Gibbs-optimal target, the authors derive four selection principles that separate the roles of correct and incorrect responses at both response and token levels. The resulting method improves accuracy across nearly all tested settings while supervising substantially fewer tokens.
Future directions suggested by the authors include:
- Applying the analysis to other offline objectives
- Settings where data are re-selected as the policy changes (online/iterative selection)
- The broader principle that "which responses and which tokens to train on can be read from the policy itself"
Related papers
- GUARD: Stabilizing Recursive Self-Improvement with Regret Guidance
GUARD achieves non-regressive agent harness self-improvement without external graders by separating edit-induced regressions from capability gaps via paired regret, guaranteeing monotonic accuracy gains.
- From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop
RepoFactory's proxy-driven environment evolution loop yields RepoHorizen-35B-A3B, a model improving coding-agent benchmarks by 22.6 points on NL2Repo and 8.5 points on average.
- The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements
Agent evolution fails because noisy evaluations cannot identify helpful improvements, not because proposals are lacking, making selection nearly random.