Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining

Summary (Overview)

• Core contribution: The paper introduces a method to identify where reinforcement learning (RL) fails in language model reasoning and repairs the base model before RL via targeted continued pretraining (CPT) on a tiny, failure-guided corpus.

• Key finding: RL failures are localized — whether a rollout fails is strongly predicted by a presence of a small set of characteristic n-grams (e.g., "let's recheck", "the inverse of A"). These failure n-grams persist throughout RL training.

• Method: Failure n-grams mined from RL rollouts serve as exact-match queries to retrieve text from pretraining corpora (Olmo-mix, Dolmino-mix). The resulting corpus is only 35M tokens — 3–4 orders of magnitude smaller than typical mid-training.

• Results: The approach outperforms matched controls (same pipeline, content-neutral queries) in 10 out of 11 settings, including +2.0 points on GSM8K when inserted before mid-training (3 RL seeds). The corpus transfers across model families (OLMo2 → Qwen3).

• Theoretical insight: RL under GRPO cannot self-repair low-value states because on-policy sampling rarely produces successful continuations from those states; the deficit is conditional (local), not a global coverage gap — 46% of affected questions have a correct sibling rollout.

---

Introduction and Theoretical Foundation

Background and Motivation

Reinforcement learning with verifiable rewards (RLVR) is the dominant recipe for improving reasoning in language models, yet growing evidence suggests it sharpens rather than expands what the base model can already do. RL makes successful solutions more likely and failing ones less likely, but rarely produces solutions the base model could not already sample.

The central practical question: when RL fails, what exactly is the base model missing, and can we supply it before RL?

Theoretical Foundation

The paper frames rollouts as trajectories through latent reasoning states. After emitting $t$ tokens, the model is in state $s_t$ summarizing the prefix. The value of a state under policy $\pi$ is:

$$V^{\pi}(s) = \Pr[\text{eventual success} \mid s] \tag{3}$$

An n-gram $g$ marks states $s_g$ where the model emits it. The conditional accuracy of rollouts containing $g$:

$$a(g) = \Pr\left[r(x, y) = 1 \mid E_g(y) = 1\right] \tag{2}$$

estimates the average value $\bar{V}^{\pi}(s_g)$ of marked states. Failure n-grams mark low-value states — states from which the model rarely recovers.

GRPO Objective

The policy is trained with GRPO (Shao et al., 2024), maximizing:

$$\mathcal{J}(\theta) = \mathbb{E}_{x, \{y_i\}} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min\left(\rho_{i,t} \hat{A}_i, \mathrm{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon) \hat{A}_i\right) \right] - \beta \mathbb{D}_{\mathrm{KL}}\left[\pi_{\theta} \parallel \pi_0\right] \tag{1}$$

where $\rho_{i,t} = \pi_{\theta}(y_{i,t} \mid x, y_{i,<t}) / \pi_{\theta_{\mathrm{old}}}(y_{i,t} \mid x, y_{i,<t})$ is the token-level importance ratio, $\epsilon$ the clipping range, and $\beta$ the KL penalty weight.

Key theoretical argument: Because advantages are estimated only from rollouts the policy itself samples, RL learns how to act in a given state only from continuations sampled there. When $V^{\pi}(s_g)$ is low, few rollouts succeed from $s_g$, so RL has little signal to repair it.

Two Archetypes of Failure States

1. Unproductive behavior: The model reflects (e.g., "let's recheck") and then spirals rather than recovers — the pieces exist, but their composition fails.
2. Missing skill: The model stalls on a specific operation (e.g., row reduction, "the inverse of A") the subproblem requires.

---

Methodology

Step 1: Mining Failure n-Grams

• N-grams are mined from reasoning spans of $\pi_{\mathrm{RL}}$'s training rollouts
• Filtering criteria:
  • Occur at least 20 times and in ≥ 0.1% of rollouts
  • Remove candidates dominated by digits or formatting
  • Merge near-duplicates into longest common span
  • Rank by $a(g)$ (conditional accuracy)
• Lowest-accuracy phrases form the failure set $\mathcal{G}$, which also serves as retrieval queries

Step 2: Retrieval

• Each $g \in \mathcal{G}$ is an exact-match query to an infini-gram index (Liu et al., 2024) of the model's pretraining and mid-training corpora
• Extract 2k-token window around every match
• Sample documents with per-query balanced budget of 35M tokens total

Step 3: Continued Pretraining + RL

• Base model trained on retrieved text with next-token objective:
  • Either on released base model, or (for OLMo2) after stage 1, before mid-training
• GRPO re-run with all hyperparameters held fixed
• Matched control: identical pipeline and budget with content-neutral queries (e.g., "step 1")

Experimental Setup

Models: OLMo2 (1B, 7B), Qwen3 (0.6B, 1.7B, 4B)

Three RL settings (increasing breadth):

1. GSM8K (elementary math)
2. Multi-task mixture: GSM8K : Social IQA : MMLU : TriviaQA at 8:2:1:1
3. General Reasoner: WebInstruct-verified prompts spanning math, science, logic

RL hyperparameters: GRPO with $G=5$ rollouts per prompt for settings (i) and (ii) (210 and 595 steps); General Reasoner recipe with $G=8$ for (iii) (226 steps).

---

Empirical Validation / Results

Failure n-Grams Localize RL Failures

Across all models and settings, a small number of n-grams predicts rollout failure:

Content varies with model strength and task breadth:

• OLMo2-1B/GSM8K: reflection/self-correction phrases (there's a mistake, Let's recheck)
• Qwen3-4B/multi-task: hedging phrases on trivia questions
• Stronger Qwen3/General Reasoner: specific operations (matrix conversion, Coulomb's law)

Main Results: Targeted CPT Repairs RL

‡: OLMo2-1B-mined corpus; ¶: compared at matched RL step

The Deficit Is Local, Not a Coverage Gap

• 46% of affected questions have a correct sibling rollout — the answer is within reach; what's missing is a way through the marked state
• RL does not change entry rate: over 210 GRPO steps, the fraction of rollouts entering failure states stays flat at 5–7%, while success rate from those states rises only in step with overall accuracy
• RL does not expand coverage: pass@32 after RL ends below the base model on MATH-500 (90.6 vs 91.2); targeted CPT raises pass@32 by 1.2–1.6 points

Insertion Point Matters

Token Budget Analysis

Key finding: 35–100M tokens suffice; scaling to 1B hurts (catastrophic forgetting), especially before RL (−19 points for Qwen3-4B).

Missing Skills Are Repaired More Fully Than Unproductive Behaviors

• Operations Qwen3-4B never completed correctly go from 0% → 50–62% success after CPT
• Reflection phrases of OLMo2-1B move by only 1–2 points
• The repair raises how often the model reflects far more than how often reflecting succeeds

---

Theoretical and Practical Implications

Theoretical Implications

1. RL's ceiling is set by the base model: The paper provides direct evidence that RL "sharpens" rather than "expands" — pass@32 does not grow under RL alone, but targeted CPT moves it.

2. Failure localization as a supervision signal: A model's own failures are an underused, label-free signal pointing to the pretraining data it needs most — an alternative to task-similarity or hand-crafted data selection.

3. State-value theory connects rollouts to data selection: The associational measure $a(g)$ bridges observable n-grams and latent state values, making the "where RL breaks" question tractable.

Practical Implications

1. Extreme data efficiency: 35M tokens (vs. hundreds of billions for mid-training) — a 3–4 order-of-magnitude reduction in data requirements.

2. Transferability: A corpus mined from OLMo2-1B's failures improves Qwen3-0.6B (+2.4) and Qwen3-4B (+1.9) — the signal transfers across model families and even across pretraining corpora.

3. Insertion before mid-training is critical: The largest gains (+2.0 on GSM8K) come from inserting CPT before mid-training, which absorbs the perturbation CPT leaves on a finished model.

4. Model- and task-specific failure modes: The content of failure n-grams reveals what kind of capability a model lacks — reflection discipline vs. specific operations — guiding what data to retrieve.

---

Conclusion

Main Takeaways

• Where RL fails is localized and diagnosable: A small set of failure n-grams (reflection phrases, specific operations) strongly predicts rollout failure and persists throughout RL training.

• The deficit is conditional, not coverage: For 46% of affected questions, a correct answer is within the model's reach; what's missing is a path through the marked low-value state.

• A tiny, targeted corpus repairs it: Failure-n-gram-guided CPT on 35M tokens outperforms matched controls in 10 of 11 settings, with the largest gains before mid-training.

• The signal transfers: Corpora mined from one model's failures improve other models, even across model families.

Future Directions

• Scaling the approach to larger models and more diverse reasoning tasks
• Combining failure-n-gram signals with other data selection criteria (e.g., importance resampling, influence functions)
• Exploring whether the retrieved text can be used to shape the RL objective directly (e.g., preference optimization on failure states)
• Investigating why reflection phrases repair less fully than missing operations, and whether different retrieval strategies could close that gap

"A model's own failures are an underused supervision signal: a cheap, label-free pointer to the pretraining data it needs most."