Summary (Overview)

• This paper challenges the prevailing notion that "RL generalizes, whereas SFT memorizes" by demonstrating that this conclusion depends critically on the properties of the SFT training dataset—specifically its source and scale.
• The authors show that data source matters: SFT data generated by a model closer to the base model being fine-tuned (RL-tuned model) yields better generalization than data from a stronger teacher model (Qwen3-14B), and an appropriate mixture of both sources performs best.
• Data scale is crucial: When the SFT training dataset size is matched to RL's rollout budget (including both correct and incorrect rollouts), SFT achieves generalization comparable to—or even better than—RL.
• Using LLM annotations to characterize solution methods, the authors demonstrate that larger datasets cover more tail methods (rare reasoning patterns), which provide generalizable signals for out-of-distribution reasoning tasks.
• Theoretical analysis of shallow transformer training dynamics confirms these findings: with sufficient method coverage through data scaling, SFT can match RL's OOD generalization.

---

Introduction and Theoretical Foundation

Background and Motivation

Large Language Models (LLMs) trained via Reinforcement Learning (RL) or Supervised Fine-Tuning (SFT) have achieved remarkable improvements in reasoning ability, exemplified by DeepSeek-R1 and s1. Prior work [3–5] has suggested that "RL generalizes, whereas SFT memorizes," indicating that SFT often struggles to match RL on reasoning tasks beyond the training domain.

However, the authors argue that this conclusion may be premature because previous comparisons did not give SFT comparable data resources to RL. Specifically, RL typically generates multiple rollouts per prompt (e.g., 8 in GRPO), while SFT often trains on only one correct rollout per prompt. This paper revisits the question through two key inquiries:

1. What kind of data source best unlocks the generalization potential of SFT?
2. How should we match the data consumed by SFT and RL to compare their generalization fairly?

Theoretical Foundation: RLVR and SFT Objectives

Reinforcement Learning with Verifiable Rewards (RLVR): The regularized RLVR objective for model parameter $\theta$ is:

$$J_{\mathrm{RL}}(\theta) = \mathbb{E}_{X \sim \mathcal{D}_{\mathrm{RL}}, Y \sim \mathbb{P}_{\theta}(\cdot | X)}[r(X, Y)] - \beta \mathbb{E}_{X \sim \mathcal{D}_{\mathrm{RL}}, Y \sim \mathbb{P}_{\theta}(\cdot | X)}\left[\log\left(\mathbb{P}_{\theta}(Y \mid X) / \mathbb{P}_{\mathrm{ref}}(Y \mid X)\right)\right]$$

where $r(X, Y)$ evaluates the correctness of response $Y$ to question $X$, and the second term regularizes the model to remain close to the reference model $\mathbb{P}_{\mathrm{ref}}$.

In practice, Group Relative Policy Optimization (GRPO) maximizes the surrogate objective:

$$J_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|Y_i|}\sum_{t=1}^{|Y_i|}\left(\min\left(w_{i,t}(\theta)A_i, \operatorname{clip}_{\varepsilon}(w_{i,t}(\theta)A_i\right) - \beta \mathbf{KL}(\mathbb{P}_{\theta}\|\mathbb{P}_{\mathrm{ref}})\right)\right]$$

where $w_{i,t}(\theta)$ is the importance-sampling ratio, $\operatorname{clip}_{\varepsilon}(x)$ clips $x$ to $[1-\varepsilon, 1+\varepsilon]$, and $A_i$ is the group-normalized advantage.

SFT Objective: The long-reasoning SFT loss is:

$$L_{\mathrm{SFT}}(\theta) = -\mathbb{E}_{(X,Y) \sim \mathcal{D}_{\mathrm{SFT}}}\left[\sum_{t=1}^{T}\log \mathbb{P}_{\theta}(Y_t \mid X, Y_{<t})\right]$$

---

Methodology

Experimental Setup

• Base models: Qwen2.5-7B and Qwen3-8B-Base
• Training data: Mathematical problems from the DAPO dataset
• RL training: GRPO implemented in VeRL with standard setup
• SFT training: LlamaFactory with rollouts generated by teacher models
• Evaluation:
  • Math reasoning: AIME24, Minerva Math, GSM8K, MATH500, OlympiadBench
  • General reasoning: 12 tasks from Language Model Evaluation Harness (GPQA-Diamond, MMLU-Pro, etc.)

Data Sources for SFT

Two primary data sources were constructed via rejection sampling:

1. Qwen3-14B dataset (size: 15,607): Generated by a strong teacher model (Qwen3-14B), sampling at most 128 rollouts per question and retaining one correct rollout.
2. RL-tuned dataset (size: 14,512): Generated by the RL-tuned model (derived from the base model), expected to be more aligned with the base model parameters.

Mixture datasets (Mix-k) were created by interpolating between these sources with ratios $(11-2k)\times 10\%$ RL-tuned data and $(2k-1)\times 10\%$ Qwen3-14B data for $k \in \{1,\ldots,5\}$.

Data Scale Matching Criteria

Three matching criteria were used to scale SFT datasets relative to RL:

Method Annotation

Qwen3-32B was used to annotate solution methods in training rollouts, identifying 138 unique methods for solving math problems. Tail methods were defined as those in the bottom 20% by frequency.

---

Empirical Validation / Results

Observation 1: Data Source Matters

Table 1 shows key results (selected columns):

Key findings:

• Both SFT variants achieve comparable math performance to RL, but lag on general reasoning.
• SFT with RL-tuned data outperforms SFT with Qwen3-14B data on general reasoning.
• Mix-2 (70% RL-tuned + 30% Qwen3-14B) achieves the best overall performance among SFT models.

Observation 1: The training data source is crucial for generalization: data from a model closer to the one being fine-tuned can be more effective than data from a stronger model, while data from a single model is not necessarily optimal; incorporating an appropriate amount of more diverse data can further improve general reasoning ability.

Observation 2: Data Scale Matters

Scaling the SFT dataset under different matching criteria revealed:

• Solvable-prompt matching → comparable math but better general reasoning than prompt matching
• Correct-rollout matching → substantially improves SFT generalization, approaching RL performance
• Total-rollout matching (including incorrect rollouts) → SFT achieves comparable or better performance than RL on both math and general reasoning

Observation 2: The amount of training data substantially influences SFT generalization: matching the number of correct rollouts greatly improves SFT generalization, while further matching the total rollout budget enables SFT to achieve comparable generalization to RL.

Observation 3: Compounding Effects

Combining optimal data source (90% RL-tuned + 10% Qwen3-14B) with matched rollout budget (138,000 pairs) yields SFT performance better than RL on general reasoning tasks.

Observation 3: With a carefully mixed dataset and a rollout budget matched to RL, SFT achieves promisingly better general reasoning ability than RL, undermining "RL generalizes, SFT memorizes."

Observation 4: Tail Methods Drive Generalization

Method distribution analysis (Figure 10) revealed:

• The method distribution is long-tailed: many methods appear only infrequently.
• Larger datasets cover more tail methods (methods in the bottom 20% by frequency).
• Tail methods are used more frequently and solve more questions on general reasoning tasks (MMLU-Pro, MMLU-Pro+) as training data scale increases.

Observation 4: One benefit of increasing the training samples is improved coverage of long-tailed patterns. Training on these rare but informative samples can enhance performance on downstream general reasoning tasks.

---

Theoretical and Practical Implications

Theoretical Analysis: Shallow Transformer Case Study

The authors provide theoretical grounding using a simplified data model with a one-layer attention transformer:

$$F(P_{\mathrm{in}}, X_q; W) = \mathsf{soft}(\mathsf{soft}(X_q W P_{\mathrm{in}}^{\top}) P_{\mathrm{in}} W_{\mathrm{lm}}) \in \mathbb{R}^{1 \times (LK+1)}$$

Assumption 4.1 (Long-Tailed Distribution in SFT Data): For each question $l$, $p_{l,1} = \Theta(1)$ (head answer) and $p_{l,k} = o(1)$ for $k \neq 1$ (tail answers).

Theorem 4.2 (In-Distribution Convergence): Both SFT and RL achieve high accuracy on in-distribution problems after one gradient step, with wrong-answer probability bounded by $\epsilon = \log(K)/K$. RL additionally learns a balanced distribution across all correct answers: $|F_{(l-1)K+k} - 1/K| \le \epsilon/K$.

Theorem 4.3 (OOD Generalization):

• RL always generalizes: For any OOD mixture, $1 - \mathrm{Eval_{RL}} = O(K^{-3})$.
• SFT generalizes on covered methods: If OOD distribution is supported on answers with $p_{l,k} \ge 1/\log K$, then $1 - \mathrm{Eval_{SFT}} = O(K^{-3})$. However, if supported on answers with $p_{l,k} \le 1/K$, then $\mathrm{Eval_{SFT}} \le 1/2$.

Practical Implications

1. Data curation for SFT: Practitioners should prioritize data from models architecturally or behaviorally close to the base model, while incorporating diverse data to cover rare reasoning patterns.
2. Fair comparison protocols: When benchmarking SFT vs. RL, the total rollout budget (including incorrect rollouts) should be matched, as incorrect rollouts contain valuable intermediate reasoning steps.
3. Scaling strategy: Increasing SFT dataset size improves tail-method coverage, which directly translates to better OOD generalization.

---

Conclusion

This work rethinks the claim that "RL generalizes, whereas SFT memorizes" by showing that this conclusion requires specific prerequisites on SFT training data. The key findings are:

1. Data source is critical: data from a model closer to the base model, mixed with appropriate amounts of diverse data, unlocks SFT's generalization potential.
2. Data scale matters: matching SFT's rollout budget to RL's (including incorrect rollouts) enables SFT to generalize comparably to RL.
3. Tail-method coverage is the mechanism: larger datasets cover more rare reasoning patterns that drive OOD generalization.
4. Theoretical validation: Training dynamics analysis confirms that sufficient method coverage allows SFT to match RL's OOD generalization.

Limitation: The analysis focuses on LLMs; extending to other generative models (e.g., diffusion models) is left for future work.

Future directions include exploring optimal data mixing strategies, understanding the role of incorrect rollouts more deeply, and extending the theoretical framework to deeper architectures and other generative model families.