SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning Paradigms for LLMs

Authors: Kejian Zhu, Zhuoran Jin, Shangqing Tu, Hongbang Yuan, Yushi Bai, Kang Liu, Juanzi Li, Jun Zhao

---

Summary (Overview)

• Core Finding: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in multi-task training: SFT suffers from severe task conflicts (catastrophic forgetting) during multi-stage training, while RL enables stable task coexistence with cumulative performance gains.
• Mechanistic Insight: At the parameter level, RL induces sparse, minimal updates with approximately orthogonal directions across tasks (cosine similarity $10^{-5}$), whereas SFT produces dense, large-magnitude updates with high inter-task similarity ($10^{-1}$ to 1.0).
• Theoretical Contribution: The paper proves that gradient interference in SFT is norm-limited (bounded by absolute gradient magnitude $M_i$), while RL interference is variance-limited (bounded by intra-group rollout variance $V_i$), explaining why RL naturally decouples multi-task optimization.
• Practical Application: The authors propose Parallel-RL, a decoupled training paradigm where tasks are trained independently in parallel and updates are merged, achieving performance comparable to or exceeding single-task RL while improving training efficiency and modularity.
• Empirical Validation: Experiments on DeepSeek-R1-Distill-Qwen-1.5B/7B across math, science, logic, and code tasks confirm that multi-stage SFT degrades performance by 23.1% on average, while multi-stage RL improves it by 24.9%.

---

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a critical gap in understanding how SFT and RL behave differently in multi-task learning scenarios for large language models (LLMs). While previous work has examined single-task differences between these paradigms, their multi-task behaviors remain underexplored.

Key Observation: In practice, researchers typically use:

• Mixed-data training for SFT (to mitigate forgetting)
• Multi-stage training for RL (where each stage focuses on a single task)

This divergence in practice suggests an underlying mechanistic difference that the paper seeks to explain.

The Central Phenomenon

The paper identifies two critical empirical findings:

1. Multi-stage SFT collapses (average decline of 23.1% vs. base model), while multi-stage RL improves (average gain of 24.9%).
2. Single-task SFT harms untrained tasks (average decline of 5.1% on other tasks), while single-task RL preserves or improves them (average gain of 2.3%).

This is termed "SFT Conflicts and RL Coexists."

Theoretical Foundation

The paper formalizes the gradient structure of both paradigms. Let $\pi_\theta$ denote the policy parameterized by $\theta$, and $\pi_{\text{expert}}$ be the supervision distribution:

$$\begin{array}{l} g_{\mathrm{SFT}} = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_{\mathrm{expert}}} [ \nabla_{\theta} \log \pi_{\theta} (y | x) ] \\ g_{\mathrm{RL}} = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_{\theta}} [ A (x, y) \nabla_{\theta} \log \pi_{\theta} (y | x) ] \end{array}\tag{1}$$

The two fundamental distinctions are:

1. Policy Source: SFT is off-policy (samples from fixed expert distribution); RL is on-policy (samples from current policy $\pi_\theta$).
2. Advantage Function: RL incorporates a scalar weighting term $A(x,y)$ that reweights gradient contributions.

---

Methodology

Experimental Setup

• Base Models: DeepSeek-R1-Distill-Qwen-1.5B and 7B
• RL Algorithm: GRPO (Group Relative Policy Optimization)
• Efficiency Technique: LoRA for parameter-efficient fine-tuning (in preliminary experiments); full-parameter training for main experiments
• Tasks and Benchmarks:
  • Math: MATH500, AIME2025
  • Science: MMLU, GPQA
  • Logic: Knights & Knaves (KK)
  • Code: LiveCodeBench

Analytical Framework

The paper analyzes parameter update vectors $\Delta W_i$ for each task $T_i$ using two geometric properties:

1. Magnitude ($L_2$ norm)
2. Pairwise cosine similarity between tasks

GRPO Gradient Definition

For input $x$ from task $i$ with $G$ group-wise rollouts $\{y_k\}_{k=1}^G$:

$$g_{i} (x) = \frac{1}{G} \sum_{k=1}^{G} \hat{A}_{i, k} (x) \nabla_{\theta} \log \pi_{\theta} (y_{k} | x),\tag{3}$$

where the advantage is standardized: $\hat{A}_{i,k}(x) = (r_{i,k} - \mu_{r_i}(x)) / \sigma_{r_i}(x)$, with $\mu_{r_i}(x)$ and $\sigma_{r_i}(x)$ being the group mean and standard deviation of rewards.

Key property (zero-sum): \sum_{k=1}^{G} \hat{A}_{i,k}(x) = 0 \tag{4}

Parallel-RL Framework

The proposed paradigm:

1. Launch $N$ parallel RL training processes, one per task
2. Each produces a task-specific update $\Delta W_i$
3. Merge updates: $W_{\text{final}} = W_{\text{base}} + \mathcal{M}(\Delta W_1, \ldots, \Delta W_N)$

Three merging strategies explored:

• Naive Parallel-RL: Sum or mean of updates
• Sparse Parallel: TIES method or SVD rank-1 retention
• Adapted Parallel-RL: Light post-merge adaptation with 5% of original training data

---

Empirical Validation / Results

Parameter-Level Analysis

Observation 1: RL updates are minimal and sparse

• Average $L_2$ norm of $\Delta W$: ~$3 \times 10^{-2}$ for RL vs. $7.4$ for SFT (over two orders of magnitude difference)
• Only ~20% of RL parameters have magnitudes exceeding $10^{-5}$, compared to 93% for SFT

Observation 2: RL optimization directions are approximately orthogonal

• Pairwise cosine similarity between different RL tasks: ~$10^{-5}$
• SFT similarity: on the order of $10^{-1}$ to 1.0, with some updates pointing in opposite directions

Quantitative Score Function Analysis

Main Results (Table 4 Summary)

DeepSeek-R1-Distill-Qwen-1.5B:

Key Results for 7B Model:

• Adapted Parallel-RL achieves +8.0% over base, with 102.4% retention of single-task RL performance
• Naive Parallel-SFT retains only ~67% of single-task SFT performance, demonstrating severe conflicts

Ablation Study

Removing a specific $\Delta W_i$ from the merged model causes:

• Average performance drop on the corresponding task ($\Delta_{\text{Target}}$): 7.1%
• Performance on other tasks remains robust, with slight improvement ($\Delta_{\text{Others}}$): +0.6%

This confirms that Parallel-RL effectively decouples task-specific capabilities.

---

Theoretical and Practical Implications

Theoretical Analysis

The paper builds on RL's Razor (Shenfeld et al., 2025):

Proposition 4.1 (Convergence to KL-Minimal Solution): Under binary rewards, policy gradient converges to:

$$\pi^{\text{updated}} = \arg\min_{\pi \in \mathcal{P}^* \cap \Pi} D_{KL}(\pi \mid\mid \pi_{0}).\tag{2}$$

This means RL implicitly selects the optimal policy closest in KL-divergence to initialization, explaining the sparsity of updates.

Gradient Interference Decomposition

Lemma 4.3 (Gradient Inner Product Decomposition): The RL interference term decomposes as:

$$\mathcal{I}_{RL}(i, j) = \mathbb{E}_{x, x'} \left[ \frac{1}{G^2} \sum_{k=1}^{G} \sum_{l=1}^{G} \hat{A}_{i,k}(x) \hat{A}_{j,l}(x') \langle \delta S_{i,k}(x), \delta S_{j,l}(x') \rangle \right],\tag{7}$$

where $\delta S_{i,k}(x) = S_{i,k}(x) - \bar{S}_i(x)$ is the residual score function.

Key insight: The zero-sum property of advantages removes the mean gradient direction $\bar{S}_i(x)$, transforming interference into a function of intra-group residuals only.

Upper Bound Theorem

Theorem 4.5 (Upper Bound on Gradient Interference):

1. SFT Upper Bound (Norm-Limited):

$$\left| \mathcal{I}_{SFT}(i, j) \right| \leq M_i \cdot M_j.\tag{UB-SFT}$$

2. RL Upper Bound (Variance-Limited):

$$\left| \mathcal{I}_{RL}(i, j) \right| \leq V_i \cdot V_j.\tag{UB-RL}$$

Where $M_i$ bounds the expected norm of expert score functions, and $V_i$ bounds the intra-group variance of residual score functions.

Mechanism Summary

1. Advantage Function as Filter: Removes the dense mean gradient, leaving only intra-group residuals
2. On-Policy Nature: Ensures residuals $\delta S$ remain minimal (limited by intra-group divergence for fixed input and parameters)
3. Concentration of Measure: Independent zero-mean sparse vectors in high-dimensional space $\mathbb{R}^d$ are approximately orthogonal with high probability: $\mathbb{P}(|\langle \delta S_i, \delta S_j \rangle| \geq t) \leq 2\exp(-ct^2 d)$

Practical Implications

• Training Paradigm Shift: Parallel-RL enables modular, composable training where tasks are decoupled
• Efficiency Gains: Tasks can be trained in parallel rather than sequentially, reducing wall-clock time
• Flexibility: Task-specific modules can be added or removed without retraining
• Performance: Adapted Parallel-RL can even surpass single-task RL performance, suggesting beneficial cross-task generalization

---

Conclusion

Main Takeaways

1. SFT Conflicts vs. RL Coexists: SFT's dense, high-magnitude updates cause catastrophic interference across tasks, while RL's sparse, orthogonal updates enable coexistence.

2. Norm-Limited vs. Variance-Limited Interference: The theoretical analysis proves that SFT interference scales with absolute gradient magnitude, while RL interference is bounded by intra-group rollout variance—a fundamental consequence of the advantage function and on-policy optimization.

3. Parallel-RL as a New Paradigm: The orthogonality of RL updates enables a decoupled training paradigm where tasks are trained independently and merged, achieving superior efficiency and modularity.

Future Directions

• The paper suggests Parallel-RL requires further exploration of:
  • Task selection strategies (which tasks can be effectively parallelized)
  • Single-task training tricks that maximize compatibility
  • Extension to other RL algorithms beyond GRPO
  • Scaling to larger models and more diverse task sets

Impact Statement

The work aims to advance multi-task learning for LLMs toward AGI, while emphasizing the importance of maintaining safety and reliability for each individual task during efficient multi-task training. The authors note no distinct ethical risks beyond those inherent in LLM development.