# SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

> Multi-task RL induces sparse, orthogonal parameter updates enabling task coexistence, while SFT's dense, interfering updates cause catastrophic forgetting, motivating the decoupled Parallel-RL paradigm.

- **Source:** [arXiv](https://arxiv.org/abs/2608.03573)
- **Published:** 2026-08-11
- **Permalink:** https://picx.dev/p/eBG0Py
- **Whiteboard:** https://picx.dev/p/eBG0Py/image

## Summary

# SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning Paradigms for LLMs

**Authors:** Kejian Zhu, Zhuoran Jin, Shangqing Tu, Hongbang Yuan, Yushi Bai, Kang Liu, Juanzi Li, Jun Zhao

---

## Summary (Overview)

- **Core Finding:** Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in multi-task training: SFT suffers from severe **task conflicts** (catastrophic forgetting) during multi-stage training, while RL enables stable **task coexistence** with cumulative performance gains.
- **Mechanistic Insight:** At the parameter level, RL induces sparse, minimal updates with approximately orthogonal directions across tasks (cosine similarity ~$10^{-5}$), whereas SFT produces dense, large-magnitude updates with high inter-task similarity (~$10^{-1}$ to 1.0).
- **Theoretical Contribution:** The paper proves that gradient interference in SFT is **norm-limited** (bounded by absolute gradient magnitude $M_i$), while RL interference is **variance-limited** (bounded by intra-group rollout variance $V_i$), explaining why RL naturally decouples multi-task optimization.
- **Practical Application:** The authors propose **Parallel-RL**, a decoupled training paradigm where tasks are trained independently in parallel and updates are merged, achieving performance comparable to or exceeding single-task RL while improving training efficiency and modularity.
- **Empirical Validation:** Experiments on DeepSeek-R1-Distill-Qwen-1.5B/7B across math, science, logic, and code tasks confirm that multi-stage SFT degrades performance by 23.1% on average, while multi-stage RL improves it by 24.9%.

---

## Introduction and Theoretical Foundation

### Background and Motivation

The paper addresses a critical gap in understanding how SFT and RL behave differently in **multi-task learning** scenarios for large language models (LLMs). While previous work has examined single-task differences between these paradigms, their multi-task behaviors remain underexplored.

**Key Observation:** In practice, researchers typically use:
- **Mixed-data training** for SFT (to mitigate forgetting)
- **Multi-stage training** for RL (where each stage focuses on a single task)

This divergence in practice suggests an underlying mechanistic difference that the paper seeks to explain.

### The Central Phenomenon

The paper identifies two critical empirical findings:

1. **Multi-stage SFT collapses** (average decline of 23.1% vs. base model), while **multi-stage RL improves** (average gain of 24.9%).
2. **Single-task SFT harms untrained tasks** (average decline of 5.1% on other tasks), while **single-task RL preserves or improves them** (average gain of 2.3%).

This is termed **"SFT Conflicts and RL Coexists."**

### Theoretical Foundation

The paper formalizes the gradient structure of both paradigms. Let $\pi_\theta$ denote the policy parameterized by $\theta$, and $\pi_{\text{expert}}$ be the supervision distribution:

$$
\begin{array}{l} g_{\mathrm{SFT}} = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_{\mathrm{expert}}} [ \nabla_{\theta} \log \pi_{\theta} (y | x) ] \\ g_{\mathrm{RL}} = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_{\theta}} [ A (x, y) \nabla_{\theta} \log \pi_{\theta} (y | x) ] \end{array}\tag{1}
$$

The two fundamental distinctions are:
1. **Policy Source:** SFT is off-policy (samples from fixed expert distribution); RL is on-policy (samples from current policy $\pi_\theta$).
2. **Advantage Function:** RL incorporates a scalar weighting term $A(x,y)$ that reweights gradient contributions.

---

## Methodology

### Experimental Setup

- **Base Models:** DeepSeek-R1-Distill-Qwen-1.5B and 7B
- **RL Algorithm:** GRPO (Group Relative Policy Optimization)
- **Efficiency Technique:** LoRA for parameter-efficient fine-tuning (in preliminary experiments); full-parameter training for main experiments
- **Tasks and Benchmarks:**
  - Math: MATH500, AIME2025
  - Science: MMLU, GPQA
  - Logic: Knights & Knaves (KK)
  - Code: LiveCodeBench

### Analytical Framework

The paper analyzes parameter update vectors $\Delta W_i$ for each task $T_i$ using two geometric properties:
1. **Magnitude** ($L_2$ norm)
2. **Pairwise cosine similarity** between tasks

### GRPO Gradient Definition

For input $x$ from task $i$ with $G$ group-wise rollouts $\{y_k\}_{k=1}^G$:

$$
g_{i} (x) = \frac{1}{G} \sum_{k=1}^{G} \hat{A}_{i, k} (x) \nabla_{\theta} \log \pi_{\theta} (y_{k} | x),\tag{3}
$$

where the advantage is standardized: $\hat{A}_{i,k}(x) = (r_{i,k} - \mu_{r_i}(x)) / \sigma_{r_i}(x)$, with $\mu_{r_i}(x)$ and $\sigma_{r_i}(x)$ being the group mean and standard deviation of rewards.

**Key property (zero-sum):** $\sum_{k=1}^{G} \hat{A}_{i,k}(x) = 0 \tag{4}$

### Parallel-RL Framework

The proposed paradigm:
1. Launch $N$ parallel RL training processes, one per task
2. Each produces a task-specific update $\Delta W_i$
3. Merge updates: $W_{\text{final}} = W_{\text{base}} + \mathcal{M}(\Delta W_1, \ldots, \Delta W_N)$

Three merging strategies explored:
- **Naive Parallel-RL:** Sum or mean of updates
- **Sparse Parallel:** TIES method or SVD rank-1 retention
- **Adapted Parallel-RL:** Light post-merge adaptation with 5% of original training data

---

## Empirical Validation / Results

### Parameter-Level Analysis

**Observation 1: RL updates are minimal and sparse**
- Average $L_2$ norm of $\Delta W$: ~$3 \times 10^{-2}$ for RL vs. $7.4$ for SFT (over two orders of magnitude difference)
- Only ~20% of RL parameters have magnitudes exceeding $10^{-5}$, compared to 93% for SFT

**Observation 2: RL optimization directions are approximately orthogonal**
- Pairwise cosine similarity between different RL tasks: ~$10^{-5}$
- SFT similarity: on the order of $10^{-1}$ to 1.0, with some updates pointing in opposite directions

### Quantitative Score Function Analysis

| Indicator | SFT | RL |
|-----------|-----|-----|
| $\|S\|_2$ | ~7.1 | ~$10^{-1}$ |
| $\|\delta S\|_2$ | - | ~$10^{-2}$ |
| CosSim($S_i$, $S_j$) | ~$10^{-1}$ | ~$10^{-3}$ |

### Main Results (Table 4 Summary)

**DeepSeek-R1-Distill-Qwen-1.5B:**

| Method | Δ Base (%) | Retention (%) |
|--------|-----------|---------------|
| Single-Task SFT | +5.5 | - |
| Single-Task RL | +9.3 | - |
| Mixed Data SFT | +5.2 | - |
| Multi-Stage SFT | -8.3 | - |
| Mixed Data RL | +8.9 | - |
| Multi-Stage RL | +10.2 | - |
| Naive Parallel-RL (sum) | +6.6 | 94.2 |
| TIES Parallel-RL | +8.0 | 97.4 |
| **Adapted Parallel-RL** | **+10.7** | **103.2** |

**Key Results for 7B Model:**
- Adapted Parallel-RL achieves +8.0% over base, with 102.4% retention of single-task RL performance
- Naive Parallel-SFT retains only ~67% of single-task SFT performance, demonstrating severe conflicts

### Ablation Study

Removing a specific $\Delta W_i$ from the merged model causes:
- Average performance drop on the corresponding task ($\Delta_{\text{Target}}$): **7.1%**
- Performance on other tasks remains robust, with slight improvement ($\Delta_{\text{Others}}$): **+0.6%**

This confirms that Parallel-RL effectively **decouples task-specific capabilities**.

---

## Theoretical and Practical Implications

### Theoretical Analysis

The paper builds on **RL's Razor** (Shenfeld et al., 2025):

**Proposition 4.1 (Convergence to KL-Minimal Solution):** Under binary rewards, policy gradient converges to:

$$
\pi^{\text{updated}} = \arg\min_{\pi \in \mathcal{P}^* \cap \Pi} D_{KL}(\pi \mid\mid \pi_{0}).\tag{2}
$$

This means RL implicitly selects the optimal policy closest in KL-divergence to initialization, explaining the sparsity of updates.

### Gradient Interference Decomposition

**Lemma 4.3 (Gradient Inner Product Decomposition):** The RL interference term decomposes as:

$$
\mathcal{I}_{RL}(i, j) = \mathbb{E}_{x, x'} \left[ \frac{1}{G^2} \sum_{k=1}^{G} \sum_{l=1}^{G} \hat{A}_{i,k}(x) \hat{A}_{j,l}(x') \langle \delta S_{i,k}(x), \delta S_{j,l}(x') \rangle \right],\tag{7}
$$

where $\delta S_{i,k}(x) = S_{i,k}(x) - \bar{S}_i(x)$ is the residual score function.

**Key insight:** The zero-sum property of advantages removes the mean gradient direction $\bar{S}_i(x)$, transforming interference into a function of intra-group residuals only.

### Upper Bound Theorem

**Theorem 4.5 (Upper Bound on Gradient Interference):**

1. **SFT Upper Bound (Norm-Limited):**
$$
\left| \mathcal{I}_{SFT}(i, j) \right| \leq M_i \cdot M_j.\tag{UB-SFT}
$$

2. **RL Upper Bound (Variance-Limited):**
$$
\left| \mathcal{I}_{RL}(i, j) \right| \leq V_i \cdot V_j.\tag{UB-RL}
$$

Where $M_i$ bounds the expected norm of expert score functions, and $V_i$ bounds the intra-group variance of residual score functions.

### Mechanism Summary

1. **Advantage Function as Filter:** Removes the dense mean gradient, leaving only intra-group residuals
2. **On-Policy Nature:** Ensures residuals $\delta S$ remain minimal (limited by intra-group divergence for fixed input and parameters)
3. **Concentration of Measure:** Independent zero-mean sparse vectors in high-dimensional space $\mathbb{R}^d$ are approximately orthogonal with high probability: $\mathbb{P}(|\langle \delta S_i, \delta S_j \rangle| \geq t) \leq 2\exp(-ct^2 d)$

### Practical Implications

- **Training Paradigm Shift:** Parallel-RL enables modular, composable training where tasks are decoupled
- **Efficiency Gains:** Tasks can be trained in parallel rather than sequentially, reducing wall-clock time
- **Flexibility:** Task-specific modules can be added or removed without retraining
- **Performance:** Adapted Parallel-RL can even **surpass** single-task RL performance, suggesting beneficial cross-task generalization

---

## Conclusion

### Main Takeaways

1. **SFT Conflicts vs. RL Coexists:** SFT's dense, high-magnitude updates cause catastrophic interference across tasks, while RL's sparse, orthogonal updates enable coexistence.

2. **Norm-Limited vs. Variance-Limited Interference:** The theoretical analysis proves that SFT interference scales with absolute gradient magnitude, while RL interference is bounded by intra-group rollout variance—a fundamental consequence of the advantage function and on-policy optimization.

3. **Parallel-RL as a New Paradigm:** The orthogonality of RL updates enables a decoupled training paradigm where tasks are trained independently and merged, achieving superior efficiency and modularity.

### Future Directions

- The paper suggests Parallel-RL requires further exploration of:
  - Task selection strategies (which tasks can be effectively parallelized)
  - Single-task training tricks that maximize compatibility
  - Extension to other RL algorithms beyond GRPO
  - Scaling to larger models and more diverse task sets

### Impact Statement

The work aims to advance multi-task learning for LLMs toward AGI, while emphasizing the importance of maintaining safety and reliability for each individual task during efficient multi-task training. The authors note no distinct ethical risks beyond those inherent in LLM development.

---

_Markdown view of https://picx.dev/p/eBG0Py, served by PicX — AI-generated visual whiteboard summaries of research papers._