Self-Distilled RLVR: Summary

Summary (Overview)

• Identifies a fundamental flaw in On-Policy Self-Distillation (OPSD): The information asymmetry between a teacher (with privileged information) and a student (without it) creates an irreducible mutual information gap $I(Y_t; R | X, Y_{<t}) > 0$ in the distribution-matching objective. This leads to privileged information leakage and eventual performance degradation.
• Proposes RLSD (Reinforcement Learning with Self-Distillation): A new paradigm that repurposes self-distillation from a distribution-matching target to a token-level credit assignment mechanism. The environment reward determines the direction of updates (reinforce/penalize), while the privileged teacher's evidence ratio $P_T(y_t)/P_S(y_t)$ modulates the update magnitude per token.
• Achieves superior performance and stability: RLSD unifies the reliable direction of RLVR (e.g., GRPO) with the dense, fine-grained signals of self-distillation. It achieves higher convergence ceilings, faster training, and avoids the leakage and collapse seen in OPSD, as validated on multimodal reasoning benchmarks.

Introduction and Theoretical Foundation

Reinforcement Learning with Verifiable Rewards (RLVR) methods like GRPO train models using sparse, sequence-level rewards (e.g., answer correctness). On-Policy Distillation (OPD) complements this by using a stronger external teacher model to provide dense, token-level supervision, but incurs high computational cost. On-Policy Self-Distillation (OPSD) emerged as an efficient alternative, where a single model acts as both teacher (conditioned on privileged information $r$, like a reference answer) and student (conditioned only on the query $x$).

However, this paper demonstrates that OPSD suffers from systematic privileged information leakage—the model begins to reference invisible "reference solutions" during inference—and unstable long-term training, where performance peaks early then degrades.

The theoretical foundation explains this failure. In OPD, teacher and student are information-symmetric (same input). In OPSD, they are information-asymmetric: the teacher conditions on $r$, which the student cannot observe. This makes the distribution-matching objective ill-posed.

Theorem 1 (KL Decomposition) formalizes the problem. The OPSD objective $L_{OPSD}$ and the ideal objective $L^*$ (matching the teacher's marginal distribution) are related by:

$$L_{OPSD} = L^* + I(Y_t; R | X, Y_{<t})$$

where $I(Y_t; R | X, Y_{<t})$ is the conditional mutual information between the current token $Y_t$ and the privileged information $R$. This term is strictly positive and independent of the student's parameters $\theta$, creating an irreducible optimization gap that drives leakage.

Methodology

The core insight is to decouple the roles of the environment reward and the teacher signal:

• Update Direction: Must be reliable and determined by the verifiable environment reward.
• Update Magnitude: Benefits from being dense and fine-grained, provided by the teacher.

RLSD (Algorithm 1) implements this as follows:

1. On-Policy Rollout & Sequence-Level Advantage: For a query $x$, sample $G$ responses $\{y^{(1)}, ..., y^{(G)}\}$ from the student policy $\pi_\theta(\cdot|x)$. A verifier provides a binary reward $R(x, y^{(i)}) \in \{0, 1\}$. Compute a group-relative advantage for each response:

   $$A^{(i)} = \frac{R(x, y^{(i)}) - \mu_G}{\sigma_G}$$

   where $\mu_G, \sigma_G$ are the mean and standard deviation of rewards in the group.

2. Token-Level Credit Assignment via Self-Distillation: For each token $y_t^{(i)}$ in a trajectory:

   • Compute the privileged information gain: $$\Delta_t = \text{sg}(\log P_T(y_t^{(i)}) - \log P_S(y_t^{(i)}))$$ where $\text{sg}$ is the stop-gradient operator, $P_S(\cdot) = \pi_\theta(\cdot|x, y_{<t})$, and $P_T(\cdot) = \pi_\theta(\cdot|x, r, y_{<t})$.
   • Construct a direction-aware evidence weight: $$w_t = \exp(\text{sign}(A^{(i)}) \cdot \Delta_t) = \left( \frac{P_T(y_t^{(i)})}{P_S(y_t^{(i)})} \right)^{\text{sign}(A^{(i)})}$$ This weight has a Bayesian interpretation as the belief update ratio $P(r | x, y_{\le t}) / P(r | x, y_{<t})$.
   • Apply clipping for stability and interpolate with a uniform baseline: $$\hat{A}_t^{(i)} = A^{(i)} \cdot \left( (1 - \lambda) + \lambda \cdot \text{clip}(w_t, 1 - \epsilon_w, 1 + \epsilon_w) \right)$$ The mixing coefficient $\lambda$ decays from 0.5 to 0 over early training.

3. Policy Update: Update the parameters $\theta$ by maximizing the RLSD objective:

   $$L_{RLSD}(\theta) = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|y^{(i)}|} \sum_{t=1}^{|y^{(i)}|} \min\left( w_t A^{(i)}, \text{clip}(w_t, 1-\epsilon_w, 1+\epsilon_w) A^{(i)} \right) \right]$$

RLSD requires only one extra forward pass (for the teacher logits) and serves as a drop-in replacement for the uniform advantage in GRPO.

Empirical Validation / Results

Experiments were conducted on the Qwen3-VL-8B-Instruct model, trained on the challenging MMFineReason-123K dataset and evaluated on five multimodal reasoning benchmarks.

Table 2: Multimodal reasoning results on the Qwen3-VL-8B-Instruct model.

Key Findings:

1. RLSD achieves the highest average accuracy (56.18%), outperforming the base LLM by +4.69% and GRPO by +2.32%.
2. Training Dynamics (Figure 5): RLSD shows a steeper initial ascent and higher final reward than GRPO, while avoiding OPSD's late-stage collapse. It also maintains higher policy entropy than GRPO, indicating less uniform token suppression.
3. Case Study (Figure 6): Token-level credit heatmaps show RLSD successfully assigns larger credit/blame to decisive reasoning steps (e.g., key calculations) and down-weights generic narration or neutral setup tokens.

Theoretical and Practical Implications

• Theoretical: Provides a formal analysis of why information-asymmetric distribution matching (OPSD) fails, framing it as an ill -posed objective with an irreducible mutual information gap. Introduces the "impossibility trilemma" for shared-parameter self-distillation: objective stability, sustained improvement, and leakage-free training cannot all be achieved simultaneously under distribution matching. RLSD resolves this trilemma.
• Practical: RLSD offers a computationally efficient, drop-in improvement for existing RLVR pipelines like GRPO. It requires only the final ground-truth answer as privileged information (no reasoning traces), provides fine-grained credit assignment without training auxiliary models, and ensures training stability anchored to environmental feedback.

Conclusion

This work diagnoses the fundamental limitations of on-policy self-distillation, proving that information asymmetry leads to an ill-posed objective and inevitable leakage. The proposed RLSD paradigm circumvents these issues by repurposing self-distillation from a generative target to a credit assignment modulator. By anchoring update directions to the environment reward and using the teacher's evidence ratio to control token-level magnitudes, RLSD unifies the strengths of RLVR and self-distillation, achieving higher performance, faster convergence, and robust training stability. Future work will explore RLSD's applicability to broader domains beyond multimodal reasoning.