Summary (Overview)

• Problem Identification: The paper identifies a critical failure mode in Multi-Teacher On-Policy Distillation (MOPD): when domain updates are poorly compatible (e.g., Math × Medical or Code × Medical), domain-routed supervision induces persistent gradient conflicts in shared parameters, causing asymmetric performance degradation—particularly in the disadvantaged domain.

• Proposed Solution: The authors propose Cross-Teacher Guarding for MOPD (ROAD MOPD), which repurposes non-routed teachers as auxiliary cross-domain critics on the same student-generated trajectories, coordinating multi-teacher supervision through token-level credit assignment and gradient-level preventive projection.

• Two Key Mechanisms: (1) Cross-teacher-guided credit assignment uses teacher agreement to strengthen token-level corrections jointly supported by teachers and attenuate conflicting signals; (2) Preventive multi-teacher gradient projection constrains conflicting updates while adaptively preserving target-domain learning signals via a norm-retention floor.

• Empirical Results: ROAD MOPD improves Medical accuracy by 5.6–8.8 percentage points over Vanilla MOPD in high-conflict settings, achieves the best two-domain average in all four paired settings tested, and generalizes to low-conflict (Math × Code) and three-domain (Math × Code × Medical) settings.

• Key Insight: Cross-teacher disagreement empirically correlates with gradient conflict (conflict rates rise from 24%→40% in Math and 46%→76% in Medical across disagreement quintiles), validating the use of auxiliary teacher feedback as a compatibility signal.

---

Introduction and Theoretical Foundation

Background

Reinforcement Learning (RL) has become a dominant paradigm for post-training large language models (LLMs). However, outcome-level rewards lack direct feedback on intermediate reasoning steps, making credit assignment difficult. On-Policy Distillation (OPD) addresses this by using a strong teacher to provide dense token-level supervision on trajectories generated by the student's current policy, combining on-policy sampling with fine-grained teacher feedback.

The OPD Objective

OPD minimizes the sequence-level reverse-KL objective:

$$\mathcal {L} _ {\mathrm{OPD}} (\theta ; \pi_ {T}, \mathcal {D}) = \mathbb {E} _ {x \sim \mathcal {D}} D _ {\mathrm{KL}} (\pi_ {\theta} (\cdot | x) \| \pi_ {T} (\cdot | x)).\tag{1}$$

Following prior work, OPD can be expressed as relative-reward maximization with a KL penalty to a frozen reference $\pi _ { \mathrm { r e f } }$:

$$\mathcal {J} _ {\lambda} (\theta) = \mathbb {E} _ {\theta} \left[ \lambda \log \frac {\pi_ {T} (y \mid x)}{\pi_ {\mathrm{ref}} (y \mid x)} - D _ {\mathrm{KL}} \left(\pi_ {\theta} (\cdot \mid x) \| \pi_ {\mathrm{ref}} (\cdot \mid x)\right) \right].\tag{2}$$

where $\lambda = 1$ recovers standard OPD, $\lambda < 1$ attenuates teacher reward, and $\lambda > 1$ amplifies it (ExOPD).

Multi-Teacher On-Policy Distillation (MOPD)

For K domain–teacher pairs $\{ ( \mathcal { D } _ { d } , \pi _ { T _ { d } } ) \} _ { d = 1 } ^ { K }$, MOPD routes each prompt to its domain teacher:

$$\mathcal {L} _ {\mathrm{MOPD}} (\theta) = \frac {1}{K} \sum_ {d = 1} ^ {K} \mathcal {L} _ {\mathrm{OPD}} (\theta ; \pi_ {T _ {d}}, \mathcal {D} _ {d}).\tag{3}$$

The Identified Failure Mode

The paper demonstrates that while Math × Code combinations work well (few gradient conflicts, preserved/improved performance), Medical pairings with either Math or Code consistently produce:

• High proportions of layers with conflicting gradients (increasing during training)
• Medical performance falling below single-domain OPD baselines

This reveals that domain routing alone is insufficient to coordinate capability transfer through shared parameters.

---

Methodology

3.1 Cross-Teacher-Guided Credit Assignment

The method queries an auxiliary teacher $T_a$ on the same student-generated prefixes and computes token-wise agreement between routed and auxiliary teachers relative to the shared reference:

$$s _ {t} = \underbrace {\tanh \left(\frac {\ell_ {t} ^ {\mathrm{r}} - \ell_ {t} ^ {\mathrm{ref}}}{\tau}\right)} _ {\text {routed - teacher evidence}} \times \underbrace {\tanh \left(\frac {\ell_ {t} ^ {\mathrm{a}} - \ell_ {t} ^ {\mathrm{ref}}}{\tau}\right)} _ {\text {auxiliary - teacher evidence}}, \qquad s _ {t} \in [ - 1, 1 ].\tag{4}$$

• Positive $s_t$: both teachers favor the same correction direction → strengthen supervision
• Negative $s_t$: opposing preferences → attenuate supervision
• Near zero: little evidence beyond reference

This agreement modulates the global reward scale into a token-wise coefficient:

$$\lambda_ {t} = \mathrm{clip} (1 + \rho_ {\mathrm{g}} s _ {t}, \lambda_ {\min}, \lambda_ {\max}).\tag{5}$$

3.2 Preventive Multi-Teacher Gradient Projection

Token-level agreement doesn't guarantee compatible parameter updates. The method formulates gradient correction as a constrained optimization problem:

1. Compute target-domain gradient $g_d$ from the agreement-modulated loss
2. Construct auxiliary gradient $g'_d$ by replacing the routed teacher with the auxiliary teacher (using plain OPD loss, no agreement modulation) on the same trajectories:

$$\mathcal {L} _ {d \to \mathrm{a}} ^ {\prime} = \frac {1}{| \mathcal {B} _ {d} |} \sum_ {i \in \mathcal {B} _ {d}} \frac {1}{T _ {i}} \sum_ {t = 1} ^ {T _ {i}} \ell_ {d \to \mathrm{a}, i, t} ^ {\mathrm{OPD}}, \qquad g _ {d} ^ {\prime} = \nabla_ {\theta} \mathcal {L} _ {d \to \mathrm{a}} ^ {\prime}.\tag{6}$$

3. For conflicting pairs ($\langle g, g' \rangle < 0$), restrict the corrected gradient to the line segment between $g$ and its full projection onto the hyperplane orthogonal to $g'$:

$$g _ {\text { full }} = g - \frac {\langle g , g ^ {\prime} \rangle}{\| g ^ {\prime} \| _ {2} ^ {2}} g ^ {\prime}, \qquad \mathcal {S} (g, g ^ {\prime}) = \operatorname{conv} \{g, g _ {\text { full }} \}.\tag{7}$$

4. Choose the gradient with least residual conflict subject to a norm-retention floor $m \in (0, 1]$:

$$\widetilde {g} = \underset {v \in \mathcal {S} (g, g ^ {\prime})} {\arg \min} - \langle v, g ^ {\prime} \rangle \quad \text { subject to } \quad \| v \| _ {2} ^ {2} \geq m ^ {2} \| g \| _ {2} ^ {2}.\tag{8}$$

The solution is a partial projection:

$$\widetilde {g} = g - \alpha^ {\star} \frac {\langle g , g ^ {\prime} \rangle}{\| g ^ {\prime} \| _ {2} ^ {2}} g ^ {\prime}, \quad \alpha^ {\star} \in [ 0, 1 ].\tag{9}$$

where $\alpha^\star = 1$ (full projection) is used when it satisfies the norm floor; otherwise, as much conflict is removed as the floor permits.

3.3 Overall Objective

The overall objective includes only target-domain losses (auxiliary losses guide only gradient correction):

$$\mathcal {L} _ {\text { overall }} (\theta) = \frac {1}{K} \sum_ {d = 1} ^ {K} \frac {1}{| \mathcal {B} _ {d} |} \sum_ {i \in \mathcal {B} _ {d}} \frac {1}{T _ {i}} \sum_ {t = 1} ^ {T _ {i}} \ell_ {d, i, t} ^ {\text { agr }} (\theta).\tag{10}$$

The final update averages only projected target-domain gradients:

$$g _ {\mathrm{update}} = \frac {1}{K} \sum_ {d = 1} ^ {K} \widetilde {g} _ {d}.\tag{11}$$

---

Empirical Validation / Results

Experimental Setup

• Students: Qwen3-4B and Qwen3-1.7B (thinking disabled)
• Teachers: Specialized Qwen3-4B checkpoints for Math/Code; II-Medical-8B for Medical
• Benchmarks: 11 total across Code (HumanEval+, MBPP+, LiveCodeBench), Math (AIME 2024/2025, HMMT Feb/Nov 2025), Medical (MedQA, MedXpertQA, MedMCQA, SuperGPQA-Med)
• Baselines: Vanilla MOPD, ExOPD, CaMOPD, Open-MOPD

Main Results (Paired Domains)

Table 1: Math × Medical with Qwen3-1.7B student

Table 1: Code × Medical with Qwen3-4B student

Key findings:

• ROAD MOPD achieves the highest two-domain average in all four paired settings
• With 1.7B students, Medical gains over strongest baselines increase to 3.88–5.17 points
• Medical recovery accompanies gains in the other domain (no trade-off)

Low-Conflict and Three-Domain Generalization

Table 2: Math × Code (low-conflict), Qwen3-4B

Table 3: Math × Code × Medical (three-domain), Qwen3-4B

Ablation Study

Table 4: Ablations of ROAD MOPD

Each component contributes to overall performance; removing any one lowers the two-domain mean.

Comparison with PCGrad

ROAD MOPD's preventive projection outperforms PCGrad (post-hoc correction) across all four domain-average accuracies by 0.18–2.12 percentage points. The key difference: PCGrad compares independent domain gradients, while ROAD MOPD constrains target-domain updates using auxiliary-teacher gradients on the same trajectory, addressing local conflicts that persist even when independent gradients align.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Update compatibility as a central challenge: The paper reframes MOPD from a supervision-allocation problem to an update-coordination problem, showing that domain routing alone is insufficient when domain updates are poorly compatible.

2. Cross-teacher agreement as a compatibility signal: The empirical correlation between teacher disagreement and gradient conflict (24%→40% in Math, 46%→76% in Medical) provides a principled basis for using auxiliary teacher feedback as a local compatibility indicator.

3. Preventive vs. post-hoc gradient correction: The paper distinguishes its preventive approach (using same-trajectory auxiliary gradients as constraints) from PCGrad's post-hoc surgery on independently computed gradients, showing that same-trajectory constraints address conflicts that independent gradient comparisons miss.

Practical Implications

• Domain compatibility matters: The contrast between Math × Code (compatible) and Medical pairings (incompatible) highlights that joint-learning benefits do not generalize across domain combinations—practitioners must consider update compatibility, not just teacher expertise.

• Auxiliary teachers as free critics: Non-routed teachers can be repurposed as cross-domain critics on the same trajectories without additional rollouts, providing valuable compatibility signals at modest computational cost.

• Adaptive norm retention: The norm-retention floor balances conflict reduction with target-domain learning, preventing full projection from erasing useful learning signals when gradients strongly oppose.

---

Conclusion

Main Takeaways

1. Identified failure mode: Domain-routed MOPD can induce persistent gradient conflicts in shared parameters when domain updates are poorly compatible, causing asymmetric performance degradation.

2. ROAD MOPD solution: Using non-routed teachers as cross-domain critics on the same student trajectories, with agreement-guided token weighting and preventive gradient projection under a norm-retention constraint, effectively mitigates update interference while preserving target-domain learning.

3. Empirical validation: ROAD MOPD improves adversely affected domains under high conflict (5.6–8.8 points over Vanilla MOPD), preserves low-conflict gains, and generalizes to three domains, outperforming standard MOPD and gradient-conflict correction baselines.

Limitations and Future Directions

• Evaluation is limited to three text-based reasoning domains (Math, Code, Medical)
• Auxiliary scoring and gradient computation add training cost
• Extending to broader domains and larger teacher pools calls for:
  • Efficient teacher selection strategies
  • Improved compatibility estimation methods