# Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation

> ROAD MOPD coordinates multi-teacher on-policy distillation by using non-routed teachers as cross-domain critics, mitigating gradient conflicts from incompatible domains and improving disadvantaged-domain performance.

- **Source:** [arXiv](https://arxiv.org/abs/2610.13521)
- **Published:** 2026-10-16
- **Permalink:** https://picx.dev/p/cEv7ty
- **Whiteboard:** https://picx.dev/p/cEv7ty/image

## Summary

## Summary (Overview)

- **Problem Identification**: The paper identifies a critical failure mode in Multi-Teacher On-Policy Distillation (MOPD): when domain updates are poorly compatible (e.g., Math × Medical or Code × Medical), domain-routed supervision induces persistent gradient conflicts in shared parameters, causing asymmetric performance degradation—particularly in the disadvantaged domain.

- **Proposed Solution**: The authors propose **Cross-Teacher Guarding for MOPD (ROAD MOPD)**, which repurposes non-routed teachers as auxiliary cross-domain critics on the same student-generated trajectories, coordinating multi-teacher supervision through token-level credit assignment and gradient-level preventive projection.

- **Two Key Mechanisms**: (1) **Cross-teacher-guided credit assignment** uses teacher agreement to strengthen token-level corrections jointly supported by teachers and attenuate conflicting signals; (2) **Preventive multi-teacher gradient projection** constrains conflicting updates while adaptively preserving target-domain learning signals via a norm-retention floor.

- **Empirical Results**: ROAD MOPD improves Medical accuracy by 5.6–8.8 percentage points over Vanilla MOPD in high-conflict settings, achieves the best two-domain average in all four paired settings tested, and generalizes to low-conflict (Math × Code) and three-domain (Math × Code × Medical) settings.

- **Key Insight**: Cross-teacher disagreement empirically correlates with gradient conflict (conflict rates rise from 24%→40% in Math and 46%→76% in Medical across disagreement quintiles), validating the use of auxiliary teacher feedback as a compatibility signal.

---

## Introduction and Theoretical Foundation

### Background

Reinforcement Learning (RL) has become a dominant paradigm for post-training large language models (LLMs). However, outcome-level rewards lack direct feedback on intermediate reasoning steps, making credit assignment difficult. **On-Policy Distillation (OPD)** addresses this by using a strong teacher to provide dense token-level supervision on trajectories generated by the student's current policy, combining on-policy sampling with fine-grained teacher feedback.

### The OPD Objective

OPD minimizes the sequence-level reverse-KL objective:

$$
\mathcal {L} _ {\mathrm{OPD}} (\theta ; \pi_ {T}, \mathcal {D}) = \mathbb {E} _ {x \sim \mathcal {D}} D _ {\mathrm{KL}} (\pi_ {\theta} (\cdot | x) \| \pi_ {T} (\cdot | x)).\tag{1}
$$

Following prior work, OPD can be expressed as relative-reward maximization with a KL penalty to a frozen reference $\pi _ { \mathrm { r e f } }$:

$$
\mathcal {J} _ {\lambda} (\theta) = \mathbb {E} _ {\theta} \left[ \lambda \log \frac {\pi_ {T} (y \mid x)}{\pi_ {\mathrm{ref}} (y \mid x)} - D _ {\mathrm{KL}} \left(\pi_ {\theta} (\cdot \mid x) \| \pi_ {\mathrm{ref}} (\cdot \mid x)\right) \right].\tag{2}
$$

where $\lambda = 1$ recovers standard OPD, $\lambda < 1$ attenuates teacher reward, and $\lambda > 1$ amplifies it (ExOPD).

### Multi-Teacher On-Policy Distillation (MOPD)

For K domain–teacher pairs $\{ ( \mathcal { D } _ { d } , \pi _ { T _ { d } } ) \} _ { d = 1 } ^ { K }$, MOPD routes each prompt to its domain teacher:

$$
\mathcal {L} _ {\mathrm{MOPD}} (\theta) = \frac {1}{K} \sum_ {d = 1} ^ {K} \mathcal {L} _ {\mathrm{OPD}} (\theta ; \pi_ {T _ {d}}, \mathcal {D} _ {d}).\tag{3}
$$

### The Identified Failure Mode

The paper demonstrates that while Math × Code combinations work well (few gradient conflicts, preserved/improved performance), Medical pairings with either Math or Code consistently produce:
- High proportions of layers with conflicting gradients (increasing during training)
- Medical performance falling **below** single-domain OPD baselines

This reveals that **domain routing alone is insufficient** to coordinate capability transfer through shared parameters.

---

## Methodology

### 3.1 Cross-Teacher-Guided Credit Assignment

The method queries an auxiliary teacher $T_a$ on the same student-generated prefixes and computes token-wise agreement between routed and auxiliary teachers relative to the shared reference:

$$
s _ {t} = \underbrace {\tanh \left(\frac {\ell_ {t} ^ {\mathrm{r}} - \ell_ {t} ^ {\mathrm{ref}}}{\tau}\right)} _ {\text {routed - teacher evidence}} \times \underbrace {\tanh \left(\frac {\ell_ {t} ^ {\mathrm{a}} - \ell_ {t} ^ {\mathrm{ref}}}{\tau}\right)} _ {\text {auxiliary - teacher evidence}}, \qquad s _ {t} \in [ - 1, 1 ].\tag{4}
$$

- **Positive $s_t$**: both teachers favor the same correction direction → strengthen supervision
- **Negative $s_t$**: opposing preferences → attenuate supervision
- **Near zero**: little evidence beyond reference

This agreement modulates the global reward scale into a token-wise coefficient:

$$
\lambda_ {t} = \mathrm{clip} (1 + \rho_ {\mathrm{g}} s _ {t}, \lambda_ {\min}, \lambda_ {\max}).\tag{5}
$$

### 3.2 Preventive Multi-Teacher Gradient Projection

Token-level agreement doesn't guarantee compatible parameter updates. The method formulates gradient correction as a constrained optimization problem:

1. **Compute target-domain gradient** $g_d$ from the agreement-modulated loss
2. **Construct auxiliary gradient** $g'_d$ by replacing the routed teacher with the auxiliary teacher (using plain OPD loss, no agreement modulation) on the same trajectories:

$$
\mathcal {L} _ {d \to \mathrm{a}} ^ {\prime} = \frac {1}{| \mathcal {B} _ {d} |} \sum_ {i \in \mathcal {B} _ {d}} \frac {1}{T _ {i}} \sum_ {t = 1} ^ {T _ {i}} \ell_ {d \to \mathrm{a}, i, t} ^ {\mathrm{OPD}}, \qquad g _ {d} ^ {\prime} = \nabla_ {\theta} \mathcal {L} _ {d \to \mathrm{a}} ^ {\prime}.\tag{6}
$$

3. **For conflicting pairs** ($\langle g, g' \rangle < 0$), restrict the corrected gradient to the line segment between $g$ and its full projection onto the hyperplane orthogonal to $g'$:

$$
g _ {\text { full }} = g - \frac {\langle g , g ^ {\prime} \rangle}{\| g ^ {\prime} \| _ {2} ^ {2}} g ^ {\prime}, \qquad \mathcal {S} (g, g ^ {\prime}) = \operatorname{conv} \{g, g _ {\text { full }} \}.\tag{7}
$$

4. **Choose the gradient** with least residual conflict subject to a norm-retention floor $m \in (0, 1]$:

$$
\widetilde {g} = \underset {v \in \mathcal {S} (g, g ^ {\prime})} {\arg \min} - \langle v, g ^ {\prime} \rangle \quad \text { subject   to } \quad \| v \| _ {2} ^ {2} \geq m ^ {2} \| g \| _ {2} ^ {2}.\tag{8}
$$

The solution is a **partial projection**:

$$
\widetilde {g} = g - \alpha^ {\star} \frac {\langle g , g ^ {\prime} \rangle}{\| g ^ {\prime} \| _ {2} ^ {2}} g ^ {\prime}, \quad \alpha^ {\star} \in [ 0, 1 ].\tag{9}
$$

where $\alpha^\star = 1$ (full projection) is used when it satisfies the norm floor; otherwise, as much conflict is removed as the floor permits.

### 3.3 Overall Objective

The overall objective includes only target-domain losses (auxiliary losses guide only gradient correction):

$$
\mathcal {L} _ {\text { overall }} (\theta) = \frac {1}{K} \sum_ {d = 1} ^ {K} \frac {1}{| \mathcal {B} _ {d} |} \sum_ {i \in \mathcal {B} _ {d}} \frac {1}{T _ {i}} \sum_ {t = 1} ^ {T _ {i}} \ell_ {d, i, t} ^ {\text { agr }} (\theta).\tag{10}
$$

The final update averages only projected target-domain gradients:

$$
g _ {\mathrm{update}} = \frac {1}{K} \sum_ {d = 1} ^ {K} \widetilde {g} _ {d}.\tag{11}
$$

---

## Empirical Validation / Results

### Experimental Setup

- **Students**: Qwen3-4B and Qwen3-1.7B (thinking disabled)
- **Teachers**: Specialized Qwen3-4B checkpoints for Math/Code; II-Medical-8B for Medical
- **Benchmarks**: 11 total across Code (HumanEval+, MBPP+, LiveCodeBench), Math (AIME 2024/2025, HMMT Feb/Nov 2025), Medical (MedQA, MedXpertQA, MedMCQA, SuperGPQA-Med)
- **Baselines**: Vanilla MOPD, ExOPD, CaMOPD, Open-MOPD

### Main Results (Paired Domains)

**Table 1: Math × Medical with Qwen3-1.7B student**

| Method | Math Avg. | Medical Avg. | All Avg. |
|--------|-----------|-------------|----------|
| Single OPD | 44.27 | 47.90 | 46.09 |
| Vanilla MOPD | 45.97 | 39.45 | 42.71 |
| ExOPD | 47.19 | 43.87 | 45.53 |
| Open-MOPD | 45.34 | 44.52 | 44.93 |
| CaMOPD | 46.64 | 47.75 | 47.20 |
| **Ours** | **47.60** | **48.23** | **47.92** |

**Table 1: Code × Medical with Qwen3-4B student**

| Method | Code Avg. | Medical Avg. | All Avg. |
|--------|-----------|-------------|----------|
| Single OPD | 60.91 | 47.90 | 54.41 |
| Vanilla MOPD | 60.53 | 40.75 | 50.64 |
| ExOPD | 61.65 | 44.93 | 53.29 |
| Open-MOPD | 60.83 | 47.34 | 54.09 |
| CaMOPD | 60.64 | 47.28 | 53.96 |
| **Ours** | **63.02** | **47.75** | **55.38** |

**Key findings**:
- ROAD MOPD achieves the highest two-domain average in **all four paired settings**
- With 1.7B students, Medical gains over strongest baselines increase to 3.88–5.17 points
- Medical recovery accompanies gains in the other domain (no trade-off)

### Low-Conflict and Three-Domain Generalization

**Table 2: Math × Code (low-conflict), Qwen3-4B**

| Method | Math | Code | All |
|--------|------|------|-----|
| ExOPD | **48.28** | 61.82 | 55.05 |
| **Ours** | 47.73 | **63.07** | **55.40** |

**Table 3: Math × Code × Medical (three-domain), Qwen3-4B**

| Method | Math | Code | Medical | All |
|--------|------|------|---------|-----|
| CaMOPD | 45.96 | 60.97 | 48.20 | 51.71 |
| **Ours** | **47.29** | **62.94** | **48.45** | **52.89** |

### Ablation Study

**Table 4: Ablations of ROAD MOPD**

| Setting | w/o Credit Assign. | w/o Grad. Protect. | w/o Adaptive Ret. | Full ROAD MOPD |
|---------|-------------------|-------------------|-------------------|----------------|
| Math × Med. Avg. | 46.76 | 47.65 | 47.09 | **47.92** |
| Code × Med. Avg. | 53.50 | 54.94 | 54.94 | **55.38** |

Each component contributes to overall performance; removing any one lowers the two-domain mean.

### Comparison with PCGrad

ROAD MOPD's preventive projection outperforms PCGrad (post-hoc correction) across all four domain-average accuracies by 0.18–2.12 percentage points. The key difference: PCGrad compares independent domain gradients, while ROAD MOPD constrains target-domain updates using auxiliary-teacher gradients on the **same trajectory**, addressing local conflicts that persist even when independent gradients align.

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Update compatibility as a central challenge**: The paper reframes MOPD from a supervision-allocation problem to an update-coordination problem, showing that domain routing alone is insufficient when domain updates are poorly compatible.

2. **Cross-teacher agreement as a compatibility signal**: The empirical correlation between teacher disagreement and gradient conflict (24%→40% in Math, 46%→76% in Medical) provides a principled basis for using auxiliary teacher feedback as a local compatibility indicator.

3. **Preventive vs. post-hoc gradient correction**: The paper distinguishes its preventive approach (using same-trajectory auxiliary gradients as constraints) from PCGrad's post-hoc surgery on independently computed gradients, showing that same-trajectory constraints address conflicts that independent gradient comparisons miss.

### Practical Implications

- **Domain compatibility matters**: The contrast between Math × Code (compatible) and Medical pairings (incompatible) highlights that joint-learning benefits do not generalize across domain combinations—practitioners must consider update compatibility, not just teacher expertise.

- **Auxiliary teachers as free critics**: Non-routed teachers can be repurposed as cross-domain critics on the same trajectories without additional rollouts, providing valuable compatibility signals at modest computational cost.

- **Adaptive norm retention**: The norm-retention floor balances conflict reduction with target-domain learning, preventing full projection from erasing useful learning signals when gradients strongly oppose.

---

## Conclusion

### Main Takeaways

1. **Identified failure mode**: Domain-routed MOPD can induce persistent gradient conflicts in shared parameters when domain updates are poorly compatible, causing asymmetric performance degradation.

2. **ROAD MOPD solution**: Using non-routed teachers as cross-domain critics on the same student trajectories, with agreement-guided token weighting and preventive gradient projection under a norm-retention constraint, effectively mitigates update interference while preserving target-domain learning.

3. **Empirical validation**: ROAD MOPD improves adversely affected domains under high conflict (5.6–8.8 points over Vanilla MOPD), preserves low-conflict gains, and generalizes to three domains, outperforming standard MOPD and gradient-conflict correction baselines.

### Limitations and Future Directions

- Evaluation is limited to three text-based reasoning domains (Math, Code, Medical)
- Auxiliary scoring and gradient computation add training cost
- Extending to broader domains and larger teacher pools calls for:
  - Efficient teacher selection strategies
  - Improved compatibility estimation methods

---

_Markdown view of https://picx.dev/p/cEv7ty, served by PicX — AI-generated visual whiteboard summaries of research papers._