# Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

> Waypoint-OPD gates endpoint policy-shift distillation by temporal agreement between early and late teacher RL shifts, achieving state-of-the-art transfer across math and code reasoning.

- **Source:** [arXiv](https://arxiv.org/abs/2610.21601)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/f9wv7E
- **Whiteboard:** https://picx.dev/p/f9wv7E/image

## Summary

## Summary (Overview)

- **Waypoint-OPD** is a novel on-policy distillation method that uses an intermediate teacher checkpoint (Waypoint) from the same RL run to gate the transfer of RL-induced policy shifts to student models.
- The key insight is that the endpoint (Initial-to-Final) policy shift alone cannot distinguish whether early and late teacher policy changes reinforce or cancel each other; Waypoint-OPD uses directional agreement between consecutive shifts to weight the endpoint signal.
- The method requires only **one additional frozen-teacher scoring pass** compared to endpoint-only transfer, with no new teacher training or verifier calls during student transfer.
- Experiments with Qwen3-4B teachers and Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B students across mathematical and code reasoning show **the highest domain-average scores in all six settings** compared to baselines.
- Ablations confirm that temporal gating, prefix-specific gate assignment, and batch-scale normalization each contribute to the method's performance gains.

## Introduction and Theoretical Foundation

### Background and Motivation

Reinforcement learning improves language-model reasoning using verifiable rewards, but requires repeated generation, verification, and optimization of on-policy responses. Reusing a teacher's RL improvements offers a way to share this training effort across target models. The central question is: **which local changes in the teacher's policy should guide a student on its own generated trajectories?**

### Theoretical Basis

Standard on-policy distillation (OPD) trains the student to match the teacher's token distributions on student-generated prefixes:

$$\mathcal{L}_{\mathrm{OPD}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_{\theta}(\cdot | x)} \left[ \sum_{t=1}^{T} D_{\mathrm{KL}}(\pi_{\theta}(\cdot | s_t) \| \pi_1(\cdot | s_t)) \right] \tag{1}$$

Direct-OPD uses the teacher's RL-induced policy shift as an implicit reward, decomposing the Initial-to-Final shift into token-level log-probability ratios:

$$\Delta^{(0,1)}(y \mid x) = \log \frac{\pi_1(y \mid x)}{\pi_0(y \mid x)} = \sum_{t=1}^{T} \delta_t(y_t) \tag{2}$$

where

$$\delta_t(a) = \log \pi_1(a \mid s_t) - \log \pi_0(a \mid s_t) \tag{3}$$

**Key limitation addressed:** The Initial-to-Final log-probability ratio captures only the change between pre- and post-RL policies. The same endpoint signal can arise from early and late policy changes that point in similar or opposing directions. Endpoint-only transfer cannot distinguish these cases.

**Hypothesis:** Agreement between early and late teacher policy shifts provides evidence for deciding how much weight to assign an endpoint shift at a student-generated prefix.

## Methodology

### 3.1 Student-Local Trajectory Consistency

At a student-visited prefix $s_t$, the top-k most probable next-token candidates are selected via $\mathcal{K}_t = \mathrm{TopK}(\pi_{\theta}(\cdot | s_t), k)$. The normalized student distribution on this support is:

$$q_t(a) = \frac{\pi_{\theta}(a \mid s_t)}{\sum_{b \in \mathcal{K}_t} \pi_{\theta}(b \mid s_t)}, \qquad a \in \mathcal{K}_t \tag{5}$$

The two consecutive teacher shifts (early and late) are:

$$\delta_t^{(1)}(a) = \log \pi_m(a \mid s_t) - \log \pi_0(a \mid s_t) \tag{6}$$

$$\delta_t^{(2)}(a) = \log \pi_1(a \mid s_t) - \log \pi_m(a \mid s_t) \tag{7}$$

These telescope to $\delta_t = \delta_t^{(1)} + \delta_t^{(2)}$.

To remove action-independent components (uniform shifts across candidates), a $q_t$-centering operator is applied:

$$C_{q_t}[z](a) = z(a) - \mathbb{E}_{b \sim q_t}[z(b)] \tag{8}$$

yielding $u_t = C_{q_t}[\delta_t^{(1)}]$, $v_t = C_{q_t}[\delta_t^{(2)}]$, and $d_t = C_{q_t}[\delta_t] = u_t + v_t$.

### 3.2 Agreement Gating and Batch-Scale Normalization

The local temporal agreement (cosine similarity weighted by student probability) is:

$$c_t = \frac{\langle u_t, v_t \rangle_{q_t}}{\|u_t\|_{q_t} \|v_t\|_{q_t}} \tag{9}$$

When both centered segment norms exceed $\varepsilon = 10^{-8}$, the positive-cosine gate is:

$$g_t = \max(0, c_t) \tag{10}$$

If either norm is at most $\varepsilon$, a low-norm fallback sets $g_t = 1$. The gate $g_t \in [0,1]$ is shared by all candidates at prefix $s_t$, removing the endpoint signal when $c_t \leq 0$ and scaling its magnitude by $c_t$ otherwise.

**Illustrative example:** For a nonzero centered endpoint $d$: the pair $u = v = d/2$ gives $c = 1$ and $g = 1$, whereas $u = 2d, v = -d$ gives $c = -1$ and $g = 0$. Both yield $u + v = d$, so endpoint-only transfer cannot distinguish them.

Batch-scale normalization stabilizes the average gate scale:

$$\kappa_B = \operatorname{clip}\left(\frac{\tau}{\max\left(|B_{\text{pref}}|^{-1} \sum_{t \in B_{\text{pref}}} g_t, \varepsilon_{\text{num}}\right)}, \kappa_{\min}, \kappa_{\max}\right) \tag{11}$$

### 3.3 Dense Candidate Objective

The final coefficient for candidate action $a \in \mathcal{K}_t$ is:

$$A_t(a) = \operatorname{sg}[\kappa_B q_t(a) g_t d_t(a)] \tag{12}$$

The local on-policy surrogate to maximize:

$$\mathcal{J}_{\mathrm{WP}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}} \Bigg[ \mathbb{E}_{y \sim \pi_{\theta}(\cdot | x)} \left[ \sum_{t=1}^{T} \sum_{a \in \mathcal{K}_t} A_t(a) \log \pi_{\theta}(a \mid s_t) \right] - \beta \widehat{D}_{\mathrm{KL}}(\pi_{\theta}(\cdot \mid x) \| \pi_S(\cdot \mid x)) \Bigg] \tag{13}$$

The implementation uses dual-clipped PPO, which has the same gradient as this surrogate at the rollout policy.

## Empirical Validation / Results

### Experimental Setup

- **Teachers:** Three checkpoints from the same Qwen3-4B RL run: Initial (before RL), Waypoint (step 600), Final (step 1200)
- **Students:** Qwen3-8B, Qwen3-14B, Qwen3-30B-A3B
- **Domains:** Mathematical reasoning (7 benchmarks) and code reasoning (5 evaluations)
- **Baselines:** Base, OPD, OPD², Direct-OPD

### Main Results — Mathematics

**Table 1: Mathematical reasoning across three student sizes (mean@32 accuracy, %)**

| Method | AIME24 | AIME25 | AIME26 | HMMT25Feb | HMMT25Nov | HMMT26Feb | AMC23 | Avg. |
|--------|--------|--------|--------|-----------|-----------|-----------|-------|------|
| **Teacher: Qwen3-4B** | | | | | | | | |
| Initial | 21.67 | 21.67 | 21.67 | 12.50 | 10.00 | 15.15 | 71.25 | 24.84 |
| Waypoint | 58.96 | 54.79 | 56.77 | 31.35 | 35.73 | 31.72 | 93.44 | 51.82 |
| Final | 65.83 | 60.00 | 60.83 | 35.83 | 40.83 | 35.13 | 94.45 | 56.13 |
| **Student: Qwen3-8B** | | | | | | | | |
| Base | 29.17 | 20.83 | 18.33 | 11.67 | 13.33 | 18.94 | 70.00 | 26.04 |
| OPD | 63.65 | 49.69 | 54.90 | 29.06 | 38.65 | 32.58 | 88.75 | 51.04 |
| OPD² | 64.48 | 48.33 | 53.96 | 28.44 | 40.52 | 31.53 | 92.97 | 51.46 |
| Direct-OPD | 67.29 | 55.31 | 57.40 | 31.77 | 40.83 | 33.71 | 91.41 | 53.96 |
| **Waypoint-OPD** | **68.85** | **57.92** | **60.21** | **34.38** | **42.60** | **36.36** | **92.66** | **56.14** |
| **Student: Qwen3-14B** | | | | | | | | |
| Base | 30.00 | 27.50 | 20.00 | 10.00 | 11.67 | 18.94 | 73.13 | 27.32 |
| OPD | 69.17 | 55.52 | 58.44 | 36.77 | 45.21 | 31.82 | 96.25 | 56.17 |
| OPD² | 69.48 | 46.98 | 54.38 | 33.13 | 41.98 | 31.91 | 91.95 | 52.83 |
| Direct-OPD | 70.10 | 56.46 | 61.98 | 37.92 | 47.60 | 35.13 | 94.06 | 57.61 |
| **Waypoint-OPD** | **70.21** | **60.52** | **63.75** | **40.21** | 47.08 | **36.55** | **96.09** | **59.20** |
| **Student: Qwen3-30B-A3B** | | | | | | | | |
| Base | 31.67 | 22.50 | 22.50 | 10.00 | 13.33 | 21.21 | 75.00 | 28.03 |
| OPD | 63.54 | 49.06 | 57.50 | 30.83 | 37.71 | 28.03 | 91.88 | 51.22 |
| OPD² | 64.90 | 51.77 | 60.42 | 31.56 | 38.54 | 30.21 | 91.33 | 52.68 |
| Direct-OPD | 69.17 | 54.69 | 61.98 | 38.96 | 41.56 | 33.05 | 92.34 | 55.96 |
| **Waypoint-OPD** | **70.94** | **58.44** | **63.65** | **39.90** | **44.69** | 32.77 | **94.53** | **57.85** |

Waypoint-OPD improves over Direct-OPD by **2.18, 1.59, and 1.89 percentage points** for Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B, respectively, improving 19 of 21 benchmark comparisons.

### Main Results — Code

**Table 2: Average code reasoning performance (average mean@8 scores, %)**

| Method | Qwen3-8B | Qwen3-14B | Qwen3-30B-A3B |
|--------|----------|-----------|---------------|
| Base | 41.03 | 47.72 | 50.33 |
| OPD | 43.91 | 51.66 | 51.91 |
| OPD² | 51.21 | 55.60 | 55.96 |
| Direct-OPD | 50.59 | 56.30 | 56.35 |
| **Waypoint-OPD** | **53.25** | **57.82** | **57.54** |

Waypoint-OPD improves over Direct-OPD by **2.66, 1.52, and 1.19 percentage points** on average, with gains in all 15 code comparisons.

### Learning Dynamics

Waypoint-OPD delivers consistent gains over Direct-OPD across training checkpoints (steps 0–100), leading on all three benchmarks from step 60 onward. On HMMT November 2025, Waypoint-OPD exceeds the Final teacher's 40.83 at steps 60, 80, and 100, reaching 42.60 at step 60, demonstrating ability to surpass the Final teacher's score.

### Ablation Results

**Table 3: Gate and normalization ablations on Qwen3-8B mathematical reasoning**

| Method | AIME24 | AIME25 | AIME26 | HMMT25Feb | HMMT25Nov | HMMT26Feb | AMC23 | Avg. |
|--------|--------|--------|--------|-----------|-----------|-----------|-------|------|
| No gate | 66.25 | 54.27 | 56.67 | 32.60 | 39.79 | 34.66 | 90.23 | 53.50 |
| Shuffled gate | 63.33 | 52.92 | 54.58 | 30.00 | 37.08 | 31.06 | 89.45 | 51.20 |
| No normalization | 68.13 | 56.46 | 58.33 | 34.79 | 41.56 | 35.23 | 92.03 | 55.22 |
| Direct-OPD | 67.29 | 55.31 | 57.40 | 31.77 | 40.83 | 33.71 | 91.41 | 53.96 |
| **Waypoint-OPD** | **68.85** | **57.92** | **60.21** | **34.38** | **42.60** | **36.36** | **92.66** | **56.14** |

Key ablation findings:
- **Removing the gate:** Lowers Avg. from 56.14 to 53.50 (−2.64 points)
- **Shuffling gates:** Lowers Avg. to 51.20 (−4.94 points vs. full method), more detrimental than removing gates entirely
- **Removing normalization:** Lowers Avg. by 0.92 points to 55.22, though still exceeding Direct-OPD by 1.26 points

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Temporal agreement as a selection criterion:** The paper introduces the use of an intermediate Waypoint checkpoint to assess temporal agreement at student-generated prefixes, revealing directional information unavailable from the endpoint pair alone. This addresses a fundamental limitation of endpoint-only distillation methods.

2. **Decomposition of policy shifts:** The method formalizes that the same endpoint shift $\delta_t = \delta_t^{(1)} + \delta_t^{(2)}$ can arise from early and late shifts that reinforce or cancel each other, and provides a principled way to distinguish these cases using cosine similarity of centered segment shifts.

3. **Gating as weighting, not filtering:** The positive-cosine gate $g_t = \max(0, c_t)$ both removes conflicting signals ($c_t \leq 0$) and scales aligned signals by their agreement strength, providing a continuous weighting mechanism rather than a binary filter.

### Practical Implications

- **Efficiency:** Waypoint-OPD requires only one additional frozen-teacher scoring pass compared to Direct-OPD, with no new teacher training or verifier calls during student transfer — making it practical for large-scale deployment.
- **Scalability:** The method works across student scales (8B to 30B-A3B) and architectures (dense and MoE), demonstrating robustness.
- **Teacher surpassing:** The 14B and 30B-A3B students exceed the Final teacher's average by 3.07 and 1.72 points (math) and 2.45 and 2.17 points (code), showing that students can surpass their teacher's capabilities.
- **Practical guidance:** Intermediate checkpoints from the same RL run provide valuable evidence for weighting local policy changes during student learning, suggesting that practitioners should retain and reuse intermediate checkpoints rather than only the final one.

## Conclusion

Waypoint-OPD presents a novel approach to policy-shift distillation that leverages directional agreement between consecutive teacher RL shifts to gate endpoint transfer signals. The method addresses a fundamental limitation of endpoint-only transfer: the inability to distinguish between early and late policy changes that reinforce versus cancel each other.

**Key takeaways:**
- Intermediate teacher checkpoints provide valuable consistency evidence for weighting local policy changes during student learning
- The method achieves state-of-the-art transfer performance across mathematics and code reasoning at multiple student scales
- Prefix-specific gate assignment and batch-scale normalization are both important components

**Future directions** suggested by the work include:
- Investigating the role of Waypoint checkpoint position (the paper explores steps 500, 600, and 700 in Appendix F)
- Exploring whether directional agreement alone guarantees beneficial supervision (the paper notes it does not)
- Extending the approach to other domains and model families
- Studying the interaction between gate design and student architecture

---

_Markdown view of https://picx.dev/p/f9wv7E, served by PicX — AI-generated visual whiteboard summaries of research papers._