Weak-to-Strong Generalization via Direct On-Policy Distillation

Summary (Overview)

• Core idea: Direct On-Policy Distillation (Direct-OPD) transfers the RL-induced policy shift from a small, weak teacher model to a stronger student model, rather than imitating the teacher's final policy. This is done by using the log-ratio between the post-RL teacher and its pre-RL reference as a dense implicit reward for the student.
• Key advantage: The method avoids expensive RL on the large target model. Instead, RL is run cheaply on a small model, and the policy shift is transferred via a short, lightweight distillation step. For example, Direct-OPD boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs.
• Consistent weak-to-strong gains: The policy shift from a small teacher (e.g., R1-Distill-1.5B → JustRL-1.5B) improves multiple stronger students (R1-Distill-7B, Qwen3-1.7B, Qwen3-4B), even when the student already outperforms the post-RL teacher.
• Compute efficiency: At matched RL steps, the weak-to-strong route (small-model RL + Direct-OPD transfer) outperforms direct RL on the large model in both accuracy and wall-clock time.
• Composability: Multiple independently learned policy shifts can be applied sequentially to the same student, accumulating gains.

Introduction and Theoretical Foundation

Background: Reinforcement learning with verifiable rewards (RLVR) is a powerful post-training recipe for improving language model reasoning, but it becomes expensive as models scale. Running RL on large models requires many on-policy rollouts, making post-training a bottleneck.

Motivation: The authors propose a weak-to-strong alternative: run RL on a small, cheap model, then reuse the RL outcomes to improve a stronger target model. The central question is what to transfer from the weak RL run. Simply distilling the final teacher policy fails because the teacher's final policy mixes useful RL gains with the capacity limits of the small model—imitating it can overwrite stronger behavior in the student.

Theoretical basis: The key insight is that under KL-regularized RL, the policy–reference log-ratio recovers the reward up to a constant. Given a post-RL teacher $\pi^T$ and its pre-RL reference $\pi^T_{\text{ref}}$, the policy shift

$$\Delta^T(y|x) = \log \pi^T(y|x) - \log \pi^T_{\text{ref}}(y|x)$$

is equivalent to the teacher's implicit reward $r^T$ (up to scale and per-prompt constant):

$$\Delta^T(y|x) = \frac{1}{\beta} r^T(x,y) - \log Z^T(x).$$

This identity allows Direct-OPD to transfer the direction of RL improvement—the dense reward signal—rather than the teacher's absolute distribution.

Methodology

Direct-OPD Objective: The student $\pi_\theta$ (initialized from $\pi^S$) is trained to maximize the expected teacher shift while being regularized toward its own initialization:

$$J_{\text{Direct-OPD}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}} \Big[ \mathbb{E}_{y \sim \pi_\theta(\cdot|x)} \big[ \Delta^T(y|x) \big] - \alpha D_{\text{KL}}\big( \pi_\theta(\cdot\|x) \| \pi^S(\cdot|x) \big) \Big].$$

The optimum is $\pi^* \propto \pi^S \exp(\frac{1}{\alpha} \Delta^T)$, which is equivalent to optimizing the student against the teacher's implicit reward $r^T$ with a student-specific KL penalty.

Token-level implementation: The sequence-level shift decomposes into per-token rewards:

$$r_t(v) = \log \pi^T(v|s_t) - \log \pi^T_{\text{ref}}(v|s_t).$$

At each student-visited prefix, the method uses a top-$k$ action-space restriction. The gradient is estimated via a Rao-Blackwellized per-step estimator:

$$\nabla_\theta J_{\text{analytical}} = \mathbb{E}_{x, y \sim \pi_\theta} \left[ \sum_t \sum_{v \in S_t} \bar{p}_t(v) \cdot r_t(v) \cdot \nabla_\theta \log \pi_\theta(v|s_t) \right],$$

where $\bar{p}_t$ is the renormalized student distribution on the top-$k$ support $S_t$. The weighted reward is detached to avoid extra Jacobian terms.

Adaptive KL control: Because the scale of $\Delta^T$ is fixed by the teacher's training (unobservable), a fixed KL coefficient $\alpha$ may not transfer robustly. The authors propose an adaptive controller that updates $\alpha$ based on the batch mean of the dense reward $\bar{r}_m$:

$$\alpha_{m+1} = \text{clip}\big( \alpha_m (1 + \epsilon \cdot \text{sgn}(\bar{r}_m)), \alpha_{\min}, \alpha_{\max} \big).$$

This keeps the mean teacher shift near zero, preventing the student from over-exploiting unreliable local signals.

Empirical Validation / Results

Experimental setup:

• Teacher pairs: (R1-Distill-1.5B → JustRL-1.5B) and (Nemotron-1.5B → QuestA-Nemotron-1.5B)
• Students: R1-Distill-7B, Qwen3-1.7B, Qwen3-4B
• Evaluation: AIME 2024 and AIME 2025, accuracy at ave@32

RQ1: Weak teacher improves stronger students. Direct-OPD transfers the policy shift from the JustRL-1.5B teacher into all three students, including those that already outperform the post-RL teacher. The same holds for the QuestA teacher pair.

RQ2: Weak-to-strong beats direct RL. For the same number of RL steps, running RL on the 1.5B model and transferring to R1-Distill-7B outperforms running RL directly on the 7B model in both accuracy and wall-clock time. The transfer stage adds only ~4 hours on 8 A100 GPUs, compared to ~320 hours for 7B RL.

RQ3: Sequential composition. Two independently learned policy shifts (JustRL then QuestA) can be applied sequentially to Qwen3-1.7B, achieving cumulative gains: AIME24 from 48.3 → 58.3 → 63.8; AIME25 from 36.8 → 43.2 → 46.8.

Analysis:

• Cross-pattern transfer without token-overlap imitation: Direct-OPD does not require high teacher–student top-$k$ overlap. In cross-pattern transfers, overlap remains low, yet the student improves. Actor entropy remains controlled.
• Short-horizon training generalizes to longer rollouts: Training with 2k token response length shifts the student's behavior across much longer sequences (up to 16k tokens), and validates better than longer horizons.
• Adaptive KL controls reward reliability: The best fixed KL is pair-dependent. Adaptive KL pulls the mean dense reward toward zero, preventing the student from exploiting unreliable teacher–reference differences on out-of-distribution states.

Theoretical and Practical Implications

• Theoretical contribution: The paper reframes weak-to-strong transfer as a problem of reusing the RL-induced policy shift, not the final policy. This shift is a dense implicit reward that can be extracted from a checkpoint pair and applied on the student's own on-policy states. The method is grounded in the policy-as-reward identity from KL-regularized RL.
• Practical impact: Direct-OPD makes RL post-training more scalable. Expensive RL is run only on a small model, and the resulting improvement direction is transferred to large models cheaply. This could reduce the cost and time required for improving reasoning in large language models, and enables reusing RL outcomes across model families and scales.
• Broader relevance: The method suggests that RL outcomes are reusable as implicit reward signals, not merely as final models to imitate. This connects to ideas of implicit reward extraction, weak-to-strong generalization, and efficient post-training.

Conclusion

Direct-OPD demonstrates that the policy shift learned by a small RL teacher can serve as a dense, reusable reward for weak-to-strong generalization. The key insight is that the transferable object is the log-ratio between the post-RL teacher and its pre-RL reference, evaluated on student-visited tokens. This allows a weaker teacher to improve stronger students, even when the student already surpasses the teacher. The method is empirically effective across multiple teacher pairs and student families, outperforms direct RL in compute efficiency, and supports sequential composition of multiple policy shifts.

Limitations: The signal is conditional—Direct-OPD can fail when the teacher/reference improvement is not meaningful on student-visited states. The optimal response length and KL strength remain teacher–student dependent. Future work may address these dependencies and extend the method to other training paradigms.