Summary (Overview)

• Core contribution: Introduces U-OPSD (Unsupervised On-Policy Self-Distillation), a method that eliminates the need for ground-truth supervision in on-policy self-distillation by using the model's own majority-vote consensus to construct pseudo-solutions.
• Key mechanism: U-OPSD samples multiple rollouts, identifies a pseudo-answer via majority vote, and distills the teacher's solution-conditioned distribution into the student only along disagreeing rollouts, focusing correction precisely where the model is confidently wrong.
• Results: On five math benchmarks (AIME24, AIME25, HMMT25, MATH500, AMC23), U-OPSD improves over base models by 8.5–10.7% in non-thinking mode and 1.9–2.2% in thinking mode across Qwen3-4B/8B, while matching or surpassing supervised methods (SFT, GRPO, OPSD) that use ground-truth labels.
• Key finding: Consensus-derived pseudo-solutions are substantially more effective as conditioning context for token-level distillation than as scalar rewards for policy optimization (outperforming self-rewarding RL baselines like TTRL, RENT, and Intuitor by 7.0–11.3% on average).
• Practical advantage: Requires no labeled data, environmental feedback, or external teacher, enabling scalability to unlabeled problems and domains where supervision is costly or unavailable.

---

Introduction and Theoretical Foundation

Background and Motivation

Post-training of LLMs has been driven by three paradigms: supervised fine-tuning (SFT), knowledge distillation from stronger teachers, and reinforcement learning with verifiable rewards (RLVR). Each has limitations:

• SFT: Suffers from train-inference mismatch and catastrophic forgetting.
• Knowledge distillation: Requires a separate, stronger teacher model.
• RLVR: Depends on curated ground-truth labels, which are costly and scarce.

On-policy distillation (OPD) bridges SFT and RL by training on the model's own generations, reducing train-inference mismatch while retaining dense token-level supervision. On-policy self-distillation (OPSD) removes the external teacher by having the same model act as both teacher and student, with the teacher conditioned on ground-truth solutions.

The Core Problem

Despite the name "self-distillation," existing OPSD methods still rely on external supervision—the teacher's privileged context comes from ground-truth solutions, demonstrations, or environmental feedback. The information that makes the teacher more capable originates outside the model. This dependence limits scalability to unlabeled problems.

Key Research Question

Does the teacher in on-policy self-distillation need a ground-truth solution? Can a model construct its own privileged context and perform genuine self-distillation?

Theoretical Insight

Although an individual rollout may be unreliable, agreement among multiple independently sampled rollouts provides an endogenous confidence signal. This enables deriving both the teacher reference and student trajectories entirely from the model's own on-policy samples, without any external supervision.

---

Methodology

Notation

Let $\pi_\theta$ be an LLM defining next-token distributions over vocabulary $\mathcal{V}$, and $\bar{\pi} \triangleq \pi_{\mathrm{sg}[\theta]}$ be the same network with gradients detached. For a sequence $y$, $y_{<t}$ denotes its prefix, and $\mathrm{Ans}(y) \in \mathcal{A} \cup \{\emptyset\}$ the final answer parsed from it.

The U-OPSD Algorithm

U-OPSD operates in three steps per prompt $x$:

Step 1: Sample. Draw $G$ independent rollouts:

$$y^{(1)}, \ldots, y^{(G)} \overset{\mathrm{i.i.d.}}{\sim} \bar{\pi}(\cdot|x)$$

Parse answers $a^{(g)} = \mathrm{Ans}(y^{(g)})$, where $\emptyset$ marks unparsable (invalid) rollouts.

Step 2: Vote. Compute the pseudo-answer via majority vote:

$$\tilde{a}(x) = \arg\max_{a} \sum_{g=1}^{G} \mathbb{1}[a^{(g)} = a]$$

Partition rollouts into agreeing and disagreeing sets:

$$\mathcal{V}_x^{+} = \{y^{(g)} : a^{(g)} = \tilde{a}(x)\}, \quad \mathcal{V}_x^{-} = \{y^{(g)} : a^{(g)} \neq \tilde{a}(x)\}$$

Compute the self-consistency score:

$$c(x) = \frac{1}{G} \sum_{g=1}^{G} \mathbb{1}[a^{(g)} = \tilde{a}(x)]$$

If $c(x) < \tau$ (default $\tau = 0.5$), the prompt is skipped. Invalid rollouts belong to neither set.

Step 3: Distill. Select the longest agreeing rollout $y^{+}$ as the teacher reference, and distill along disagreeing rollouts $y^{-}$:

$$\mathcal{L}_{\mathrm{U-OPSD}}(\theta) = \mathbb{E}_{x \sim \mathcal{U}} \mathbb{E}_{\{y^{(g)}\}_{g=1}^{G} \sim \bar{\pi}(\cdot|x)} \left[ \mathbb{1}[\mathcal{Y}_x^{-} \neq \emptyset] \frac{1}{|\mathcal{Y}_x^{-}|} \sum_{y^{-} \in \mathcal{Y}_x^{-}} \frac{1}{|y^{-}|} \sum_{n=1}^{|y^{-}|} D_\beta \left(\bar{\pi}(\cdot|x, y^{+}, y_{<n}^{-}) \| \pi_\theta(\cdot|x, y_{<n}^{-})\right) \right] \tag{1}$$

where $D_\beta(P\|Q)$ is a divergence between teacher and student next-token distributions, with $\beta = 0$ giving forward KL divergence.

Comparison with Prior Methods

Training Configuration

• Objective: Forward KL ($\beta = 0$) over full vocabulary with per-token pointwise clipping
• Teacher: Fixed to initial policy (stop-gradient)
• LoRA: Rank 64 ($\alpha = 128$) on attention and MLP projections
• Learning rate: $5 \times 10^{-6}$, gradient-norm clipping at 0.1
• Sampling: Temperature 1.1, top-p 0.95, top-k 20
• Rollouts: $G = 8$ per prompt, threshold $\tau = 0.5$
• Max completion length: 4,096 tokens (vs. 1,024 for OPSD)

---

Empirical Validation / Results

Non-Thinking Mode (Qwen3-4B and Qwen3-8B)

Table 1: Performance comparison on math reasoning benchmarks (non-thinking mode)

Key findings:

• U-OPSD improves over base by 8.5% (4B) and 10.7% (8B)
• Outperforms supervised OPSD by 3.2% (4B) and 2.3% (8B)
• Label-free RL baselines (TTRL, RENT, Intuitor) improve by at most 1.5%

Thinking Mode

Table 2: Performance comparison (thinking mode)

U-OPSD matches or slightly exceeds supervised OPSD in thinking mode, with smaller gains due to limited headroom.

Instruction-Tuned Models

• Qwen3-30B-A3B-Instruct-2507: U-OPSD improves from 75.77 → 77.46, surpassing OPSD by 1.1%
• Qwen3-4B-Instruct-2507: U-OPSD improves from 67.00 → 68.78, surpassing OPSD by 1.7%

Ablation Studies

Pseudo-label quality (G=8, τ=0.5): 96.3% of rollouts yield parsable answers, 94.0% of prompts receive pseudo-labels, and 86.7% of pseudo-labels match gold answers.

Self-consistency threshold τ (Figure 4, left): Performance is monotone in τ—looser thresholds are better (58.59 at τ=0.3 vs. 44.40 at τ=0.9, a 14.2% spread).

Rollouts per prompt G (Figure 4, middle): G=4 and G=8 are indistinguishable (56.99 vs. 57.10); G=12 gains 4.7% over default.

Teacher update (Figure 4, right): EMA teacher (decay 0.995) gains 2.4% over frozen teacher at best checkpoint.

Divergence computation strategy (Table 4):

Divergence metrics (Table 5): Forward KL ($\beta=0$) is essential—reverse KL causes training divergence (length exploitation), and JSD ($\beta=0.5$) performs no better than the base model (43.34 vs. 43.57).

---

Theoretical and Practical Implications

Theoretical Implications

1. Self-distillation is genuinely possible without external supervision: The paper demonstrates that a model's internal consistency (majority voting) provides sufficient signal for meaningful self-improvement, challenging the assumption that privileged contexts must come from external sources.

2. Consensus as a privileged context: The finding that consensus-derived pseudo-solutions outperform scalar rewards (as in TTRL/RENT) suggests that dense distributional supervision along disagreeing trajectories is fundamentally more informative than sparse sequence-level rewards.

3. The competence frontier curriculum: U-OPSD naturally focuses training on prompts where the model can identify a plausible solution but still assigns probability to conflicting trajectories—an emergent curriculum without external difficulty labels.

4. Forward KL is critical: The collapse under reverse KL (length exploitation: generations growing from 2.7k to 99k characters) reveals that mode-seeking divergences are unstable for self-distillation with pseudo-labels, whereas mode-covering forward KL is essential.

Practical Implications

1. Scalability to unlabeled data: U-OPSD removes the annotation bottleneck, enabling post-training on arbitrary problem corpora without ground-truth solutions.

2. Cost efficiency: Eliminates the need for external teacher models or verifiers, reducing infrastructure requirements.

3. Transferability: The recipe transfers to larger models (30B-A3B MoE) without hyperparameter tuning, suggesting broad applicability.

4. Practical optimizations: Top-k truncation (k=100) of the divergence computation improves performance (59.01 avg.) while reducing computational cost.

---

Conclusion

Main Takeaways

U-OPSD demonstrates that ground-truth solutions in on-policy self-distillation can be replaced by the model's own majority-vote consensus. The agreeing rollout serves as the teacher's reference, and distillation is applied only to self-inconsistent rollouts, enabling genuine self-distillation without any external supervision.

Key Results Summary

• Non-thinking mode: U-OPSD outperforms supervised OPSD by 3.2% (4B) and 2.3% (8B)
• Thinking mode: U-OPSD matches OPSD (0.9% ahead at 4B, level at 8B)
• vs. GRPO: Surpasses by 0.7% (4B) and 1.1% (8B) in thinking mode
• vs. self-rewarding RL: Surpasses by 7.0–11.3% in non-thinking mode

Key Insight

"In the regime of on-policy self-distillation—a base model competent enough to vote well and fallible enough to have headroom—the binding constraint is not access to gold solutions but the machinery for surfacing and correcting the model's own inconsistencies, a machinery that requires no supervision."

Limitations and Future Directions

1. Domain scope: Currently limited to competition mathematics with extractable, canonicalizable answers; open-ended generation requires softer consensus mechanisms.
2. Regime dependence: Gains are larger in non-thinking mode; thinking-mode models have less headroom.
3. Bounded by base competence: Majority-vote supervision reproduces the base model's most frequent answer, so it is bounded by that answer's accuracy (13.3% wrong pseudo-labels measured).
4. Future work: Mechanisms for detecting/down-weighting low-quality consensus (e.g., by vote margin), seed-replicated error bars, and extension to open-ended domains.