# DAPD: Dual-Anchored Policy Distillation

> Dual-Anchored Policy Distillation eliminates privilege illusion in on-policy self-distillation by matching teacher-student information access, improving reasoning and coding performance by up to 2.78 points.

- **Source:** [arXiv](https://arxiv.org/abs/2608.01735)
- **Published:** 2026-08-05
- **Permalink:** https://picx.dev/p/Y13IMA
- **Whiteboard:** https://picx.dev/p/Y13IMA/image

## Summary

## Summary (Overview)

- The paper identifies **information asymmetry** between a privileged teacher (conditioned on reference completions) and a student (without privileged information) as the **root cause of privilege illusion** in on-policy self-distillation (OPSD), where students behave as though unavailable information were present, degrading performance.
- To address this, the authors propose **Dual-Anchored Policy Distillation (DAPD)**, a unified framework with two components:
  - **Dual-Path Anchoring (DPA)** introduces a self-conditioned "bridge" distribution and aligns reference and rollout behavior along two matched-information paths (unconditioned and privileged).
  - **Dual-Source Anchoring (DSA)** applies these paths in both reference-to-rollout and rollout-to-reference directions, balancing reliable reference guidance with student-reachable rollout guidance.
- Extensive experiments on Qwen3 models (1.7B–32B) across reasoning, coding, and instruction-following benchmarks show DAPD consistently outperforms OPSD by **+2.00 points on average at 4B**, with gains persisting at scale (+2.69 at 4B, +2.78 at 32B).
- DAPD reduces "wrong claims" (unsupported answers) by **73%** compared to OPSD at late training steps, confirming mitigation of privilege illusion.
- Ablations verify that both matched-information paths and both guidance sources are complementary, and that the method can operate without curated references by using dual rollouts or a verifier.

## Introduction and Theoretical Foundation

**Background.** Post-training of language models increasingly uses on-policy (self) distillation (OPSD), where a model is trained on its own rollouts and supervised by a teacher distribution at each token prefix. To strengthen the teacher, OPSD conditions it on privileged information (e.g., a reference completion). However, the student cannot access this information at inference, leading to a **privilege illusion**: the student produces unsupported claims or continues derivations as if the reference were available.

**Root cause.** The authors hypothesize that the **information asymmetry**—the teacher sees privileged information, the student does not—causes the student to learn privilege-dependent behavior that cannot be reproduced. They formalize this as **Entangled Distillation**: the OPSD objective mixes useful reference-guided supervision with non-reproducible privilege-dependent supervision.

**Validation.** By replacing the student’s None distribution with a **self-conditioned distribution** (conditioning on the full completion being predicted), they remove the asymmetry. This intervention (called the *Privileged Anchor*) reduces wrong claims by 45% and improves Avg@12 by +6.22 points, confirming that information matching, not teacher quality, is key.

**Formal definitions.** Let $p_\theta$ be the student policy, $x$ the prompt, $y^*$ the reference completion, and $y$ an on-policy rollout. For a completion $s \in \{y, y^*\}$, define three distributions at token $t$ with prefix $s_{<t}$:

- **None**: $p^s_{\text{None}} = p_\theta(\cdot \mid x, s_{<t})$ — no privileged information.
- **Cross**: $p^s_{\text{Cross}} = p_\theta(\cdot \mid x, s_{<t}, \bar{s})$ — conditioned on the other completion (privileged).
- **Self**: $p^s_{\text{Self}} = p_\theta(\cdot \mid x, s_{<t}, s)$ — conditioned on the completion itself (privileged).

The original OPSD loss is:

$$
\mathcal{L}_{\text{OPSD}} = \mathbb{E}_{(x,y^*) \sim \mathcal{D}, y \sim p_\theta(\cdot|x)} \left[ \frac{1}{|y|} \sum_{t=1}^{|y|} D(\text{sg}[p_{\text{Cross}}] \| p_{\text{None}}) \right] \tag{1}
$$

where $D$ is a divergence (forward KL in practice) and $\text{sg}[\cdot]$ denotes stop-gradient.

## Methodology

**Three directed objectives.** For each completion source $s$, DAPD defines three losses:

- **Entangled Distillation** (original OPSD term): 
  $\mathcal{L}^s_{\text{ent}} = \mathbb{E}[D(\text{sg}[p^s_{\text{Cross}}] \| p^s_{\text{None}})]$ (2)
- **Inference Anchor** (aligns Self toward None): 
  $\mathcal{L}^s_{\text{infer}} = \mathbb{E}[D(\text{sg}[p^s_{\text{None}}] \| p^s_{\text{Self}})]$ (3)
- **Privileged Anchor** (aligns Self toward Cross): 
  $\mathcal{L}^s_{\text{priv}} = \mathbb{E}[D(\text{sg}[p^s_{\text{Cross}}] \| p^s_{\text{Self}})]$ (4)

**Dual-Path Anchoring (DPA).** For an ordered direction $s \rightarrow \bar{s}$, DPA composes these into two paths:

- **Unconditioned path** (no privileged information on either side):
  $\mathcal{L}^{s \rightarrow \bar{s}}_{\text{uncond}} = \mathcal{L}^{\bar{s}}_{\text{infer}} + \mathcal{L}^{s}_{\text{ent}}$ (5)
  This aligns the two None distributions through the Self–Cross bridge.
- **Privileged path** (both sides have a full completion): directly uses $\mathcal{L}^{s}_{\text{priv}}$.

The DPA objective for one direction is:
$\mathcal{L}^{s \rightarrow \bar{s}}_{\text{DPA}} = \mathcal{L}^{s \rightarrow \bar{s}}_{\text{uncond}} + \mathcal{L}^{s}_{\text{priv}}$ (6)

**Dual-Source Anchoring (DSA).** DSA applies DPA in both directions ($y \rightarrow y^*$ and $y^* \rightarrow y$) and balances them:

$$
\mathcal{L}_{\text{DAPD}} = \lambda \mathcal{L}^{y \rightarrow y^*}_{\text{DPA}} + (1-\lambda) \mathcal{L}^{y^* \rightarrow y}_{\text{DPA}} \tag{7}
$$

where $\lambda$ controls the relative weight of reference-guided vs. rollout-guided supervision. The reference direction provides reliable correctness-oriented guidance; the rollout direction provides student-reachable on-policy guidance.

**Implementation details.** Training uses LoRA (rank 64, scale 128), learning rate $5\times10^{-6}$, effective batch size 32, and a component-clipped forward KL divergence:

$$
\ell_c(q,p) = \sum_{v \in \mathcal{V}} \min\{ q_v(\log q_v - \log p_v), c \} \tag{8}
$$

with cap $c=0.05$. The teacher distributions are detached; the two Entangled-Distillation teachers share a periodically updated snapshot of the student, while Inference/Privileged Anchor teachers use the base (LoRA-off) model.

## Empirical Validation / Results

**Main results (Qwen3-4B, Table 1).** DAPD achieves the best six-task average of **57.34**, improving over OPSD by **+2.00 points**. Gains are seen across reasoning (AIME24 +0.55, AIME25 +4.44, HMMT25 +3.06), coding (LCB v5 +1.05, BFCL v3 +0.59), and instruction following (IFBench +2.33). DAPD also outperforms Purified OPSD (+1.09) and DOPD (+3.85).

**Scalability (Figure 1b, Table 6).** While OPSD’s gains over Base diminish with scale (from +5.19 at 1.7B to <0.28 at 8B–32B), DAPD maintains consistent gains: +1.94, +2.69, +2.41, +2.04, +2.78 points at 1.7B, 4B, 8B, 14B, 32B respectively.

**Out-of-distribution (Table 2).** A reasoning-trained model evaluated on coding and instruction tasks shows DAPD achieves the best average (49.64), with +4.82 on LCB v5 and +1.37 average over OPSD.

**Privilege-illusion dynamics (Figure 2, Table 7).** Over training steps 250–300, DAPD reduces wrong claims by **73%** relative to OPSD (9 vs. 33 per 10k generations) and improves Avg@12 by 7.66 points.

**Ablations (Table 3).**
- *Path components*: Both unconditioned and privileged paths are necessary; adding Inference Anchor improves over Entangled Distillation alone, and adding Privileged Anchor further improves (e.g., for rollout source: 63.33 → 64.91 → 65.09).
- *Guidance sources*: Combining both sources yields best Avg@12 (65.28) vs. reference-only (63.89) or rollout-only (65.09).
- *Scale-dependent weights*: The optimal reference-guidance weight $\lambda$ decreases from 0.5 at 1.7B to 0.2 at 4B–14B, and privileged-anchor weights shift with scale.
- *Reference-free variants*: Dual-rollout (two independent rollouts) improves over OPSD (+0.46, +2.41, +2.41 at 1.7B, 4B, 8B); adding a verifier further boosts scores.

**Qualitative examples (Figure 4, Table 9).** OPSD often "recalls" unsupported answers, while DAPD derives correct answers from the prompt.

## Theoretical and Practical Implications

**Theoretical contribution.** The paper provides a formal analysis of the unconditioned path:
- **Proposition 1** (paired-prefix alignment) bounds the distance between the two None distributions via the Inference Anchor, Entangled Distillation, and a Self–Cross bridge consistency term.
- **Proposition 2** (sequence-level transfer) shows that Entangled Distillation on on-policy prefixes bounds the total variation between the sequence distributions of None and Cross.
- **Equation (21)** shows that a compatible Inference Anchor step moves the Cross distribution toward the reference-side None anchor through shared parameters, providing the optimization channel for the proxy bridge.

These results justify why DPA’s joint update implicitly aligns the rollout- and reference-side None distributions.

**Practical implications.** DAPD offers a principled way to mitigate privilege illusion without sacrificing the benefits of privileged supervision. It requires no additional inference-time cost and can adapt to model scale by tuning the guidance balance. The reference-free extension suggests potential for reducing dependence on curated data.

## Conclusion

The paper diagnoses privilege illusion in on-policy self-distillation as a consequence of information asymmetry between the privileged teacher and the student. The proposed DAPD framework addresses this by introducing matched-information anchoring paths (DPA) and balancing reference- and rollout-guided supervision (DSA). Extensive experiments across Qwen3 scales and benchmarks show consistent improvements over OPSD, validating the effectiveness of both components. Future directions include adaptive source trust, broader forms of privileged information, and fully reference-free training.

**Limitations.** DAPD increases training-time computation (multiple anchored distributions) though no inference overhead. Its hyperparameters may need recalibration across architectures. The verified-rollout variant requires an automatic correctness signal, which may be unavailable for open-ended tasks.

---

_Markdown view of https://picx.dev/p/Y13IMA, served by PicX — AI-generated visual whiteboard summaries of research papers._