Summary (Overview)

• Core contribution: Introduces Self-Supervised Visual On-Policy Distillation (S²VOPD), a method that creates informative teacher–student asymmetry without privileged information (e.g., ground-truth labels, rewards, or stronger teacher models) by subtracting information from the student rather than adding it to the teacher.
• Key mechanism: The teacher observes the original image while the student generates trajectories from a strongly augmented view (e.g., downscaling + Gaussian noise). The predictive gap between the two distributions becomes the training signal.
• Main results: S²VOPD improves Qwen3.5-4B from 70.7% to 77.4% average accuracy on six fine-grained perception benchmarks—surpassing all open-source models up to Qwen3-VL-235B and GPT-5.4, while matching Qwen3.5-397B.
• Design insights: (1) Asymmetry matters—all four augmentation families improve performance while symmetric self-distillation degrades it; (2) strength matters—performance peaks at moderate teacher–student gaps; (3) the gap must remain task-consistent—augmentations that remove question-relevant evidence produce large but uninformative discrepancies.
• Versatility: S²VOPD improves both fine-grained perception (+5.7%) and math reasoning (+3.7%) at 4B scale, recovering 96% of the improvement achieved by privileged-information methods while using no external annotations.

---

Introduction and Theoretical Foundation

Background

On-policy distillation (OPD) improves a student policy by providing dense teacher supervision along the student's own generated trajectories. The central requirement is an informative teacher–student asymmetry: the teacher must possess information or capability enabling it to provide better predictive targets than the student itself.

The Problem

Conventional OPD obtains asymmetry from:

• A larger/stronger teacher model, or
• Privileged supervision such as reference solutions, environmental feedback, or ground-truth regions of interest.

This raises the fundamental question: where can informative asymmetry come from when nothing privileged is available?

Key Theoretical Insight

The authors invert the direction of asymmetry: instead of adding information to the teacher, they withhold information from the student. The teacher observes the original input, while the student operates on a degraded view. Their difference in available visual evidence induces a predictive discrepancy—creating the required asymmetry for free.

This draws inspiration from self-supervised learning (e.g., SimCLR, BYOL, DINO), where augmentation defines invariances enabling learning without external labels. However, S²VOPD uses augmentation for a different purpose: not representation-level alignment, but token-level generative distillation where the student-view transformation is the sole source of the teacher's informational advantage.

---

Methodology

Core Objective

Given an image–question pair $(x, q) \sim \mathcal{D}$, let $\pi_\theta$ be the student policy and $\pi_\phi$ its exponential-moving-average (EMA) teacher: $\phi \leftarrow (1-\eta)\phi + \eta\theta$.

For each sample, a stochastic transformation $T \sim \mathcal{T}$ produces the student view $\tilde{x} = T(x)$, while the teacher retains the clean view $x$. The student samples $n$ rollouts conditioned on its corrupted view. The training objective minimizes:

$$\mathcal{L}(\theta) = \mathbb{E}_{(x, q) \sim \mathcal{D}} \mathbb{E}_{y \sim \pi_{\theta}(\cdot | \tilde{x}, q)} \left[ \frac{1}{|y|} \sum_{t=1}^{|y|} D \bigl(\pi_{\phi}(\cdot | x, q, y_{<t}) \| \pi_{\theta}(\cdot | \tilde{x}, q, y_{<t}) \bigr) \right]$$

The inner expectation makes the objective on-policy: prefixes $y_{<t}$ are drawn from the student's own policy.

Divergence Measure

The divergence $D$ is instantiated as the generalized Jensen–Shannon divergence $D_{JS}^{\alpha}$:

$$\begin{aligned} m_t &= \alpha \pi_{\phi}(\cdot | x, q, y_{<t}) + (1-\alpha) \pi_{\theta}(\cdot | \tilde{x}, q, y_{<t}), \\ D_{JS}^{\alpha}(\pi_{\phi} \| \pi_{\theta}) &= \alpha D_{KL}(\pi_{\phi} \| m_t) + (1-\alpha) D_{KL}(\pi_{\theta} \| m_t) \end{aligned}$$

With $\alpha = 0.5$, this interpolates between forward and reverse KL divergence and remains bounded when distributions have limited overlap. Both distributions are restricted to the teacher's top-k tokens and renormalized.

Augmentation Space

The transformation $T$ is specified by three components: primitive operators with probabilities and strengths, a composition policy, and a global probability $p$ of applying augmentation. Formally:

$$z \sim \text{Bernoulli}(p), \quad b_j \sim \text{Bernoulli}(\rho_j)$$ $$T(x) = \begin{cases} x, & z = 0, \\ (A_M^{b_M}(\cdot; \lambda_M) \circ \dots \circ A_1^{b_1}(\cdot; \lambda_1))(x), & z = 1, \end{cases}$$

Four augmentation families are explored:

Best Recipe

The strongest configuration composes two information-reduction operators:

$$T(x) = A_{\text{noise}}(A_{\text{down}}(x; s); \lambda_{\text{noise}}), \quad s \sim \mathcal{U}(0.3, 0.6)$$

Downscaling is applied to every sample (without resize-back, producing fewer visual tokens), while Gaussian noise follows a DDPM forward step at $t=200$ ($\sigma \approx 0.11$) with probability $\rho = 0.5$.

---

Empirical Validation / Results

Main Results (Table 2)

S²VOPD trained on a 12K FineVision subset improves Qwen3.5-4B from 70.68% to 77.44% average accuracy across six perception benchmarks. This outperforms:

• All open-source models up to Qwen3-VL-Instruct-235B (75.75%)
• GPT-5.4 (72.77%)
• All privileged-information methods including Vision-OPD (77.07%) and ZwZ variants

Fair Comparison (Table 3)

When trained on the same Vision-OPD-6K data for 65 steps:

S²VOPD achieves the best average among all methods without privileged info at both scales, and exceeds all privileged-supervised baselines at 4B.

Key Findings on Augmentation Design

1. All augmentation families are effective (Figure 2a):

   • Information reduction: 75.65%
   • Photometric: 74.40%
   • Geometric: 74.30%
   • Occlusion: 72.44%
   • Symmetric self-distillation (no augmentation): 65.21% (degrades!)

2. Strength matters: Performance peaks at a JS divergence of ~0.014 between teacher and student distributions, then declines beyond it.

3. Task consistency is critical: Cropping degrades monotonically with strength (71.53% → 68.76% → 67.44%) because it may remove question-relevant evidence entirely.

Ablations

S²VOPD is robust to teacher decay rates (0.95–0.999, within 0.8% variation) and benefits from balanced divergence (JSD > reverse KL > forward KL).

---

Theoretical and Practical Implications

Theoretical Contributions

1. Inverting asymmetry source: Demonstrates that informative teacher–student asymmetry need not come from privileged information—subtracting information from the student creates an equivalent learning signal for free.

2. Augmentation as distillation design space: Provides the first controlled, large-scale search of augmentation space for OPD, establishing principles (asymmetry, moderate strength, task-consistency) that govern effective self-supervised distillation.

3. Divergence selection insight: Shows that asymmetric-view distillation requires a balanced divergence (JSD) that neither over-imitates inaccessible details (forward KL) nor discards corrective signals (reverse KL).

Practical Implications

1. Cost-effective scaling: A 4B model can match or exceed models 50–100× larger, reducing inference and training costs.

2. No annotation burden: Eliminates the need for ground-truth regions, reference answers, or external rewards—particularly valuable as model capabilities outpace human supervision quality.

3. Dual improvement: Simultaneously improves perception and reasoning, unlike privileged methods that trade one for the other.

4. Computational efficiency: Lower-resolution student inputs reduce both rollout and forward-pass costs.

---

Conclusion

S²VOPD creates informative teacher–student asymmetry without privileged annotations by subtracting task-relevant information from the student view. The teacher observes the original image while the student learns from a degraded view, turning the model's own predictions into a perception-aligned training signal.

Key takeaways:

• Augmentation must remove enough task-relevant evidence to create a useful learning signal, but not so much that it changes the underlying question or makes recovery impossible.
• Downscaling with Gaussian noise offers the most robust default across benchmarks.
• The method improves both fine-grained perception and mathematical reasoning, distinguishing it from privileged-supervision methods (which improve perception but weaken reasoning) and self-rewarding RL methods (which preserve reasoning but provide weaker perceptual gains).

Future directions likely include exploring augmentation design for other modalities (e.g., audio, video), extending to language-only tasks, and investigating the interaction between augmentation semantics and task-specific perceptual requirements.