Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation

Summary (Overview)

• Key Problem: Existing Distribution Matching Distillation (DMD) methods for streaming video generation treat all student rollouts, frames, and pixels as equally reliable supervision, ignoring variance in the supervision signal's quality.
• Core Insight: The DMD supervision signal exhibits two complementary axes of variance: Inter-Reliability (variation in supervision reliability across different student rollouts) and Intra-Perplexity (variation across spatiotemporal regions within a rollout in how much further refinement can improve quality).
• Main Solution: Stream-R1, a framework that adaptively reweights the DMD objective using a single reward-guided mechanism at both the rollout level (Inter-Reliability weighting) and the spatiotemporal-element level (Intra-Perplexity weighting).
• Technical Contribution: Derives a per-pixel gradient saliency map from a pretrained video reward model, factorizes it into spatial and temporal weights, and combines it with an exponential rollout-level reward weight. Includes adaptive balancing across multiple quality dimensions (visual quality, motion quality, text alignment).
• Results: Achieves consistent improvements over distillation baselines on standard benchmarks across all quality dimensions, surpasses its multi-step teacher model in some metrics, and demonstrates superior temporal stability in long video generation, with no architectural changes or inference cost overhead.

Introduction and Theoretical Foundation

Recent video diffusion models achieve high quality but suffer from prohibitive inference costs due to multi-step denoising over fixed windows. Autoregressive streaming video diffusion models offer a solution by enabling causal, frame-by-frame generation. To make this practical, Distribution Matching Distillation (DMD) is used to compress a multi-step teacher into a few-step student by minimizing the output distribution divergence.

However, existing DMD methods apply indiscriminative distillation: they minimize the per-instance discrepancy between student and teacher outputs with uniform weight for every rollout, frame, and pixel. This approach overlooks two critical sources of variance in the DMD supervision signal:

1. Inter-Reliability: The DMD gradient $g = f_{\text{fake}} - f_{\text{real}}$ is an estimate whose reliability varies across rollouts. The teacher-derived $f_{\text{real}}$ is a conditional denoiser. When a student rollout is near the teacher's high-quality mode, $g$ points toward closing the residual gap. When a rollout is far from this mode, $g$ encodes a "within-low-quality refinement" rather than a path to high quality. Averaging $g$ equally conflates these regimes.
2. Intra-Perplexity: Within a single rollout, different spatial regions and temporal frames contribute unequally to potential quality improvement. Some regions are far from the high-quality mode (high perplexity), while others are near their local optimum (low perplexity). A uniform loss wastes optimization budget on saturated regions and under-supervises high-perplexity regions.

The distillation objective should therefore address two complementary questions: whether to learn from each rollout (based on reliability) and where to concentrate optimization within each rollout (based on perplexity).

Methodology

Stream-R1 is built upon the DMD framework. Given a text prompt $c$, the student generator $G_\phi$ produces a clean latent $x_0 = G_\phi(c)$. A noisy version $x_t$ is created by adding noise at timestep $t$. Critic networks $f_{\text{real}}$ and $f_{\text{fake}}$ estimate score functions. The base DMD gradient and loss are:

$$g = f_{\text{fake}}(x_t, c) - f_{\text{real}}(x_t, c)$$ $$L_{\text{DMD}} = \frac{1}{2} \| x_0 - \text{sg}(x_0 - \hat{g}) \|^2$$

where $\hat{g}$ is the normalized gradient and $\text{sg}(\cdot)$ is the stop-gradient operator.

Stream-R1 modulates this loss with two adaptive weights derived from a single pretrained video reward model.

1. Inter-Reliability Weighting ($W_{\text{inter}}$): A pretrained reward model evaluates the student-generated rollout $V$, aggregating per-dimension rewards ${r_d}_{d \in D}$ into a balanced final reward $r_{\text{final}}$ (defined later). This score proxies supervision reliability. The per-sample loss multiplier is:

$$W_{\text{inter}} = \exp(\beta \cdot r_{\text{final}})$$

where $\beta > 0$ is a temperature. Rollouts with higher rewards (more reliable supervision) dominate the gradient.

2. Intra-Perplexity Weighting ($W_{\text{intra}}$): A per-element weight map localizes optimization pressure where refinement yields the largest expected gain. For a quality dimension $d \in \{ \text{VQ}, \text{MQ}, \text{TA} \}$, the reward model $R_d$ maps video $V \in \mathbb{R}^{F \times H \times W \times 3}$ to a scalar $r_d = R_d(V)$. The per-axis saliency map is the absolute gradient:

$$S^{(d)} = \left| \frac{\partial R_d(V)}{\partial V} \right| \in \mathbb{R}^{F \times H \times W}$$

Adaptive Gradient-Saliency Combination: Saliency maps from different dimensions are combined adaptively, allocating greater attention to dimensions with lower current scores (more room for improvement):

$$\alpha_d = \frac{\exp(-r_d / \tau)}{\sum_{d'} \exp(-r_{d'} / \tau)}, \quad S_{\text{combined}} = \sum_d \alpha_d \cdot S^{(d)}$$

where $\tau$ is a temperature parameter.

Spatiotemporal Saliency Decomposition: The combined saliency $S_{\text{combined}}$ is factored into temporal and spatial components.

• Temporal Weight Extraction: Per-frame saliency $p_f = \frac{1}{HW} \sum_{h,w} S_{\text{combined}}[f, h, w]$ is normalized and clamped: $$\hat{p}_f = \frac{p_f - p_{\text{min}}}{p_{\text{max}} - p_{\text{min}}}, \quad w^{(t)}_f = \max(\hat{p}_f, \tau_{\text{min}})$$ The weights are mean-normalized: $\frac{1}{F} \sum_f w^{(t)}_f = 1$.
• Spatial Weight Extraction: Per-frame, spatial saliency is normalized independently: $$\hat{s}_{f,h,w} = \frac{s_{f,h,w} - s^{\text{min}}_{f}}{s^{\text{max}}_{f} - s^{\text{min}}_{f}}, \quad w^{(s)}_{f,h,w} = \max(\hat{s}_{f,h,w}, \sigma_{\text{min}})$$ Each frame's spatial weights are independently mean-normalized to 1.
• Composition: The final per-element weight map is: $$W_{\text{intra}}[f, h, w] = \frac{w^{(t)}_f \cdot w^{(s)}_{f,h,w}}{\frac{1}{FHW} \sum_{f',h',w'} w^{(t)}_{f'} \cdot w^{(s)}_{f',h',w'}}$$

3. Balanced Multi-Dimensional Reward: To prevent unbalanced improvement, a balance penalty based on the standard deviation of improvements across dimensions is subtracted from the base reward: A sliding window of size $N$ tracks per-dimension reward history. Improvement for dimension $d$ is:

$$\Delta_d = \bar{r}^{\text{recent}}_{d} - \bar{r}^{\text{baseline}}_{d}$$

The balance penalty is:

$$P_{\text{bal}} = \text{std}( \{ \Delta_d \}_{d \in D} )$$

The final balanced reward is:

$$r_{\text{final}} = \frac{1}{|D|} \sum_d r_d - \lambda \cdot P_{\text{bal}}$$

where $\lambda$ is a weighting coefficient. The penalty is activated after a warmup period.

4. Overall Objective: The final Stream-R1 generator loss combines all components:

$$L_{\text{Stream-R1}} = \frac{1}{2} W_{\text{inter}} \cdot \text{mean}\left( W_{\text{intra}} \odot \| x_0 - \text{sg}(x_0 - \hat{g}) \|^2 \right)$$

where $\odot$ denotes element-wise multiplication.

Empirical Validation / Results

Implementation Details: Built upon Reward Forcing [25]. Uses Wan2.1-T2V-1.3B as student and Wan2.1-T2V-14B as teacher. Generates 5-second videos at 832×480 resolution. Training: 1,000 steps on 8 A100 GPUs.

Short Video Generation (5-second): Benchmarked against various open-source models on 946 VBench prompts.

• Stream-R1 achieves the highest Total score (84.40) among all compared methods, surpassing the multi-step teacher Wan2.1 (84.26) and Reward Forcing (84.13).
• It attains the highest Quality score (85.14) among autoregressive/streaming models.
• It achieves the best Semantic score (81.44) across all methods.
• Compared to Reward Forcing, Stream-R1 improves Total (+0.27), Quality (+0.30), and Semantic (+0.12).

Long Video Generation (10s to 180s): Stream-R1 outperforms Reward Forcing on all six VBench metrics (subject/background consistency, imaging quality, motion smoothness, aesthetic quality, drift) across all durations. The performance gap widens as video length increases, indicating superior temporal stability.

VLM-based Evaluation (Qwen3-VL): Evaluated on 60-second videos.

Stream-R1 attains the highest Visual Quality and Text Alignment scores.

Human Preference Evaluation (50 long videos): Stream-R1 is preferred over Reward Forcing on all five judged dimensions.

Ablation Study:

• Adding spatial saliency improves Quality and Long Total.
• Balanced multi-dimensional reward improves Semantic.
• Incorporating temporal decomposition produces the largest single improvement (+0.72 on Short Total, reduces Drift).
• An excessively high temporal floor ($\tau_{\text{min}}=0.40$) degrades performance, showing the importance of temporal contrast.

Visualization of Spatiotemporal Weights: A controlled experiment with injected Gaussian blur shows that the saliency map concentrates on corrupted regions and semantically/dynamically important content. Temporal weights grow monotonically as the degraded area expands, confirming that Stream-R1's weights reflect localized quality deficiency.

Theoretical and Practical Implications

• Theoretical: Reformulates DMD-based distillation as a reliability-perplexity aware process. Highlights that indiscriminative treatment caps quality upper bounds. Provides a principled framework for decomposing global reward signals into spatiotemporally localized guidance.
• Practical: Demonstrates that reward-guided distillation can push a student model beyond the teacher's quality frontier while achieving 30× higher inference speed. Offers a method to improve temporal stability and reduce quality drift in long video generation. The framework requires no architectural modification to the student and adds no additional inference cost. The principle of reward decomposition is broadly applicable to other modalities (image synthesis, 3D content generation).

Conclusion

Stream-R1 introduces a dynamic spatiotemporal reward-guided distillation framework that addresses Inter-Reliability and Intra-Perplexity variance in DMD supervision. By decomposing scalar reward signals into factored spatial and temporal saliency via reward-model gradient backpropagation, it concentrates optimization on quality-deficient regions and frames. Experiments show Stream-R1 achieves state-of-the-art performance on short and long video generation benchmarks, surpassing its teacher model in some metrics and demonstrating superior temporal stability. Human and VLM evaluations confirm balanced quality improvements. The work opens a new direction for reward-guided generation across modalities.