Summary (Overview)

• Core Problem: GRPO methods for video generation (like FlowGRPO) are less reliable than their counterparts for language models and images, because the ODE-to-SDE conversion used for exploration injects excess noise that lowers rollout quality and destabilizes post-training alignment.
• Key Insight: The paper frames GRPO for video generation as a manifold-constrained exploration problem, treating the pretrained model as defining a valid video data manifold $\mathcal{M} \subset \mathbb{R}^D$, and constraining exploration within its vicinity.
• Proposed Solution: SAGE-GRPO (Stable Alignment via Exploration) applies constraints at two levels: (1) a micro-level Precise Manifold-Aware SDE with logarithmic curvature correction plus a Gradient Norm Equalizer, and (2) a macro-level Dual Trust Region with periodic moving anchors and stepwise constraints.
• Results: Evaluated on HunyuanVideo1.5 with VideoAlign reward, SAGE-GRPO achieves consistent gains over DanceGRPO, FlowGRPO, and CPS in VQ, MQ, TA, CLIPScore, and PickScore metrics.
• User Study: A human preference study with 29 evaluators shows win rates of 80–86% for visual quality, 71–79% for motion quality, and 68–79% for semantic alignment against baselines.

---

Introduction and Theoretical Foundation

Background and Motivation

Group Relative Policy Optimization (GRPO) has proven effective for aligning language models and image generation models, but its application to video generation remains unreliable. The key challenge is that video generation has a large, structured solution space, and the exploration required for GRPO is easily disturbed.

In GRPO training for video generation, deterministic ODE samplers must be converted into SDE samplers to enable diverse exploration. However, current baselines (DanceGRPO, FlowGRPO) rely on Euler-style discretization and first-order approximations when deriving the SDE noise standard deviation, introducing truncation errors that inject excess noise energy during sampling—particularly in high-noise steps.

Theoretical Foundation

The paper builds on Flow Matching and Rectified Flow frameworks. The generation process is modeled as transport along a probability path via an ODE:

$$\frac{d\mathbf{x}_t}{dt} = \mathbf{v}_\theta(\mathbf{x}_t, t) \tag{1}$$

Rectified Flow uses a linear interpolation path:

$$\mathbf{x}_t = (1 - \sigma_t)\mathbf{x}_0 + \sigma_t \mathbf{z}_1 \tag{2}$$

with velocity field:

$$\mathbf{v}_\theta(\mathbf{x}_t, t) = \frac{d\mathbf{x}_t}{dt} = -\frac{d\sigma_t}{dt}(\mathbf{x}_0 - \mathbf{z}_1) = \frac{1}{1 - \sigma_t}(\mathbf{x}_t - \mathbf{x}_0) \tag{3}$$

The GRPO objective optimizes a group-normalized advantage:

$$\mathcal{L}_{GRPO}(\theta) = -\frac{1}{G}\sum_{i=1}^{G} A_i \cdot \sum_{t=1}^{T} \log \pi_\theta(\mathbf{x}_{t-1}^{(i)} | \mathbf{x}_t^{(i)}, \mathbf{c}) \tag{4}$$

Key Theoretical Problem

The paper identifies that the pretrained video generation model defines a valid data manifold. The core problem is formulated as: how to constrain exploration within the vicinity of the data manifold so that each update improves rollouts while keeping reward evaluation reliable. Existing methods' first-order noise approximations can push $z_t$ away from $\mathcal{M}$, producing temporal jitter and artifacts.

---

Methodology

Micro-Level Exploration: Precise SDE and Gradient Equalization

1. Precise Manifold-Aware SDE

The key contribution is deriving the correct noise standard deviation $\Sigma_t^{1/2}$ during discrete SDE discretization. For a marginal-preserving SDE with diffusion coefficient $\varepsilon_t = \eta\sqrt{\sigma_t/(1-\sigma_t)}$, the variance is integrated over the interval $[\sigma_{t+1}, \sigma_t]$:

$$\Sigma_t = \int_{\sigma_{t+1}}^{\sigma_t} \varepsilon_s^2 \mathrm{d}s = \eta^2\left[-(\sigma_t - \sigma_{t+1}) + \log\left(\frac{1-\sigma_{t+1}}{1-\sigma_t}\right)\right] \tag{5}$$

Taking the square root yields the noise standard deviation:

$$\Sigma_t^{1/2} = \eta\sqrt{-(\sigma_t - \sigma_{t+1}) + \log\left(\frac{1-\sigma_{t+1}}{1-\sigma_t}\right)} \tag{6}$$

The logarithmic term $\log\left(\frac{1-\sigma_{t+1}}{1-\sigma_t}\right)$ accounts for the geometric contraction of the signal coefficient $(1-\sigma_t)$, which linear approximations fail to capture. Applying Euler-Maruyama discretization with $\Delta t = \sigma_t - \sigma_{t+1}$:

$$\mathbf{x}_{t+\Delta t} = \mathbf{x}_t + \mathbf{v}_\theta(\mathbf{x}_t, t)\Delta t + \frac{\Sigma_t}{2}\mathbf{s}_\theta(\mathbf{x}_t) + \Sigma_t^{1/2}\boldsymbol{\epsilon} \tag{7}$$

where $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ and $\mathbf{s}_\theta(\mathbf{x}_t) \approx -(\mathbf{x}_t - \hat{\mathbf{x}}_0)/\sigma_t^2$ is the score function estimate.

Table 1: Comparison of SDE noise injection strategies

2. Gradient Norm Equalizer

The diffusion process has an inherent signal-to-noise imbalance: gradient norms vary by orders of magnitude across timesteps. For a Gaussian transition $\pi(\mathbf{x}_{t-1}|\mathbf{x}_t) = \mathcal{N}(\mu_\theta, \Sigma_t \mathbf{I})$:

$$\|\nabla_{\boldsymbol{\mu}}\log\pi\| \propto \frac{1}{\Sigma_t^{1/2}} \tag{8}$$

This causes gradients to vanish at high noise ($t \to 1$) and explode at low noise ($t \to 0$). The equalizer normalizes optimization pressure:

$$S_t = \frac{\text{Median}(\{\mathcal{N}_\tau\}_{\tau=1}^{T})}{\mathcal{N}_t + \epsilon} \tag{9}$$

where $\mathcal{N}_t = \lambda_t/\Sigma_t^{1/2}$ is a per-timestep gradient scale proxy and $\epsilon$ is a small constant.

3. Group-Normalized Advantage

Each rollout is scored by a composite reward $R(\mathbf{x}_0)$, with advantage:

$$A_i = \frac{r_i - \mu_R}{\sigma_R + \epsilon} \tag{10}$$

where $r_i = R(\mathbf{x}_0^{(i)})$, $\mu_R = \frac{1}{G}\sum_{j=1}^{G} r_j$, and $\sigma_R^2 = \frac{1}{G}\sum_{j=1}^{G}(r_j - \mu_R)^2$.

Macro-Level Exploration: Dual Trust Region Optimization

KL Divergence as Dynamic Anchor

For a Gaussian policy, the KL divergence between current policy $\pi_\theta$ and reference $\pi_{ref}$ is:

$$D_{KL}(\pi_\theta || \pi_{ref}) = \mathbb{E}_{\mathbf{x}_t \sim \pi_\theta}\left[\frac{(\boldsymbol{\mu}_\theta - \boldsymbol{\mu}_{ref})^2}{2\Sigma_t^2}\right] \approx \frac{(\boldsymbol{\mu}_\theta - \boldsymbol{\mu}_{ref})^2}{2\Sigma_t^2} \tag{11}$$

Three KL Strategies Analyzed

1. Fixed KL: $\pi_{ref} = \pi_0$ (hard constraint, limits optimality—causes underfitting when optimal policy is far from initialization)

2. Step-wise KL: $\pi_{ref} = \pi_{k-1}$ (velocity constraint only, allows unbounded cumulative drift):

$$\|\nabla_\theta D_{KL}(\pi_\theta || \pi_{k-1})\| \propto \|\boldsymbol{\mu}_\theta - \boldsymbol{\mu}_{k-1}\|/\Sigma_t \tag{12}$$

3. Periodical Moving KL: $\pi_{ref} = \pi_{k-N}$ (position control via dynamic trust region):

$$D_{KL}(\pi_\theta || \pi_{ref\_N}) = \frac{(\boldsymbol{\mu}_\theta - \boldsymbol{\mu}_{ref\_N})^2}{2\Sigma_t^2} \tag{13}$$

Dual KL: Position-Velocity Controller

$$\mathcal{L}_{KL} = \beta_{pos} \cdot D_{KL}(\pi_\theta || \pi_{ref\_N}) + \beta_{vel} \cdot D_{KL}(\pi_\theta || \pi_{k-1}) \tag{14}$$

The position term prevents long-term drift; the velocity term smooths instantaneous updates. The step-wise KL is computed using log-probability differences:

$$D_{KL}(\pi_\theta || \pi_{k-1}) \approx \mathbb{E}[\log\pi_{k-1}(\mathbf{x}_{t-1}|\mathbf{x}_t) - \log\pi_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t)] \tag{15}$$

Full SAGE-GRPO Objective

$$\mathcal{L}_{SAGE-GRPO}(\theta) = -\frac{1}{G}\sum_{i=1}^{G} A_i \cdot \sum_{t=1}^{T} S_t \cdot \log\pi_\theta(\mathbf{x}_{t-1}^{(i)}|\mathbf{x}_t^{(i)}, \mathbf{c}) - \lambda_{KL} \cdot \mathcal{L}_{KL} \tag{43}$$

with adaptive KL weighting (two-stage warm-up from $\lambda_{min}=10^{-7}$ to $\lambda_{max}=10^{-5}$ followed by proportional feedback control).

---

Empirical Validation / Results

Experimental Setup

• Model: HunyuanVideo 1.5, 81 frames per video, effective batch size 8
• Reward: VideoAlign evaluator (frozen, no fine-tuning) with composite reward $R = w_{vq}S_{vq} + w_{mq}S_{mq} + w_{ta}S_{ta}$
• Baselines: DanceGRPO, FlowGRPO, CPS
• Unified framework: All methods share infrastructure, varying only the GRPO algorithm

Main Results

Table 2: Main Comparison on Video Generation Benchmarks

In Setting B (alignment-focused), SAGE-GRPO with Dual Moving KL achieves the best performance across Overall, VQ, MQ, and CLIPScore while remaining close to the best TA, demonstrating that emphasizing alignment provides a more reliable optimization target.

User Study Results

Table 3: User Preference Study (Win rates of SAGE-GRPO against baselines)

Ablation Studies

1. Temporal Gradient Equalizer: Without balancing, low-noise timesteps dominate optimization, leading to unstable or plateaued rewards. With balancing, gradient scale variation is reduced from more than one order of magnitude to within a small constant factor.

2. KL Strategy Ablation: Dual Moving KL consistently outperforms other variants (no KL, Fixed KL, Step-wise KL, Moving KL) in both convergence speed and final reward, avoiding collapse observed in aggressive step-wise updates.

3. KL Weight Sensitivity: Two-stage schedule ($10^{-7} \to 10^{-5}$) achieves the strongest and most consistent gains across VQ, MQ, and TA compared to fixed or milder schedules.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Manifold-Aware Exploration Framework: The paper formalizes GRPO for video generation as a manifold-constrained exploration problem, providing a principled geometric interpretation of why existing ODE-to-SDE conversions fail in high-noise regions.

2. Precise SDE Derivation: The logarithmic correction term $\log\left(\frac{1-\sigma_{t+1}}{1-\sigma_t}\right)$ provides a mathematically rigorous variance computation that captures the geometric contraction of the signal coefficient, which first-order approximations miss. The Taylor expansion analysis shows the leading term $\Delta\sigma\frac{\sigma_t}{1-\sigma_t}$ represents the ideal variance scaling for a geometric schedule.

3. Position-Velocity Control Analogy: The Dual Trust Region is framed as a position-velocity controller in policy space, analogous to classical control theory—position control prevents drift from the manifold, while velocity control smooths instantaneous updates.

Practical Implications

1. Stability-Plasticity Trade-off: The paper demonstrates a concrete resolution of the stability-plasticity dilemma in video RL alignment: Fixed KL constraints limit plasticity (underfitting), while relaxed constraints trigger reward hacking or catastrophic forgetting. The Dual Trust Region with periodic moving anchors provides a practical middle ground.

2. Gradient Balancing: The empirical finding that gradient norms vary by orders of magnitude across noise levels (following $\propto 1/\Sigma_t^{1/2}$) has implications for any diffusion-based RL method, suggesting that per-timestep normalization is essential for stable training.

3. Reduced Reward Hacking: The alignment-focused setting (Setting B) shows that emphasizing semantic alignment provides a more reliable optimization target, reducing reward hacking artifacts while improving overall quality.

---

Conclusion

Main Takeaways

SAGE-GRPO presents a comprehensive solution to the instability of GRPO for video generation by treating exploration as a manifold-constrained problem. The key innovations are:

1. Micro-level: A Precise Manifold-Aware SDE with logarithmic curvature correction that keeps exploration noise closer to the flow trajectory, coupled with a Gradient Norm Equalizer that balances optimization across timesteps.

2. Macro-level: A Dual Trust Region mechanism combining position control (periodic moving anchors) and velocity control (step-wise KL constraints) that prevents long-horizon drift while maintaining plasticity.

Future Directions

The paper suggests several potential extensions:

• Applying the manifold-aware exploration framework to other flow-based generative models beyond video
• Exploring adaptive schedules for the moving anchor interval $N$ and KL weights
• Investigating the interaction between reward model design and manifold-constrained exploration
• Extending the approach to other modalities where solution spaces are large and structured (e.g., 3D generation, audio generation)

Impact Statement

The method may strengthen creative tools, scientific communication, and educational content relying on controllable video synthesis. However, stronger video generation systems can exacerbate concerns about misinformation, deepfakes, and biased content, requiring safeguards such as content moderation, dataset auditing, and human oversight in deployment.