Summary (Overview)

• Identifies the historical context-gradient gap in frozen-cache Self Forcing: future losses supervise how noisy tokens read cached history, but cannot propagate into the clean-timestep computation that writes self-generated latents into K/V memory. This missing credit-assignment path causes identity drift, scene breaks, and layout inconsistency during long extrapolation.
• Proposes Self Gradient Forcing (SGF), a two-pass training strategy that restores the missing memory-writing supervision without full rollout backpropagation. Pass 1 performs a no-gradient serial autoregressive rollout and records the self-generated context and noisy latents at a sampled denoising exit step. Pass 2 discards the rollout cache and performs a bounded parallel reconstruction where the recorded context is reprocessed (stop-gradient) and gradients flow through the clean-context K/V formation and future-to-context causal attention.
• Achieves substantially improved long-video extrapolation (60s and 240s) across frame-wise and chunk-wise generation under multiple initializations (TF, causal CD, causal ODE), with consistent gains in subject consistency, background consistency, aesthetic quality, motion smoothness, and flickering. At 5s, SGF matches Self Forcing quality.
• Demonstrates training feasibility with only modest overhead (peak memory 87.01GB vs 79.01GB, runtime 11.71s vs 10.39s per 5 steps) compared to frozen-cache Self Forcing, while direct differentiable-cache training runs out of memory (OOM).
• Validates with human preference study: GSB scores are all positive (up to 48.7%) across 1,900+ paired judgments, indicating consistent preference for SGF over Self Forcing in long-horizon generation.

Introduction and Theoretical Foundation

Background

Long-form autoregressive video generation requires scenes, objects, and dynamics to remain coherent far beyond the training window. In autoregressive video diffusion, each new frame or chunk is conditioned on the prompt and previously generated content. However, Teacher-Forcing (TF) training conditions on ground-truth video contexts, while inference conditions on self-generated histories, creating exposure bias.

Self Forcing (Huang et al., 2025) mitigates this mismatch by training the student on histories produced by its own autoregressive rollout, using distribution-matching distillation (DMD) supervision from a bidirectional video model. This self-rollout recipe has become a common training paradigm. However, to make self-rollout training feasible, these methods typically truncate cross-chunk gradient flow through the historical KV cache.

The Historical Context-Gradient Gap

During rollout, generated latents are processed by the causal DiT at the clean context timestep $t_{\text{ctx}} = 0$ and stored as a causal KV cache. Later chunks read this cache as frozen historical context. Future losses can train noisy denoising tokens to read cached history, but cannot propagate into the clean-timestep computation that wrote the historical K/V entries. The authors call this missing credit-assignment path the historical context-gradient gap.

Formally, let $C_\theta$ denote the cache-writing computation at $t_{\text{ctx}} = 0$. In serial rollout:

$$\text{KV}_i^0(\theta) = C_\theta\left(\tilde{x}_i, t_{\text{ctx}}; \text{KV}_{<i}^0\right), \quad t_{\text{ctx}} = 0$$

where $\tilde{x}_i$ is the predicted clean latent for block $i$. The new entry $\text{KV}_i^0$ becomes part of the state used by later blocks. If future losses could differentiate through this historical entry, they would assign credit to the clean-context computation that encoded $\tilde{x}_i$ into K/V memory.

Frozen-cache Self Forcing removes this signal. The historical K/V entries are treated as detached rollout state. Consequently, future losses do not supervise the $t_{\text{ctx}} = 0$ computation that produced those entries. This becomes problematic because the same DiT parameters are shared across noisy denoising and clean-context cache writing. DMD losses at noisy timesteps update the shared parameters:

$$\theta^{r+1} = \theta^r - \eta \nabla_\theta \mathcal{L}_{\text{SF}}(\theta^r), \quad \text{KV}_i^0(\theta^{r+1}) \not\equiv \text{KV}_i^0(\theta^r) \text{ in general}$$

Thus, training can change the clean-context cache writer, but frozen-cache Self Forcing provides no future-loss correction for how self-generated latents are encoded into K/V memory.

Direct Differentiable Cache (Impractical)

A direct solution would keep the historical KV cache differentiable, allowing future losses to backpropagate into the clean-timestep K/V writing computation. However, this requires retaining the autograd graph for every historical cache write until all later blocks that consume it have been processed. The graph grows with rollout length, transformer depth, and sequential cache updates, making direct differentiable-cache training impractical for long-horizon self-rollout.

Methodology

Self Gradient Forcing (SGF)

SGF is a two-pass training strategy that turns the serial graph-retention problem into a bounded parallel recomputation problem.

Pass 1: No-Gradient Self-Rollout

The first pass is the ordinary serial autoregressive rollout used at inference. For each block $i$, the model denoises with the current historical cache and records a sampled exit state: the noisy input $z^{t_\star}_i$ and the predicted clean latent $\tilde{x}_i$. The latent $\tilde{x}_i$ is then processed at $t_{\text{ctx}} = 0$ to update the serial K/V cache for later blocks. All computations are run without gradient tracking, and the recorded states are treated as fixed data for Pass 2.

Pass 2: Parallel Context-Gradient Reconstruction

SGF keeps Pass 1 and the reconstruction geometry unchanged but removes the stop-gradient boundary on the reconstructed context K/V path. The context latents $\tilde{X}_{\text{ctx}}$ themselves remain stop-gradient inputs, so SGF does not optimize the sampled rollout trajectory. Instead, the model re-encodes these fixed self-generated latents at $t_{\text{ctx}} = 0$, and the resulting K/V entries remain differentiable when future target tokens attend to them.

The reconstruction is:

$$\hat{X}_{\text{tar}} = G_\theta\left(Z^\star, t^\star; \tilde{X}_{\text{ctx}}, t_{\text{ctx}}, M_{\text{rec}}\right)$$

where $M_{\text{rec}}$ is the causal reconstruction mask that reproduces the sink-plus-window attention relation induced by the serial cache.

The gradient flow is:

$$\nabla_\theta \mathcal{L}_{\text{DMD}}(\hat{X}_{\text{tar}}) \supset \frac{\partial \mathcal{L}_{\text{DMD}}}{\partial \text{KV}_{\text{rec}}^{\text{ctx}}} \frac{\partial \text{KV}_{\text{rec}}^{\text{ctx}}}{\partial \theta}$$

Therefore, future DMD losses supervise both target-side denoising and clean-context K/V writing.

Algorithm

The full SGF training step is summarized in Algorithm 1 (see paper). Key steps:

1. Sample exit index $s \sim \text{Uniform}\{1,\dots,K\}$ and set $t^\star \leftarrow t_s$.
2. Pass 1: Disable gradients. For each block $i$ from 1 to $N$, denoise from $t_1$ to $t_s$ using current cache, record $z^{t_\star}_i$ and $\tilde{x}_i$, then update cache with $C_\theta(\tilde{x}_i, t_{\text{ctx}}; \text{KV})$.
3. Pass 2: Enable gradients. Set $X_{\text{rec}} \leftarrow \text{sg}(X_{\text{ctx}})$. Compute $\hat{X}_{\text{tar}} \leftarrow G_\theta(Z^\star, t^\star; X_{\text{rec}}, t_{\text{ctx}}, M_{\text{rec}})$. Update $\theta$ with $\mathcal{L}_{\text{DMD}}(\hat{X}_{\text{tar}})$, keeping context-K/V gradients.

Gradient Boundary

SGF is a bounded reconstruction of the sampled exit computation, not full rollout BPTT. The recorded context latents, noise samples, scheduler states, and Pass-1 serial cache trajectory are all stop-gradient. Gradients flow only through the Pass-2 reconstruction: the clean-context forward, K/V projections, future-to-context attention, and target-side denoising computation.

Streaming Context Policy

For frame-wise streaming generation, both Self Forcing and SGF use the same sink-plus-FIFO context policy. The authors use a fixed sink prefix and a FIFO window of recent latents. For the Wan video VAE, four sink latents are used to preserve the temporal boundary prefix induced by its asymmetric grouping pattern (1-frame boundary group followed by 4-frame groups).

Empirical Validation / Results

Experimental Setup

• Training window: 5 seconds for all models.
• Evaluation: 5s (VBench), 60s (VBench-Long prompts), 240s (128 MovieGen prompts).
• Matched pairs: SGF vs Self Forcing share the same initialization, prompt set, random seed, sink/FIFO policy, sliding window, chunking strategy, and sampling configuration.
• Frame-wise: sink 4, total window 21, FIFO 16, current chunk 1.
• Chunk-wise: sink 3, total window 12, FIFO 6, current chunk 3, chunk size 3.

Long Video Generation (60s and 240s)

Frame-wise Results (Table 1)

SGF improves over Self Forcing on most quality and consistency metrics across both horizons. Dynamic degree is the main exception (Self Forcing can be higher), but this is because Self Forcing rollouts often contain scene jumps, broken camera geometry, and object deformation that create incoherent apparent motion.

Chunk-wise Results (Table 2)

Similar pattern: SGF improves aesthetics, background, imaging, motion, subject, and flickering over matched Self Forcing baselines under causal CD and TF initializations.

User Study (Table 3)

Blind GSB preference study with >1,900 paired judgments:

All scores are positive, indicating consistent preference for SGF over Self Forcing.

Training Feasibility (Table 4)

SGF adds modest overhead (10% peak memory, 13% runtime) while direct differentiable-cache runs out of memory.

Two-Pass Recovery Fidelity (Table 7)

Pass 2 closely reproduces Pass-1 exit predictions across denoising steps:

Overall relative $\ell_2$ error is 1.41%, and cosine similarity is 0.999886, consistent with accumulated bf16 roundoff rather than substantive mismatch.

Sink Ablation (Table 8)

For frame-wise SGF with TF initialization at 60s:

Sink 4 reaches the stable range suggested by the VAE boundary diagnostic; sink 1 is weaker on aesthetics and flickering.

Theoretical and Practical Implications

• Theoretical contribution: Identifies and formalizes the historical context-gradient gap