Summary (Overview)

• Problem identified: In frozen-cache Self Forcing for autoregressive video diffusion, future losses can supervise how noisy denoising tokens read cached history, but cannot supervise how earlier generated latents are written into the key-value (KV) cache. The authors call this the historical context-gradient gap.
• Proposed solution: Self Gradient Forcing (SGF), a two-pass training strategy. Pass 1 performs a no-gradient serial autoregressive rollout matching inference, recording self-generated context latents and noisy exit states. Pass 2 performs a bounded parallel reconstruction where the model re-encodes the self-generated context at the clean context timestep ($t_{ctx}=0$), allowing future DMD losses to backpropagate through the context KV representations and future-to-context causal attention.
• Key results: Across frame-wise and chunk-wise generation, multiple initializations (TF, causal ODE, causal CD), and horizons of 5s, 60s, and 240s, SGF matches Self Forcing at short horizons (5s) and substantially improves long-video extrapolation at 60s and 240s, particularly in subject identity, background/layout consistency, and temporal stability.
• Human evaluation: A blind GSB preference study with over 1,900 paired judgments shows consistent preference for SGF over Self Forcing (all scores positive, ranging from 29.6% to 48.7%).
• Training feasibility: SGF avoids the memory blow-up of direct differentiable-cache training (which runs OOM), with modest overhead: peak memory increases from 79.01GB to 87.01GB, and runtime from 10.39s to 11.71s per five training steps.

---

Introduction and Theoretical Foundation

Background and Motivation

Long-form autoregressive video generation requires scenes, objects, layouts, and dynamics to remain coherent as generation extends far beyond the training window. In autoregressive video diffusion, coherence depends on extrapolating from the model's generated history—each new frame or chunk is conditioned on the prompt and previously generated content.

Teacher-Forcing (TF) trains the model on ground-truth video contexts, while inference conditions on self-generated histories, creating exposure bias. Self Forcing (Huang et al., 2025) mitigates this by training the student on histories produced by its own autoregressive rollout, using distribution-matching distillation (DMD) supervision from a bidirectional video model.

The Historical Context-Gradient Gap

The core theoretical contribution is identifying a critical limitation in frozen-cache Self Forcing. During rollout, generated latents are processed by the causal DiT at the clean context timestep ($t_{ctx}=0$) and stored as a causal KV cache. Later chunks read this cache as frozen historical context. The consequence:

"Future losses can therefore train noisy denoising tokens to read cached history, but cannot propagate into the clean-timestep computation that wrote the historical K/V entries."

This creates a cache-writing gap for self-generated histories. The problem is exacerbated because the same causal DiT shares parameters across timesteps: updates from noisy denoising steps can alter the clean $t_{ctx}=0$ cache-writing computation, while later-chunk losses do not backpropagate to the historical KV entries it wrote. The context-writing path can therefore drift from what long autoregressive rollout requires.

Mathematically, the cache-writing computation is recurrent:

$$\mathsf{KV}_{i}^{0}(\theta) = \mathcal{C}_{\theta}\big(\tilde{x}_{i}, t_{\mathrm{ctx}}; \mathsf{KV}_{<i}^{0}\big), \qquad t_{\mathrm{ctx}} = 0. \tag{1}$$

When DMD losses update parameters at noisy timesteps:

$$\theta_{r+1} = \theta_{r} - \eta \nabla_{\theta} \mathcal{L}_{\mathrm{SF}}(\theta_{r}), \quad \mathrm{KV}_{i}^{0}(\theta_{r+1}) \not\equiv \mathrm{KV}_{i}^{0}(\theta_{r}) \quad \text{in general.} \tag{2}$$

Thus, training changes the clean-context cache writer, but frozen-cache Self Forcing provides no future-loss correction for how self-generated latents are encoded into K/V memory.

Why Direct Solutions Fail

A direct solution—keeping the historical KV cache differentiable—would require retaining the autograd graph for every historical cache write until all later blocks consume it. These graphs grow with rollout length, transformer depth, and sequential cache updates, making the direct KV-gradient path computationally impractical. The memory cost scales as:

$$M_{\mathrm{direct}} \gtrsim M_{\mathrm{cache}} + \sum_{i=1}^{T} M_{\mathrm{KVformation}}(i) + M_{\mathrm{savedattention}}, \tag{7}$$

which grows with rollout length rather than remaining a fixed-window computation.

---

Methodology

Self Gradient Forcing (SGF)

SGF is a two-pass training strategy that turns the serial graph-retention problem into a bounded parallel recomputation problem.

Pass 1: No-Gradient Self-Rollout

The first pass is the ordinary serial autoregressive rollout used at inference, run without gradient tracking:

• For each block $i$, the model denoises with the current historical cache
• Records a sampled exit state: the noisy input $z_i^{t^\star}$ and the predicted clean latent $\tilde{x}_i$
• The latent $\tilde{x}_i$ is processed at $t_{ctx}=0$ to update the serial K/V cache for later blocks
• Recorded states are treated as fixed data for Pass 2

Pass 2: Parallel Context-Gradient Reconstruction

Pass 2 discards the rollout cache and reconstructs the same exit-step computation in parallel:

$$\hat{X}_{\mathrm{tar}} = G_{\theta}\left(Z^{\star}, t^{\star}; \tilde{X}_{\mathrm{ctx}}, t_{\mathrm{ctx}}, \mathcal{M}_{\mathrm{rec}}\right). \tag{3}$$

The key difference from frozen-cache Self Forcing: the stop-gradient boundary on the reconstructed context K/V path is removed. The context latents $\tilde{X}_{\mathrm{ctx}}$ remain stop-gradient inputs (so SGF does not optimize the sampled rollout trajectory), but the model re-encodes these fixed self-generated latents at $t_{ctx}=0$, and the resulting K/V entries remain differentiable when future target tokens attend to them:

$$\nabla_{\theta} \mathcal{L}_{\mathrm{DMD}}(\hat{X}_{\mathrm{tar}}) \supset \frac{\partial \mathcal{L}_{\mathrm{DMD}}}{\partial \mathsf{KV}_{\mathrm{ctx}}^{\mathrm{rec}}} \frac{\partial \mathsf{KV}_{\mathrm{ctx}}^{\mathrm{rec}}}{\partial \theta}. \tag{4}$$

Therefore, future DMD losses supervise both target-side denoising and clean-context K/V writing.

Algorithm Summary

The training loop (Algorithm 1) samples an exit index $s \sim \text{Uniform}\{1, \ldots, K\}$, performs the no-gradient rollout in Pass 1, then enables gradients for the bounded Pass-2 reconstruction using a causal mask $\mathcal{M}_{\text{rec}}$ that reproduces the sink-plus-window attention relation induced by the serial cache.

Gradient Boundary

SGF is a bounded reconstruction of the sampled exit computation, not full rollout BPTT:

• Stop-gradient: recorded context latents, noise samples, scheduler states, and Pass-1 serial cache trajectory
• Gradient-enabled: Pass-2 clean-context forward, K/V projections, future-to-context attention, and target-side denoising computation

This recovers memory-writing supervision while keeping the training graph fixed-window and parallel.

Streaming Context Policy

For frame-wise streaming generation, SGF uses the same sink-plus-FIFO context policy as Self Forcing (sink 4, total window 21, FIFO 16, current chunk 1), isolating the effect of SGF rather than context selection. For chunk-wise generation: sink 3, total window 12, FIFO 6, current chunk 3, chunk size 3. The Wan video VAE uses four sink latents to preserve the temporal boundary prefix induced by its asymmetric grouping pattern.

---

Empirical Validation / Results

Experimental Setup

• All models trained with a 5-second training window; 60s and 240s results test native extrapolation beyond the training horizon
• Matched Self Forcing baselines share the same initialization, prompt set, random seed, sink/FIFO policy, sliding window, chunking strategy, and sampling configuration
• 60s setting uses VBench-Long prompts; 240s setting uses 128 randomly sampled MovieGen prompts
• Metrics: aesthetic quality, background consistency, dynamic degree, imaging quality, motion smoothness, subject consistency, flickering

Long-Horizon Results (60s and 240s)

Frame-wise results (Table 1, selected metrics):

Chunk-wise results (Table 2, TF init, selected metrics):

Key observation on dynamic degree: Self Forcing can obtain a higher dynamic-degree score, but this does not indicate better motion quality—long Self Forcing rollouts often contain scene jumps, broken camera geometry, and object deformation, creating large but incoherent apparent motion. SGF maintains more stable image quality and plausible camera evolution.

Human Preference Study (GSB)

Table 3 reports blind GSB preference scores ($(G-B)/(G+S+B) \times 100\%$):

All scores are positive, indicating raters consistently prefer SGF over Self Forcing.

Training Feasibility

Table 4 compares training variants:

Direct differentiable-cache training runs out of memory; SGF restores context K/V gradients with modest overhead using FlexAttention with a compiled static block-sparse causal mask.

Two-Pass Recovery Fidelity

Table 7 verifies Pass 2 faithfully reproduces Pass 1 exit predictions:

The relative $\ell_2$ error (1.41%) is close to the bf16 numerical scale (1.80× $\epsilon_{bf16}$), consistent with accumulated floating-point roundoff rather than substantive mismatch.

Sink Ablation

Table 8 (frame-wise SGF, TF init, 60s):

Sink 4 is selected as the smallest sink covering the VAE boundary-transition prefix (relative L2 error drops from 0.607 for fresh stream to 0.060 at W=4 previous latent groups).

---

Theoretical and Practical Implications

Theoretical Contributions

1. Identification of a novel training gap: The paper formalizes the historical context-gradient gap—the missing credit-assignment path where future losses cannot supervise the clean-timestep computation that writes self-generated history into K/V memory. This is distinct from exposure bias (addressed by Self Forcing) and initialization mismatch (addressed by Causal Forcing).

2. Bounded gradient recovery: SGF demonstrates that the missing memory-writing supervision can be recovered through a bounded parallel recomputation rather than full rollout BPTT, with memory scaling:

   $$M_{SGF} < M_{\mathrm{direct}} < M_{\mathrm{full-BPTT}} \tag{10}$$

3. Orthogonality to existing methods: SGF is a native training framework orthogonal to existing forcing improvements (Causal Forcing, Rolling Forcing, Self-Forcing++, etc.), applicable on top of them without architectural changes.

Practical Implications

1. Extreme extrapolation capability: Models trained with only a 5-second window can extrapolate to videos lasting several minutes (240s = 48× the training window), suggesting the context-gradient signal is crucial for long-horizon memory reliability.

2. Training efficiency: The modest overhead (8GB peak memory increase, 1.3s per 5 steps) makes SGF practical for real-world training pipelines, unlike direct differentiable-cache alternatives that OOM.

3. Broad applicability: Improvements hold across multiple initializations (TF, causal ODE, causal CD, bidirectional ODE) and both frame-wise and chunk-wise generation, indicating the method is robust to different starting points.

---

Conclusion

Main Takeaways

Self Gradient Forcing (SGF) closes the historical context-gradient gap in frozen-cache Self Forcing by:

• Keeping the serial self-rollout no-gradient (Pass 1)
• Reconstructing the sampled exit computation in parallel with gradients through the clean-context K/V path (Pass 2)

This lets future losses supervise how self-generated histories are written into K/V memory without full rollout backpropagation. Across extensive experiments, SGF preserves short-horizon quality while substantially improving long-video identity, layout consistency, and temporal stability, confirmed by both automatic metrics and human preference.

Limitations and Future Directions

The authors acknowledge:

• SGF is a bounded surrogate, not full BPTT—it does not update sampled latents themselves through future losses
• Pass-2 reconstruction assumes faithful reproduction of the serial context relation; misalignment in mask, sink positions, RoPE handling, or chunk alignment could train a different writer
• SGF is not a replacement for other long-video techniques (long-rollout tuning, retrieval-augmented memory, sparse attention, stronger causal initialization)

Natural next steps: combining SGF with long-rollout exposure or retrieval, so the model both writes better short-window memory and can access richer long-range context. Code and models will be released at https://zhuang2002.github.io/SelfGradientForcing.