Summary (Overview)

• Vidu S1 is the first real-time interactive video generation model that supports continuous voice control of digital characters during generation, overcoming the offline, one-shot paradigm of prior video generation models.
• It enables infinite-length stable video generation without blur, drift, or visual collapse, using a sliding-window autoregressive framework with TwinCache and RoPE Repositioning.
• The model achieves real-time performance: 540p resolution at up to 42 FPS on consumer GPUs (RTX 5090), enabled by the TurboDiffusion inference stack and TurboServe serving system.
• Vidu S1 supports custom character uploads (real people, anime, pets) and varied voice tones, achieving state-of-the-art results on both an in-house benchmark (Vidu-StreamBench) and HDTF: CSIM 0.9192, Sync-D 7.847, DOVER 0.5660.
• The paper also introduces a comprehensive data processing pipeline and a three-stage training strategy (bidirectional teacher → causal teacher → distilled few-step generator) that jointly optimize quality, streaming stability, and inference efficiency.

Introduction and Theoretical Foundation

Background and Demand. Most current video generation models (e.g., Sora, Veo, Wan) follow an offline paradigm where users submit a prompt, wait minutes, and receive a complete video without any interaction. In contrast, human visual entertainment includes interactive experiences (live streaming, games, conversation) that demand immediate, user-controlled responses. The authors argue that the total demand for real-time interactive video generation scales as $\alpha \times N$ (per-user demand times number of users), whereas offline generation scales as $\beta \times N/m$ (shared, replayed content). With $\alpha \approx \beta$ and average views $m > 100$, real-time generation demand far exceeds offline demand.

Limitations of Existing Models. Autoregressive video models often fail to support real-time user interaction, do not use speech as a direct control signal, suffer from error accumulation in long-horizon generation (drift/collapse), and lack the efficient inference infrastructure needed for practical deployment.

Theoretical Basis. The paper builds on diffusion models for video-audio joint generation. The clean joint state of video and audio is defined as $x_0^i = [v_0^i; a_0^i]$, where $[\cdot;\cdot]$ denotes concatenation along the modality dimension. Conditioning is provided via $c$, which includes speech, text prompts, and reference images. The training uses a combination of Teacher Forcing and Diffusion Forcing to bridge the gap between training and autoregressive inference, followed by Distribution Matching Distillation (DMD) with Phased Consistency Model (PCM) regularization to compress the denoising steps into few-step generation.

Methodology

Data Preparation. A progressive pipeline (Figure 2) filters raw videos through six stages: prefiltering (removing low-FPS, low-res, A/V desync videos), clipping into single-shot clips (3–60 sec, avoiding speech mid-cut), subject filtering (ensuring single subject in frame), quality/safety filtering (aesthetic score, NSFW removal, shot stability), speech diarization (VAD + active speaker detection to ensure on-screen speaker matches audio, filtering overlaps and low-speech-energy segments), and caption generation (dual-path: visual attributes from frames, acoustic from audio, producing full-clip and speech-aware chunk-level captions).

Training. Three stages:

1. Stage 1: Bidirectional Teacher Training. Trains a bidirectional denoising model $f^{bi}_\theta$ on full video-audio sequences. Loss: $$\mathcal{L}_{bi} = \mathbb{E}_{(x_0^{1:N}, c^{1:N}), t_j} \left[ \| f^{bi}_\theta(x_{t_j}^{1:N}, t_j, c^{1:N}) - x_0^{1:N} \|_2^2 \right].$$
2. Stage 2: Causal Teacher Training. Initializes a causal model from the bidirectional teacher using a causal attention mask. Autoregressive denoising: $$\hat{x}_0^i = f_\theta(x_{t_j}^i, t_j, c^{\le i}, x_{\tau_j}^{<i}, \tau_j).$$ Hybrid training with Teacher Forcing (clean prefix, $\tau_j=0$) and Diffusion Forcing (noisy prefix, $\tau_j>0$) improves robustness. Loss is Eq. (5).
3. Stage 3: DMD with PCM Regularization. Distills the causal teacher into a few-step generator. DMD gradient (Eq. 6): $$\nabla_\theta \mathcal{L}_{DMD} = \mathbb{E}_{z,t} \left[ w(t) (\nabla_{x_t^{1:N}} \log p_t^\text{data}(x_t^{1:N}) - \nabla_{x_t^{1:N}} \log p_t^\theta(x_t^{1:N}))^\top \frac{dx_t^{1:N}}{d\theta} \right].$$ To prevent mode collapse, a PCM consistency loss (Eq. 7) is added: $$\mathcal{L}_{PCM} = \mathbb{E}\left[ \lambda(t_n) d_{\text{PERC}}\big( f_\theta(x_{t_{n+1}}^{1:N}, t_{n+1}), f_{\theta^-}(\tilde{x}_{t_n}^{1:N}, t_n) \big) \right].$$

Inference.

• Streaming inference uses sliding-window decoding with three components: persistent reference context (first frame + first generated state), cached historical states (noisy/clean via TwinCache), and current state. RoPE Repositioning avoids recomputation. TwinCache stores both noisy and clean caches for historical states, where the noisy cache acts as an implicit low-pass filter to suppress artifact accumulation, and the clean cache restores fine details at the final step.
• Inference infrastructure (based on TurboDiffusion and TurboServe) includes: attention acceleration (SageAttention, SpargeAttention, SLA), per-block W8A8 quantized GEMM linear layers, kernel fusion (RMSNorm + elementwise ops), CUDA Graph to reduce launch overhead, and Ulysses-style context parallelism across multiple GPUs.

Empirical Validation / Results

Benchmarks.

• Vidu-StreamBench (500 samples with action instruction, reference frame, audio): Pairwise A/B preference tests against HeyGen, LemonSlice, and Kling-Avatar-2.0.
• HDTF public benchmark: Compared against Wan2.2-S2V-14B, LiveAvatar, OmniAvatar-1.3B, Hallo3, StableAvatar-1.3B, and commercial systems.

Key Results.

Table 1: Quantitative comparison on HDTF

• Vidu S1 achieves the best CSIM (identity preservation), Sync-D (audio-visual synchronization), and DOVER (perceptual quality), while also being the only model that supports both instruction following and real-time generation.

Human Preference on Vidu-StreamBench (Figure 3):

• Vidu S1 vs. HeyGen: Overall 56% preference (16% same, 28% other). Subject controllability: 100% preference.
• Vidu S1 vs. LemonSlice: Overall 46% preference (24% same, 30% other). Subject controllability: 100%.
• Vidu S1 vs. Kling-Avatar-2.0: Overall 48% preference (22% same, 30% other). Subject controllability: 60%.

Qualitative comparison (Figure 4) shows Vidu S1 correctly raising head and giving thumbs-up, while Kling Avatar 2.0 fails on both instructions and introduces background artifacts.

Real-Time Performance. Vidu S1 with 3-step configuration achieves 42 FPS on RTX 5090, exceeding the 30 FPS threshold for real-time interactivity.

Theoretical and Practical Implications

Theoretical Contributions.

• Demonstrates that a combination of bidirectional pretraining, causal adaptation with hybrid Teacher/Diffusion Forcing, and DMD+PCM distillation can yield a stable, few-step autoregressive video generator.
• Introduces TwinCache, a stage-aware caching strategy that uses noisy history as an implicit low-pass filter for temporal propagation and clean history for detail refinement, balancing long-term consistency and visual fidelity.
• Provides a systematic data processing pipeline for interactive character video, emphasizing diarization, speech-aware captioning, and omni-model-based joint filtering to handle challenging cases (e.g., stylized 2D subjects, singing backgrounds).

Practical Implications.

• Vidu S1 makes real-time interactive video generation practical on consumer GPUs, enabling applications in live conversation, virtual hosts, entertainment avatars, and educational agents.
• The speech-driven control allows natural, hands-free interaction, lowering the barrier for users who cannot type or specify detailed prompts.
• The infinite-length generation without drift addresses a critical bottleneck for streaming services (e.g., 24/7 virtual influencers, interactive storytelling).

Conclusion

Vidu S1 is a real-time interactive video generation model that supports voice control, infinite-length generation, and customization of digital characters. By combining a three-stage training pipeline (bidirectional teacher, causal teacher, DMD+PCM distillation) with an efficient inference stack (TurboDiffusion, TurboServe, TwinCache, attention acceleration, quantization, CUDA Graph, multi-GPU parallelism), it achieves 540p video at 42 FPS on consumer GPUs while setting new state-of-the-art results on both identity preservation (CSIM 0.9192), audio-visual sync (Sync-D 7.847), and perceptual quality (DOVER 0.5660). The paper demonstrates that real-time interactive video generation is not only feasible but can match or exceed offline generation quality. Future directions include extending to more complex multi-subject interactions, improving handling of rapid motion, and further reducing GPU requirements for even wider deployment.