Summary

Summary (Overview)

• Vidu S2 is a real-time interactive video generation and editing system consisting of two models: Vidu S2-Avatar (digital character generation) and Vidu S2-Editing (video stream editing), both supporting real-time 720p operation.
• Key improvements over Vidu S1: higher resolution (540p → 720p at 25–42 FPS), dynamic references updateable at any moment, and stronger instruction following (e.g., dancing).
• Vidu S2-Editing enables on-the-fly editing of incoming video streams, covering style transfer, virtual try-on, character replacement, and background replacement.
• The system explores real-time spatial video generation and editing for VR head-mounted displays, converting monocular streams into synchronized left-right eye views.
• Extensive evaluation includes public benchmarks (StreamAV-Bench), internal benchmarks, and randomized human-preference comparisons (GSB protocol), showing strong advantages over commercial systems like Runway, PixVerse, and HeyGen.

---

Introduction and Theoretical Foundation

Background and Motivation

Traditional video generation models (Sora, Veo, Wan, Seedance) excel at offline content creation but fail to support real-time interactive use cases. The paper argues that:

"Real-time interactive video generation is one of the most important future directions for industry."

The demand analysis formalizes this: if each user has average demand $\alpha \in [0,1]$ for real-time interactive content, and $\alpha \approx \beta$ with $m > 100$ users, then real-time demand greatly exceeds offline demand.

Theoretical Basis

Vidu S2 builds on Vidu S1's foundation of infinite-length streaming generation without blurring, drift, or visual distortion. Key theoretical components include:

• Bidirectional training supporting both Image-to-Video (I2V) and Reference-to-Video (R2V) tasks
• Segment-wise conditional supervision: each temporal segment is supervised with its corresponding condition $c^i$ (e.g., caption), rather than a single prompt for the whole sequence
• Perceptual regularization to mitigate mode collapse and preserve generation diversity
• Sink blocks, sliding-window context, and noisy KV caches retained from Vidu S1

---

Methodology

1. Vidu S2-Avatar Data Pipeline

The pipeline retains Vidu S1's five stages—Clipping, Filtering, Speech Processing, Captioning, Embedding—with refinements:

• Data sources expanded: high-quality solo dance videos and 2D/3D animation data added to livestream/talking-head and film/TV content
• Temporally ordered dense captions replace structured descriptions, specifying temporal boundaries, constituent actions, and outcomes
• Six-dimensional filtering taxonomy retained: subject detection, frame cleanliness, visual quality, content safety, shot stability, interactivity
• High-clarity video selection operator introduced for 720p training requirements

2. Training Strategy

The model jointly trains a bidirectional model on I2V and R2V tasks:

$$v_{replay} = f(v_{history}, c_{current})$$

Each temporal segment is supervised with its condition $c^i$:

$$\mathcal{L} = \mathcal{L}_{recon}(v_{replay}, v_{target}) + \lambda \mathcal{L}_{perc}$$

where $\mathcal{L}_{perc}$ is the perceptual loss.

Replay-based training: The original rollout and KV caches are detached before replay, so gradients do not backpropagate through the rollout itself.

3. Vidu S2-Editing Data Construction

Training data derived from Vidu S2-Avatar pipeline:

• Normal-video conditioning: model learns to reconstruct $v_{raw}$ given $v_{normal}$ and $i_{ref}$, preserving spatial structure and motion
• Style transfer: original reference replaced with stylized image while retaining surface-normal video
• Task-specific fine-tuning: models with different strengths are ensembled; prompts explicitly require preserving all source-video content outside intended edits

4. Real-Time Spatial Video Generation

Two workflows for spatial video editing:

1. Monocular → Spatial: Edit monocular stream with Vidu S2-Editing, then convert to synchronized left/right views
2. Stereo editing: Edit both views in one pass with same visual references, then split output

---

Empirical Validation / Results

5.1 Digital Character Generation

Table 1: Public benchmark comparison (StreamAV-Bench)

Key findings:

• Vidu S2-Avatar achieves 100% preference over PixVerse and HeyGen for overall quality
• Unanimous preference over Runway and HeyGen for motion and expression quality
• Maintains identity, hairstyle, eye color, attire, and facial structure over time, unlike baselines

5.2 Video Editing Results

Table 4: Unpaired virtual try-on benchmark (VFID)

Vidu S2-Editing outperforms baselines on:

• Foreground- and background-oriented criteria
• Preserves facial layout, subject silhouette, pose, and temporal texture attachment
• Maintains camera motion, scene geometry, and closed boundaries

5.3 Qualitative Results

• Digital characters: preserves semantic identity and fine-grained geometry (small objects, articulated hands, facial details)
• Video editing: transfers character appearance and reference clothing (e.g., kimono) across frames
• Spatial video: enables depth perception, object scale, and scene depth for immersive VR experiences

---

Theoretical and Practical Implications

Theoretical Contributions

1. Segment-wise conditional supervision substantially improves instruction following while preserving generation quality
2. Replay-based training with detached rollouts enables learning from inference-time states without backpropagation through the rollout
3. DMD supervision on replayed blocks equips the model with real-time streaming editing capability
4. Quantization strategy: per-tensor and per-channel methods are faster but vulnerable to outliers; more aggressive methods used in less-sensitive layers

Practical Implications

• Real-time 720p at 25–42 FPS makes interactive entertainment feasible
• Dynamic references allow mid-stream updates (picking up objects, changing clothes, moving backgrounds)
• Spatial video for VR headsets offers immersive experiences with depth perception
• Streaming video editing enables live content modification without regeneration

Challenges and Future Directions

• Spatial video requires higher resolution and lower latency than monocular for headset comfort
• Panoramic spatial video extension: allowing users to turn heads and explore generated scenes freely
• Long-horizon stability: emotional-expression scores remain high, but identity drift over long sequences needs continued attention

---

Conclusion

Vidu S2 represents a significant advancement in real-time interactive video generation and editing:

1. Vidu S2-Avatar achieves real-time 720p digital character generation with dynamic references, stronger instruction following, and superior identity preservation compared to commercial systems.

2. Vidu S2-Editing enables real-time streaming video editing across four tasks (style transfer, virtual try-on, character replacement, background replacement), outperforming offline and streaming baselines.

3. Spatial video generation extends the system to VR head-mounted displays, offering immersive, continuously updated spatial experiences.

The system demonstrates that real-time interactive video is not only feasible but commercially viable, with the paper concluding: "We present Vidu S2, which comprises Vidu S2-Avatar, a real-time interactive digital-character model, and Vidu S2-Editing, a real-time video editing model, and we further explore the feasibility of real-time spatial video generation."

Future work focuses on extending to panoramic spatial video, improving long-sequence stability, and further reducing latency for headset deployment.