# Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation

> Vidu S2 achieves real-time 720p interactive video generation and editing, outperforming commercial baselines in character identity and instruction following while enabling streaming spatial video for VR.

- **Source:** [arXiv](https://arxiv.org/abs/2609.11638)
- **Published:** 2026-09-16
- **Permalink:** https://picx.dev/p/xY2jqT
- **Whiteboard:** https://picx.dev/p/xY2jqT/image

## Summary

# Summary

## Summary (Overview)

- **Vidu S2** is a real-time interactive video generation and editing system consisting of two models: **Vidu S2-Avatar** (digital character generation) and **Vidu S2-Editing** (video stream editing), both supporting real-time 720p operation.
- **Key improvements over Vidu S1**: higher resolution (540p → 720p at 25–42 FPS), dynamic references updateable at any moment, and stronger instruction following (e.g., dancing).
- **Vidu S2-Editing** enables on-the-fly editing of incoming video streams, covering style transfer, virtual try-on, character replacement, and background replacement.
- The system explores **real-time spatial video generation and editing** for VR head-mounted displays, converting monocular streams into synchronized left-right eye views.
- Extensive evaluation includes public benchmarks (StreamAV-Bench), internal benchmarks, and randomized human-preference comparisons (GSB protocol), showing strong advantages over commercial systems like Runway, PixVerse, and HeyGen.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Traditional video generation models (Sora, Veo, Wan, Seedance) excel at offline content creation but fail to support real-time interactive use cases. The paper argues that:

> "Real-time interactive video generation is one of the most important future directions for industry."

The demand analysis formalizes this: if each user has average demand $\alpha \in [0,1]$ for real-time interactive content, and $\alpha \approx \beta$ with $m > 100$ users, then real-time demand greatly exceeds offline demand.

### Theoretical Basis

Vidu S2 builds on **Vidu S1's** foundation of infinite-length streaming generation without blurring, drift, or visual distortion. Key theoretical components include:

- **Bidirectional training** supporting both Image-to-Video (I2V) and Reference-to-Video (R2V) tasks
- **Segment-wise conditional supervision**: each temporal segment is supervised with its corresponding condition $c^i$ (e.g., caption), rather than a single prompt for the whole sequence
- **Perceptual regularization** to mitigate mode collapse and preserve generation diversity
- **Sink blocks, sliding-window context, and noisy KV caches** retained from Vidu S1

---

## Methodology

### 1. Vidu S2-Avatar Data Pipeline

The pipeline retains Vidu S1's five stages—**Clipping, Filtering, Speech Processing, Captioning, Embedding**—with refinements:

- **Data sources expanded**: high-quality solo dance videos and 2D/3D animation data added to livestream/talking-head and film/TV content
- **Temporally ordered dense captions** replace structured descriptions, specifying temporal boundaries, constituent actions, and outcomes
- **Six-dimensional filtering taxonomy** retained: subject detection, frame cleanliness, visual quality, content safety, shot stability, interactivity
- **High-clarity video selection operator** introduced for 720p training requirements

### 2. Training Strategy

The model jointly trains a bidirectional model on I2V and R2V tasks:

$$v_{replay} = f(v_{history}, c_{current})$$

Each temporal segment is supervised with its condition $c^i$:

$$\mathcal{L} = \mathcal{L}_{recon}(v_{replay}, v_{target}) + \lambda \mathcal{L}_{perc}$$

where $\mathcal{L}_{perc}$ is the perceptual loss.

**Replay-based training**: The original rollout and KV caches are detached before replay, so gradients do not backpropagate through the rollout itself.

### 3. Vidu S2-Editing Data Construction

Training data derived from Vidu S2-Avatar pipeline:

- **Normal-video conditioning**: model learns to reconstruct $v_{raw}$ given $v_{normal}$ and $i_{ref}$, preserving spatial structure and motion
- **Style transfer**: original reference replaced with stylized image while retaining surface-normal video
- **Task-specific fine-tuning**: models with different strengths are ensembled; prompts explicitly require preserving all source-video content outside intended edits

### 4. Real-Time Spatial Video Generation

Two workflows for spatial video editing:

1. **Monocular → Spatial**: Edit monocular stream with Vidu S2-Editing, then convert to synchronized left/right views
2. **Stereo editing**: Edit both views in one pass with same visual references, then split output

---

## Empirical Validation / Results

### 5.1 Digital Character Generation

**Table 1: Public benchmark comparison (StreamAV-Bench)**

| System | Overall Quality | Motion Quality | Expression Quality |
|--------|----------------|----------------|-------------------|
| **Vidu S2-Avatar** | **85.7% preference** | **Unanimous** | **Unanimous** |
| Runway Character GWM-1 | 14.3% | - | - |
| PixVerse Image Avatar | 0% | - | 0% |
| HeyGen | 0% | - | - |

Key findings:
- Vidu S2-Avatar achieves **100% preference** over PixVerse and HeyGen for overall quality
- **Unanimous preference** over Runway and HeyGen for motion and expression quality
- Maintains identity, hairstyle, eye color, attire, and facial structure over time, unlike baselines

### 5.2 Video Editing Results

**Table 4: Unpaired virtual try-on benchmark (VFID)**

| System | VFID ↓ |
|--------|--------|
| **Vidu S2-Editing** | **9.9515** |
| CatV²TON | 19.5131 |
| ViViD | 21.8032 |

Vidu S2-Editing outperforms baselines on:
- Foreground- and background-oriented criteria
- Preserves facial layout, subject silhouette, pose, and temporal texture attachment
- Maintains camera motion, scene geometry, and closed boundaries

### 5.3 Qualitative Results

- **Digital characters**: preserves semantic identity and fine-grained geometry (small objects, articulated hands, facial details)
- **Video editing**: transfers character appearance and reference clothing (e.g., kimono) across frames
- **Spatial video**: enables depth perception, object scale, and scene depth for immersive VR experiences

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Segment-wise conditional supervision** substantially improves instruction following while preserving generation quality
2. **Replay-based training with detached rollouts** enables learning from inference-time states without backpropagation through the rollout
3. **DMD supervision on replayed blocks** equips the model with real-time streaming editing capability
4. **Quantization strategy**: per-tensor and per-channel methods are faster but vulnerable to outliers; more aggressive methods used in less-sensitive layers

### Practical Implications

- **Real-time 720p at 25–42 FPS** makes interactive entertainment feasible
- **Dynamic references** allow mid-stream updates (picking up objects, changing clothes, moving backgrounds)
- **Spatial video for VR headsets** offers immersive experiences with depth perception
- **Streaming video editing** enables live content modification without regeneration

### Challenges and Future Directions

- Spatial video requires **higher resolution and lower latency** than monocular for headset comfort
- **Panoramic spatial video** extension: allowing users to turn heads and explore generated scenes freely
- **Long-horizon stability**: emotional-expression scores remain high, but identity drift over long sequences needs continued attention

---

## Conclusion

Vidu S2 represents a significant advancement in real-time interactive video generation and editing:

1. **Vidu S2-Avatar** achieves real-time 720p digital character generation with dynamic references, stronger instruction following, and superior identity preservation compared to commercial systems.

2. **Vidu S2-Editing** enables real-time streaming video editing across four tasks (style transfer, virtual try-on, character replacement, background replacement), outperforming offline and streaming baselines.

3. **Spatial video generation** extends the system to VR head-mounted displays, offering immersive, continuously updated spatial experiences.

The system demonstrates that real-time interactive video is not only feasible but commercially viable, with the paper concluding: *"We present Vidu S2, which comprises Vidu S2-Avatar, a real-time interactive digital-character model, and Vidu S2-Editing, a real-time video editing model, and we further explore the feasibility of real-time spatial video generation."*

Future work focuses on extending to panoramic spatial video, improving long-sequence stability, and further reducing latency for headset deployment.

---

_Markdown view of https://picx.dev/p/xY2jqT, served by PicX — AI-generated visual whiteboard summaries of research papers._