Summary of JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion

Summary (Overview)

• JoyAI-Video-Edit is a 16B-parameter autoregressive diffusion framework that enables real-time, open-ended video editing at approximately 30 FPS on a single Nvidia B200 GPU at 720p resolution, without requiring access to future frames or predefined video duration.
• The method introduces chunk-wise autoregressive adaptation to convert a bidirectional offline editor into a causal streaming model, using a sliding temporal window and a global sink to bound computation and memory independently of stream length.
• Source-Anchored Distribution Matching Distillation (SA-DMD) distills the iterative diffusion process into a two-step generator, using source-anchored classifier-free guidance to preserve source fidelity and mitigate drift from imperfect autoregressive history.
• Long-Horizon Autoregressive Distillation (LHAD) performs segmented optimization over extended rollouts to directly supervise states affected by accumulated autoregressive errors, preventing temporal drift in long streams.
• Extensive evaluations show JoyAI-Video-Edit substantially outperforms existing streaming editors (e.g., outperforming XMax-X2.0 by 1.59 points overall on long videos) and remains competitive with strong offline systems like Kling-3.0 Omni and Bernini-R.

Introduction and Theoretical Foundation

Background and Motivation

Instruction-guided video editing is evolving from an offline post-production tool into a continuously available visual capability for applications such as live broadcasting, video communication, interactive entertainment, and real-time content creation. In these scenarios, edited frames must be produced as the source video arrives, requiring:

• Causal output: No access to future frames.
• Low response latency: Immediate visual feedback.
• Bounded computation and memory: Independent of stream duration.
• Stable editing quality: Over an unknown, potentially long duration.

Limitations of Existing Approaches

Most high-quality video editors are designed for offline, fixed-length inputs, processing entire clips using bidirectional or global temporal interactions. This approach has three key drawbacks for streaming:

1. Dependence on future frames prevents causal output.
2. Temporal tokens, attention states, and activation memory grow with input length.
3. Bidirectional computation limits the reuse of cached states.

Applying an offline editor independently to consecutive clips introduces visible discontinuities at clip boundaries.

Key Challenges in Streaming Video Editing

Simply causalizing an offline editor is insufficient due to three core challenges:

1. Train–Inference Mismatch: During training, autoregressive models condition on clean target history; at inference, they consume their own imperfect predictions. This causes reconstruction errors, color deviations, and appearance changes to propagate and accumulate into long-term drift.
2. Source Fidelity vs. Temporal Consistency: Generated history promotes continuity but may propagate errors, while source conditioning preserves fidelity but may weaken editing consistency if not properly balanced.
3. Few-Step Generation Gap: Real-time throughput requires few-step generation, which introduces a gap between the iterative, classifier-free-guided training target and the single-branch model used at deployment.

Methodology

3.1 Model Architecture

JoyAI-Video-Edit consists of three main components:

1. Multimodal Large Language Model (MLLM): Given the first frame of the source video and the editing instruction, it jointly processes visual and textual inputs to extract condition tokens that encode source-aware visual information and editing intent.

2. Causal Video VAE: Encodes the source video into latent sequences and maps optional reference images into the same latent space. Uses a spatiotemporal compression ratio of $8 \times 24 \times 24$ (temporal compression factor of 8, spatial compression factors of 24×24), so each latent frame represents eight video frames.

3. Multimodal Diffusion Transformer (MM-DiT): Jointly models condition tokens and latent visual tokens to generate edited-video latents.

The model supports both Video-to-Video (V2V) and Image-and-Video-to-Video (IV2V) editing.

3.2 Foundation Training

The training follows a multi-stage curriculum:

Stage 1: Text-to-Image (T2I) Pretraining at resolutions $256 \times 256$ and $512 \times 512$ to establish text-visual alignment and high-resolution appearance priors.

Stage 2: Text-to-Video (T2V) Training starting from the T2I checkpoint, introducing temporal modeling while retaining the T2I objective. Training begins at low resolution (256p) and low frame rates (12/24 fps), progressively increasing to 480p at 24 fps.

Stage 3: Image-Editing Training introducing I2I supervision to learn instruction-conditioned transformation and content preservation.

Stage 4: Bidirectional Video-Editing Training jointly optimizing V2V and IV2V supervision with preceding tasks. For a source video $x^s$, editing instruction $c$, optional reference image, and target $x^t$, the VAE produces latents $z^s$, $z^r$, and $z_0$. For noise level $\sigma \in [0, 1]$ and Gaussian noise $\epsilon$:

$$z_{\sigma} = (1 - \sigma) z_0 + \sigma \epsilon, \qquad v^{\star} = \epsilon - z_0.$$

The bidirectional editor $v_{\theta_{\mathrm{bi}}}$ is trained with:

$$\mathcal{L}_{\mathrm{V2V}} = \mathrm{E} \left[ \| v_{\theta_{\mathrm{bi}}}(z_{\sigma}, \sigma; c, z^s, z^r) - v^{\star} \|_2^2 \right].$$

4.1 Chunk-wise Autoregressive Adaptation

The bidirectional model is adapted into a causal paradigm:

• Chunk-wise attention: Bidirectional attention within each chunk, causal across chunks.
• Sliding window: Active chunk attends to a fixed number of recent history chunks plus the first chunk as a global sink.
• Resampling Forcing: Clean history is replaced with on-policy estimates (single-step denoising rollout, detached from gradients) to reduce train–inference mismatch.

4.2 Source-Anchored Distribution Matching Distillation (SA-DMD)

To achieve real-time throughput, the model is distilled into a two-step generator using Distribution Matching Distillation (DMD). The system comprises:

• Causal generator $G_{\theta}$
• Trainable fake-score model $F_{\psi}$
• Frozen real-score model $R_{\phi}$

SA-DMD anchors the teacher to the temporally aligned source chunk using classifier-free guidance (CFG) along independent axes:

$$v_{\phi}^g = v_{\phi}^{\mathrm{cond}} + w_{\mathrm{txt}} \big( v_{\phi}^{\mathrm{cond}} - v_{\phi}^{-\mathrm{txt}} \big) + w_{\mathrm{src}} \big( v_{\phi}^{\mathrm{cond}} - v_{\phi}^{-\mathrm{src}} \big),$$

where $v_{\phi}^{-\mathrm{src}}$ omits the aligned source latent $S^k$, and $v_{\phi}^{-\mathrm{txt}}$ omits the text condition. This renders the guided teacher a source-sharpened posterior, with $w_{\mathrm{src}}$ balancing history continuity against source fidelity.

4.3 Long-Horizon Autoregressive Distillation (LHAD)

To expose distillation to compounded errors from long rollouts without OOM errors:

• Segmented optimization: Divide the sequence into shorter consecutive clips, compute SA-DMD backward pass per clip, clear the graph before generating the next. Gradients are accumulated across clips.
• Dynamic mirror looping: When target rollouts exceed available source video length, conditioning is extended via alternating forward and reversed sequences to preserve temporal continuity.

5. Real-Time Deployment

At deployment, the incoming video stream is divided into consecutive eight-frame chunks. Optimizations include:

• FP8 quantization
• Operator fusion
• Computation-graph compilation
• Bounded-history KV caching (first chunk as global sink + sliding window of recent chunks)

The stage-wise runtime on a single Nvidia B200 GPU: VAE encoding (22ms), DiT denoising (185ms), VAE decoding (19ms) per chunk, yielding 266ms cycle = 30.1 FPS.

Empirical Validation / Results

7.2 Automatic Evaluation

Short-video editing (OpenVE-Bench):

JoyAI-Video-Edit outperforms all streaming methods and is competitive with strong offline systems.

Long-video editing (LongV2VBench):

JoyAI-Video-Edit achieves 44.4% higher throughput than XMax-X2.0 at a substantially lower resolution and more than double the throughput of SANA-Streaming.

7.3 Human Evaluation

Pairwise human preference results:

• JoyAI-Video-Edit receives 90%, 87%, 81%, and 87% preference votes against LiveEdit, SANA-Streaming, XMax-X2.0, and StreamDiffusionV2 respectively.
• Against Bernini-R: 48% JoyAI-Video-Edit vs. 44% Bernini-R.
• Kling-3.0 Omni and Seedance 2.0 each receive 56% preference in their comparisons.

7.4 Deployment Efficiency

7.5 Ablation Study

SA-DMD provides the largest single performance boost (overall +0.42), while LHAD offers complementary gains (+0.25). Both together yield the best results.

Theoretical and Practical Implications

Theoretical Significance

• Source-Anchored Distribution Matching Distillation introduces a principled way to balance history continuity and source fidelity in autoregressive diffusion models by applying classifier-free guidance along independent axes. This provides a theoretical framework for few-step distillation in streaming settings.
• Long-Horizon Autoregressive Distillation demonstrates that segmented optimization over extended rollouts can effectively address error accumulation without incurring OOM, providing a scalable approach for long-duration generation.
• The combination of chunk-wise causal attention with a global sink provides a theoretical bound on per-step computation and memory independent of stream length.

Practical Implications

• Real-time deployment: The system achieves 30 FPS at 720p on a single GPU, making it practical for live broadcasting, video communication, and interactive entertainment.
• Open-ended editing: The model can edit videos of arbitrary length without predefined duration, enabling persistent stream processing for personalized visual effects and adaptive content transformation.
• Immediate visual feedback: The causal pipeline allows users to apply or revise instructions during playback without waiting for complete video processing.
• Diverse editing capabilities: Supports global style transfer, scene/background replacement, localized object manipulation, appearance modification, and motion-aware editing.

Conclusion

JoyAI-Video-Edit is a 16B-parameter autoregressive diffusion framework that transforms video editing from a fixed-clip post-production process into a continuous, interactive workflow. The key contributions are:

1. Chunk-wise autoregressive adaptation with sliding window and global sink for bounded-memory causal generation.
2. Source-Anchored Distribution Matching Distillation (SA-DMD) for two-step generation with source fidelity preservation.
3. Long-Horizon Autoregressive Distillation (LHAD) for mitigating temporal drift in extended rollouts.

The system achieves 30 FPS at 720p on a single Nvidia B200 GPU, substantially outperforms existing streaming editors, and remains competitive with strong offline systems. This opens up new possibilities for real-time video editing in live broadcasting, real-time digital humans, interactive entertainment, game content creation, telepresence, and embodied simulation, providing a new interaction paradigm where video content can be modified continuously as it is captured, transmitted, or consumed.

Future directions: The paper suggests that this approach extends video editing beyond conventional offline production to continuous and interactive workflows, enabling creators to reduce repeated rendering and shorten iteration cycles, and enabling platforms to provide personalized visual effects and adaptive content transformation at scale.