A Survey Toward Unified Video World Models: Reading, Writing, Sharing, and Interacting with the World through Video

Summary (Overview)

• This survey proposes a unifying "world-modeling lens" for video foundation models, viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world $W$, and organizes the literature around four complementary operations: reading (inferring world properties from video), writing (synthesizing video from world conditions), sharing (unifying understanding and generation), and interacting (predicting/updating world states under actions).

• The paper introduces a two-axis taxonomy for video world models: (1) prediction space (observation-space vs. latent-state prediction) and (2) transition interface (passive, action-conditioned, or interactive), providing a systematic framework for comparing models' objectives, capabilities, and evaluation criteria.

• A key conceptual contribution is the distinction between temporal coherence (local frame-to-frame smoothness) and world consistency (compatibility with at least one admissible world trajectory satisfying dynamics, identity persistence, and physical/causal constraints), arguing that neither property implies the other.

• The survey traces a shift from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, distilling seven open challenges including long-horizon consistency, physical/causal grounding, measurable synergy in unified models, and standardized evaluation.

• The paper provides comprehensive coverage across understanding, generation, unified models, and world models, including detailed analysis of architectures, training paradigms, control mechanisms, evaluation protocols, and applications across driving, robotics, embodied AI, and creative content.

---

Introduction and Theoretical Foundation

Background and Motivation

Video holds a distinctive position among modalities because it directly captures how visible entities, relations, and actions evolve across space and time. Recent advances have driven progress in three complementary directions:

• Video understanding: Video-LLMs such as Video-LLaVA [271] and Qwen3-VL [12] enable language-based interaction with visual sequences, with context extending to 256K tokens.
• Video generation: Models such as Sora [31], Wan [462], HunyuanVideo [219], and Movie Gen [372] demonstrate increasingly capable synthesis under diverse conditions.
• World modeling: Emerging models such as Genie [32], GAIA-1 [172], Cosmos [355], V-JEPA 2 [9], and Dreamer 4 [147] explore predictive representation and imagined trajectories for planning and control.

The World-Modeling Lens

The survey formalizes the video–world relationship through a series of definitions:

Definition 2.1 (Latent temporal world): The latent temporal world is a stochastic process $W = (w_t)_{t \geq 1}$ over a state space $\mathcal{W}$, with dynamics:

$$w_{t+1} \sim \mathcal{T}(\cdot | w_t, a_t)$$

where $a_t$ denotes an optional intervention or control input.

Definition 2.2 (Observed video): An observed video $V = (x_1, \dots, x_T)$ is a temporally ordered sequence generated by a sensing process $x_t \sim \mathcal{O}(\cdot \mid w_t, \pi_t)$, where $\pi_t$ is the camera configuration. The trace is: partial (depends only on visible portion), lossy (sensing/compression removes information), and temporally ordered (frame order carries evidence about motion and state transitions).

Definition 2.3 (Represented state): A model constructs $s_t \sim q_\theta(\cdot \mid x_{\leq t}, a_{<t})$, which in general satisfies $s_t \neq w_t$.

Two Forms of Consistency

Definition 2.4 (Temporal coherence and world consistency): A trace $\hat{V}$ is temporally coherent if $d(\hat{x}_t, \hat{x}_{t+1}) \leq \delta$ for all $t$. It is world-consistent if:

$$\exists w_{1:T} \in \mathcal{C}, \pi_{1:T} \quad \text{s.t.} \quad \hat{x}_t \in \operatorname{supp} \mathcal{O}(\cdot | w_t, \pi_t), \quad \forall t$$

Observation 2.5: Neither property implies the other. An object that changes identity after leaving the frame can yield locally smooth frames that no trajectory in $\mathcal{C}$ explains; conversely, a hard cut violates local smoothness while remaining world-consistent.

The Four Operations

Definition 2.6 (Four operations) formalizes reading, writing, sharing, and interacting as distinct ways models connect video observations to the underlying world, with an evidence ladder (Table 2) ranging from plausible output through decision utility.

---

Methodology

Overview of Shared Foundations

The survey covers three pillars of shared background:

Video Large Language Models: Video-LLMs connect video to an LLM through an encoder $E_v$ and connector $P$:

$$z_{1:L} = P(E_v(x_{1:T}))$$

Connectors range from linear projectors to learned-query resamplers and gated cross-attention.

Generative Modeling Foundations: The survey formalizes four generative objectives:

The 2 × 3 Taxonomy of Video World Models

Video world models are classified along two independent axes:

1. Prediction space:

   • Observation-space models predict future frames, video tokens, or generative latents decoded to video (e.g., GAIA [172], EVA [71])
   • Latent-state models predict video-derived features that need not be decoded (e.g., V-JEPA [20], DINO-WM [652])

2. Transition interface:

   • Passive models predict from observations alone
   • Action-conditioned models receive a fixed control sequence before rollout
   • Interactive models accept new controls during rollout (e.g., GameNGen [459], Genie 2 [366])

---

Empirical Validation / Results

Reading: Video Comprehension

Four families of representation learning are identified: contrastive, masked, predictive, and generative learning. Key insights include:

• Contrastive learning (e.g., CLIP4Clip [316], InternVideo [483]) pulls positive pairs together but can drop fine-grained detail.
• Masked learning (e.g., VideoMAE [453], MaskFeat [497]) reconstructs hidden content exploiting temporal redundancy.
• Predictive learning (e.g., V-JEPA [20], V-JEPA 2 [9]) predicts latent features of masked or future content.
• Generative learning (e.g., Gen4U [213], GenCeption [468]) shapes representations through content production.

Key finding: The World Embedding Benchmark [301] shows that alignment and recoverable physical state can diverge—physics-specific contrastive tuning improves video–text matching but weakens physical-quantity regression.

Writing: Video Generation

Four latent representation families for video generation:

Generative paradigms: Diffusion-based (Video Diffusion Models [169], CogVideoX), autoregressive/masked (VideoPoet [217], MAGVIT-v2, Lumos-1 [595]), and flow-based (HunyuanVideo, Wan [462]) approaches, plus LLM-assisted and agentic generation systems.

Control strategies (from Table 13):

Sharing: Unified Video Models

Four architecture families for unification:

1. Discrete-token unification (Emu3 [479], Video-LaVIT [206], VILA-U [521]): One vocabulary, one sequence format, one likelihood objective. Tokenizer becomes bottleneck.

2. Continuous latent-space unification (TUNA [306], Show-o2 [531], PixelUMM [499]): VAE latents shared; understanding benefits from invariance while generation needs detail—a fundamental tension.

3. Decoupled/modular unification (UniVideo [498], Omni-Video [435], Lance [111]): Separate visual pathways linked through LLM or adapter; bridge can become information bottleneck.

4. Agentic/tool-augmented (UniVA [270], VideoAgent [653]): Controller coordinates specialized tools sharing task state.

Key insight: Lance's data-mixture ablations [111] provide bidirectional video evidence: adding understanding data improves VBench generation scores; adding generation data improves MVBench understanding scores.

Interacting: Video World Models

Core technical challenges identified:

1. Action-conditioned controllability: Latent-action learning (LAPO [403], Genie [32]) must separate agent actions from camera/background motion; LAOM [351] shows visual distractors lead to wrong control codes.

2. Long-horizon consistency and memory: The "forcing family" addresses training-inference misalignment—Diffusion Forcing [44], CausVid [582], Self-Forcing [190]. Memory mechanisms range from visual retrieval (WorldMem [528], RELIC [171]) to fixed-size state (Honeycomb [411], WorldTrace [520]).

3. Real-time responsiveness: Recent systems report real-time rates—Vidu S1 [616] at up to 42 FPS, Wan-Streamer [188] at 25 FPS—but display rate can overstate responsiveness since controls may enter less often than frames appear.

4. Physical and causal grounding: Explicit simulation integration (PhysGen [294], PhysGen3D [43]) vs. implicit alignment (VideoREPA [627], PhyGDPO [37]).

---

Theoretical and Practical Implications

The Evidence Ladder

The survey establishes an evidence hierarchy for world-modeling claims, critical for evaluating whether a model genuinely tracks world state:

Recurring Insights

The survey distills several cross-cutting insights:

• "Video understanding is limited by state, not by context length": Longer context helps only when it corrects or stabilizes the state estimate.

• "Rendering fidelity and stateful control are distinct scaling axes": Progress should be measured by matched rollouts with/without control, not single-rollout quality.

• "Synergy requires controlled cross-direction transfer": Ablate the cross-direction path under matched resources and measure the change.

• "Scaling world models needs interventions, not just more clips": Observational scale shows co-occurrence; interventions show what changes what.

---

Conclusion

Key Takeaways

The survey's central contribution is reframing video foundation models around the video–world relationship rather than any single architecture, task, or modality. The four-operation framework makes explicit what different research communities share: all connect video observations to an underlying dynamic world, but at different levels of commitment to state persistence, controllability, and physical/causal validity.

Future Directions

Seven open challenges are identified, each requiring both new methods and better evaluation:

1. Long-horizon world consistency: Memory that updates hidden state during occlusion, trained to predict how unseen entities evolve—not just replay stored frames. Report degradation curves stratified by rollout length.

2. Physical and causal grounding: Three test levels: recognition → generation plausibility → intervention-level validity (changing one parameter should change the affected outcome while leaving unrelated state intact).

3. Measurable synergy in unified video models: Matched ablations (same tasks, data, capacity, compute) differing only in cross-direction information flow; round-trip consistency as both training signal and test.

4. Standardized evaluation of world properties: Match protocol to prediction space and transition interface; report horizon, action frequency, conditioning, judge model; real-time claims need control-to-display latency metrics.

5. Scaling control from unlabeled video: Latent actions must predict controllable effects, stay stable under camera/background changes, and transfer through simple mappings—not just reconstruct.

6. Generalization across scenes and embodiments: Test held-out scene–body pairs; evaluate effect equivalence, controllability, and safety after transfer.

7. Stochastic worlds and rare events: Calibrated outcome probabilities (not just diversity); proper scoring rules, calibration curves, and mode coverage at fixed sample budgets.

The survey concludes that the field must move "from rendering convincing video to modeling how the world changes"—a shift requiring stronger evidence at every level, from perceptual fidelity toward state persistence, controllable effects, and decision utility.