# Reading, Writing, Sharing, and Interacting with the World through Video

> This survey reframes video foundation models as four operations—reading, writing, sharing, interacting—unifying them through a world-modeling lens and formalizing a taxonomy for evaluation.

- **Source:** [arXiv](https://arxiv.org/abs/2610.31668)
- **Published:** 2026-10-27
- **Permalink:** https://picx.dev/p/2Hnz89
- **Whiteboard:** https://picx.dev/p/2Hnz89/image

## Summary

# A Survey Toward Unified Video World Models: Reading, Writing, Sharing, and Interacting with the World through Video

## Summary (Overview)

- This survey proposes a unifying **"world-modeling lens"** for video foundation models, viewing an observed video as a *partial, lossy, and temporally ordered projection* of a latent temporal world $W$, and organizes the literature around **four complementary operations**: **reading** (inferring world properties from video), **writing** (synthesizing video from world conditions), **sharing** (unifying understanding and generation), and **interacting** (predicting/updating world states under actions).

- The paper introduces a **two-axis taxonomy** for video world models: (1) *prediction space* (observation-space vs. latent-state prediction) and (2) *transition interface* (passive, action-conditioned, or interactive), providing a systematic framework for comparing models' objectives, capabilities, and evaluation criteria.

- A key conceptual contribution is the distinction between **temporal coherence** (local frame-to-frame smoothness) and **world consistency** (compatibility with at least one admissible world trajectory satisfying dynamics, identity persistence, and physical/causal constraints), arguing that neither property implies the other.

- The survey traces a shift from clip-level recognition and perceptual fidelity toward **persistent state, controllable evolution, physical and causal validity**, and evaluation grounded in interaction and downstream decisions, distilling **seven open challenges** including long-horizon consistency, physical/causal grounding, measurable synergy in unified models, and standardized evaluation.

- The paper provides **comprehensive coverage** across understanding, generation, unified models, and world models, including detailed analysis of architectures, training paradigms, control mechanisms, evaluation protocols, and applications across driving, robotics, embodied AI, and creative content.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Video holds a distinctive position among modalities because it directly captures how visible entities, relations, and actions evolve across space and time. Recent advances have driven progress in three complementary directions:

- **Video understanding**: Video-LLMs such as Video-LLaVA [271] and Qwen3-VL [12] enable language-based interaction with visual sequences, with context extending to 256K tokens.
- **Video generation**: Models such as Sora [31], Wan [462], HunyuanVideo [219], and Movie Gen [372] demonstrate increasingly capable synthesis under diverse conditions.
- **World modeling**: Emerging models such as Genie [32], GAIA-1 [172], Cosmos [355], V-JEPA 2 [9], and Dreamer 4 [147] explore predictive representation and imagined trajectories for planning and control.

### The World-Modeling Lens

The survey formalizes the video–world relationship through a series of definitions:

**Definition 2.1 (Latent temporal world)**: The latent temporal world is a stochastic process $W = (w_t)_{t \geq 1}$ over a state space $\mathcal{W}$, with dynamics:

$$w_{t+1} \sim \mathcal{T}(\cdot | w_t, a_t)$$

where $a_t$ denotes an optional intervention or control input.

**Definition 2.2 (Observed video)**: An observed video $V = (x_1, \dots, x_T)$ is a temporally ordered sequence generated by a sensing process $x_t \sim \mathcal{O}(\cdot \mid w_t, \pi_t)$, where $\pi_t$ is the camera configuration. The trace is: **partial** (depends only on visible portion), **lossy** (sensing/compression removes information), and **temporally ordered** (frame order carries evidence about motion and state transitions).

**Definition 2.3 (Represented state)**: A model constructs $s_t \sim q_\theta(\cdot \mid x_{\leq t}, a_{<t})$, which in general satisfies $s_t \neq w_t$.

### Two Forms of Consistency

**Definition 2.4 (Temporal coherence and world consistency)**: A trace $\hat{V}$ is *temporally coherent* if $d(\hat{x}_t, \hat{x}_{t+1}) \leq \delta$ for all $t$. It is *world-consistent* if:

$$\exists w_{1:T} \in \mathcal{C}, \pi_{1:T} \quad \text{s.t.} \quad \hat{x}_t \in \operatorname{supp} \mathcal{O}(\cdot | w_t, \pi_t), \quad \forall t$$

**Observation 2.5**: Neither property implies the other. An object that changes identity after leaving the frame can yield locally smooth frames that no trajectory in $\mathcal{C}$ explains; conversely, a hard cut violates local smoothness while remaining world-consistent.

### The Four Operations

**Definition 2.6 (Four operations)** formalizes reading, writing, sharing, and interacting as distinct ways models connect video observations to the underlying world, with an **evidence ladder** (Table 2) ranging from plausible output through decision utility.

---

## Methodology

### Overview of Shared Foundations

The survey covers three pillars of shared background:

**Video Large Language Models**: Video-LLMs connect video to an LLM through an encoder $E_v$ and connector $P$:

$$z_{1:L} = P(E_v(x_{1:T}))$$

Connectors range from linear projectors to learned-query resamplers and gated cross-attention.

**Generative Modeling Foundations**: The survey formalizes four generative objectives:

| Objective | Formula | Purpose |
|-----------|---------|---------|
| **Autoregressive** | $p_\theta(z_{1:N} \mid c) = \prod_{i=1}^{N} p_\theta(z_i \mid z_{<i}, c)$ | Ordered sequence generation |
| **Masked modeling** | $\mathcal{L}_{\text{mask}} = \mathbb{E}\left[-\sum_{i \in M} \log p_\theta(z_i \mid z_{\bar{M}}, c)\right]$ | Parallel prediction of hidden units |
| **Diffusion** | $\mathcal{L}_{\text{diff}} = \mathbb{E}\left[\|\epsilon_\theta(z_k, k, c) - \epsilon\|^2\right]$ | Noise prediction and denoising |
| **Flow matching** | $\mathcal{L}_{\text{fm}} = \mathbb{E}\left[\|v_\theta(z_k, k, c) - (\epsilon - z_0)\|^2\right]$ | Velocity field learning for transport |

### The 2 × 3 Taxonomy of Video World Models

Video world models are classified along two independent axes:

1. **Prediction space**: 
   - *Observation-space* models predict future frames, video tokens, or generative latents decoded to video (e.g., GAIA [172], EVA [71])
   - *Latent-state* models predict video-derived features that need not be decoded (e.g., V-JEPA [20], DINO-WM [652])

2. **Transition interface**:
   - *Passive* models predict from observations alone
   - *Action-conditioned* models receive a fixed control sequence before rollout
   - *Interactive* models accept new controls during rollout (e.g., GameNGen [459], Genie 2 [366])

---

## Empirical Validation / Results

### Reading: Video Comprehension

**Four families of representation learning** are identified: contrastive, masked, predictive, and generative learning. Key insights include:

- **Contrastive learning** (e.g., CLIP4Clip [316], InternVideo [483]) pulls positive pairs together but can drop fine-grained detail.
- **Masked learning** (e.g., VideoMAE [453], MaskFeat [497]) reconstructs hidden content exploiting temporal redundancy.
- **Predictive learning** (e.g., V-JEPA [20], V-JEPA 2 [9]) predicts latent features of masked or future content.
- **Generative learning** (e.g., Gen4U [213], GenCeption [468]) shapes representations through content production.

**Key finding**: The World Embedding Benchmark [301] shows that alignment and recoverable physical state can diverge—physics-specific contrastive tuning improves video–text matching but weakens physical-quantity regression.

### Writing: Video Generation

**Four latent representation families** for video generation:

| Type | Examples | Characteristics |
|------|----------|----------------|
| **Continuous latents** | VideoLDM [28], CV-VAE [641], CogVideoX [570] | 3D/factorized spatiotemporal compression; causal 3D VAEs avoid future leakage |
| **Discrete tokens** | MAGVIT-v2 [589], FSQ [342], BSQ [644] | Codebook-based or codebook-free quantization; LFQ gives very large implicit vocabularies |
| **Hybrid** | OneVAE [660], Divot [122], DiCoDe [261] | Combine continuous and discrete forms or change decoder (e.g., diffusion decoding) |
| **Semantically aligned** | VILA-U [521], PyraTok [432] | One tokenizer for reconstruction and text alignment |

**Generative paradigms**: Diffusion-based (Video Diffusion Models [169], CogVideoX), autoregressive/masked (VideoPoet [217], MAGVIT-v2, Lumos-1 [595]), and flow-based (HunyuanVideo, Wan [462]) approaches, plus LLM-assisted and agentic generation systems.

**Control strategies** (from Table 13):

| Family | Entry point | Example | Trade-off |
|--------|-------------|---------|-----------|
| Attention | Cross/self-attention maps | Prompt-to-Prompt [163], TokenFlow [126] | Local edits, but large motion breaks correspondences |
| Adapter | Block features via side branch | ControlNet [620], VACE [203] | Keeps pretrained generator; needs paired data |
| Latent space | Noisy latent / denoising trajectory | SDEdit [340], DDIM inversion [421] | Noise level trades fidelity vs. edit strength |
| Concatenation | Model input | VideoComposer [475], FullDiT [207] | Simple; many conditions; attention cost |
| Guidance | Sampling update | CFG [168], FreeDoM [588] | Test-time constraints; reduces diversity |
| Tuning | Parameters/embeddings | DreamBooth [398], LoRA [174] | Faithful identity; needs adaptation data |

### Sharing: Unified Video Models

**Four architecture families** for unification:

1. **Discrete-token unification** (Emu3 [479], Video-LaVIT [206], VILA-U [521]): One vocabulary, one sequence format, one likelihood objective. Tokenizer becomes bottleneck.

2. **Continuous latent-space unification** (TUNA [306], Show-o2 [531], PixelUMM [499]): VAE latents shared; understanding benefits from invariance while generation needs detail—a fundamental tension.

3. **Decoupled/modular unification** (UniVideo [498], Omni-Video [435], Lance [111]): Separate visual pathways linked through LLM or adapter; bridge can become information bottleneck.

4. **Agentic/tool-augmented** (UniVA [270], VideoAgent [653]): Controller coordinates specialized tools sharing task state.

**Key insight**: Lance's data-mixture ablations [111] provide bidirectional video evidence: adding understanding data improves VBench generation scores; adding generation data improves MVBench understanding scores.

### Interacting: Video World Models

**Core technical challenges identified**:

1. **Action-conditioned controllability**: Latent-action learning (LAPO [403], Genie [32]) must separate agent actions from camera/background motion; LAOM [351] shows visual distractors lead to wrong control codes.

2. **Long-horizon consistency and memory**: The "forcing family" addresses training-inference misalignment—Diffusion Forcing [44], CausVid [582], Self-Forcing [190]. Memory mechanisms range from visual retrieval (WorldMem [528], RELIC [171]) to fixed-size state (Honeycomb [411], WorldTrace [520]).

3. **Real-time responsiveness**: Recent systems report real-time rates—Vidu S1 [616] at up to 42 FPS, Wan-Streamer [188] at 25 FPS—but display rate can overstate responsiveness since controls may enter less often than frames appear.

4. **Physical and causal grounding**: Explicit simulation integration (PhysGen [294], PhysGen3D [43]) vs. implicit alignment (VideoREPA [627], PhyGDPO [37]).

---

## Theoretical and Practical Implications

### The Evidence Ladder

The survey establishes an evidence hierarchy for world-modeling claims, critical for evaluating whether a model genuinely tracks world state:

| Level | Reading | Writing | Sharing | Interacting |
|-------|---------|---------|---------|-------------|
| **1. Plausible output** | Correct labels on clips | Perceptual quality | Both directions work | Plausible predicted frames |
| **2. Faithful to evidence** | Shortcuts controlled | Condition adherence | Source-faithful transformations | Follows controls |
| **3. Consistent world state** | State across occlusion | World consistency | Measured transfer | Persistent state, valid interventions |
| **4. Decision utility** | Supports decisions | Improves downstream | Improves planning | Better policy training/ranking |

### Recurring Insights

The survey distills several cross-cutting insights:

- **"Video understanding is limited by state, not by context length"**: Longer context helps only when it corrects or stabilizes the state estimate.

- **"Rendering fidelity and stateful control are distinct scaling axes"**: Progress should be measured by matched rollouts with/without control, not single-rollout quality.

- **"Synergy requires controlled cross-direction transfer"**: Ablate the cross-direction path under matched resources and measure the change.

- **"Scaling world models needs interventions, not just more clips"**: Observational scale shows co-occurrence; interventions show what changes what.

---

## Conclusion

### Key Takeaways

The survey's central contribution is reframing video foundation models around the **video–world relationship** rather than any single architecture, task, or modality. The four-operation framework makes explicit what different research communities share: all connect video observations to an underlying dynamic world, but at different levels of commitment to state persistence, controllability, and physical/causal validity.

### Future Directions

Seven open challenges are identified, each requiring both new methods and better evaluation:

1. **Long-horizon world consistency**: Memory that updates hidden state during occlusion, trained to predict how unseen entities evolve—not just replay stored frames. Report degradation curves stratified by rollout length.

2. **Physical and causal grounding**: Three test levels: recognition → generation plausibility → intervention-level validity (changing one parameter should change the affected outcome while leaving unrelated state intact).

3. **Measurable synergy in unified video models**: Matched ablations (same tasks, data, capacity, compute) differing only in cross-direction information flow; round-trip consistency as both training signal and test.

4. **Standardized evaluation of world properties**: Match protocol to prediction space and transition interface; report horizon, action frequency, conditioning, judge model; real-time claims need control-to-display latency metrics.

5. **Scaling control from unlabeled video**: Latent actions must predict controllable effects, stay stable under camera/background changes, and transfer through simple mappings—not just reconstruct.

6. **Generalization across scenes and embodiments**: Test held-out scene–body pairs; evaluate effect equivalence, controllability, and safety after transfer.

7. **Stochastic worlds and rare events**: Calibrated outcome probabilities (not just diversity); proper scoring rules, calibration curves, and mode coverage at fixed sample budgets.

The survey concludes that the field must move "from rendering convincing video to modeling how the world changes"—a shift requiring stronger evidence at every level, from perceptual fidelity toward state persistence, controllable effects, and decision utility.

---

_Markdown view of https://picx.dev/p/2Hnz89, served by PicX — AI-generated visual whiteboard summaries of research papers._