Summary (Overview)

• New benchmark for world models: WorldExam introduces a hierarchical diagnostic benchmark with 1,474 test cases across eight tasks, evaluating video generation models as world models beyond visual quality.
• Four diagnostic levels: The benchmark spans Visual Quality, Control Adherence, Spatial Consistency, and World Reactivity — the ability to infer scene-conditioned consequences not explicitly stated in the input.
• Unified evaluation across paradigms: Supports camera-driven, action-driven, and language-driven models via interface adaptation of atomic control units into native formats (SE(3) trajectories, discrete actions, or natural language).
• Key finding: Evaluation of 20 representative models reveals a clear capability split — no model combines broad task coverage with consistently strong performance. High visual quality and explicit instruction fulfillment do not guarantee inherent reactivity.
• Reliable evaluation: Strong human–VLM agreement (Spearman’s ρ = 0.8614) and stable rankings under alternative reconstruction backends (DA3, mean absolute relative change < 3.1%) support the findings.

Introduction and Theoretical Foundation

Controllable video generation models are increasingly being developed as world models — systems that predict future visual states from an initial observation and control instructions (camera trajectories, action sequences, language prompts). Evaluating such models extends beyond the apparent appearance of generated videos to the inherent reactivity of the worlds they depict: the ability to infer from the scene state how the world should react and generate plausible consequences not explicitly described in the input.

Existing benchmarks primarily assess:

• Visual quality and temporal consistency
• Explicit instruction fulfillment (e.g., “does the camera follow the specified trajectory?”)
• Structured layout control, action interfaces, or prompt-specified interaction effects

However, they leave underexamined the model’s ability to infer additional consequences implied by the initial scene state but not specified in the instruction. For example:

• When a subject moves onto stairs, its motion should adapt to the terrain.
• When it approaches an obstacle, the world should show contact, avoidance, or blockage.
• When it enters another agent’s personal space, that agent should respond plausibly.

WorldExam is designed around this distinction. It represents each controllable behavior as a composition of atomic control units (e.g., W for forward, S for backward, A for left, D for right, ↑↓←→ for camera rotations, ∅ for stop) and adapts these units to each model’s native interface. For World Reactivity cases, the model-facing instruction specifies only the explicit control or goal, leaving the expected scene-conditioned reactions unstated.

Methodology

Benchmark Structure

WorldExam organizes evaluation into four diagnostic levels:

1. Visual Quality – Perceptual plausibility, temporal stability, aesthetic quality (task-agnostic metrics)
2. Control Adherence – Whether the controlled camera or subject follows the input control (Camera Control, Subject Control tasks)
3. Spatial Consistency – Whether the model preserves a coherent world when revisiting a previously observed viewpoint (Scene Revisit task)
4. World Reactivity – Scene-conditioned reactions and goal-directed behaviors beyond explicit input (Terrain Interaction, Object Interaction, Social Interaction, Physical Reaction, Goal Completion tasks)

The eight evaluation tasks are assigned to two tracks to avoid penalizing models for unsupported capabilities:

• Static-scene track (Camera Control, Scene Revisit) – available to all three paradigms
• Dynamic-interaction track (Subject Control + five World Reactivity tasks) – for action- and language-driven models only (Goal Completion is language-only)

Test Case Curation

For dynamic-interaction tasks, a pipeline (Figure 3) constructs cases:

1. Sample task pattern from a domain-specific library
2. Schema-guided LLM generates structured draft (scene description, initial-image prompt, control intent, optional checklist)
3. Generate N candidate initial images via text-to-image, human-filter for validity
4. Image-conditioned refiner revises draft to match selected image

The final case consists of: initial image $I_0$, control intent or high-level goal, grounded text prompt, and optional checklist $L = \{\ell_k\}_{k=1}^K$.

Evaluation Protocols

Geometry-based evaluation (Camera Control, Scene Revisit, Subject Control, Terrain Interaction):

• Use VGGT-Ω [45] to estimate camera poses, intrinsics, depths from generated videos
• For Camera Control, compute translation and rotation errors between recovered and reference trajectories:

$$e_t = \min_{s \ge 0} \frac{1}{T} \sum_{t=1}^T \| s \mathbf{t}_t - \mathbf{t}_t^* \|_2, \quad e_r = \frac{180}{\pi} \frac{1}{T} \sum_{t=1}^T \arccos\left( \frac{\operatorname{tr}(\mathbf{R}_t (\mathbf{R}_t^*)^\top) - 1}{2} \right)$$

• Normalize to scores: $s_t = \max(0, 1 - e_t/0.5)$, $s_r = \max(0, 1 - e_r/15^\circ)$, then $S_{\text{cam}} = 100 \sqrt{s_t s_r}$
• For Scene Revisit: find frame with recovered pose closest to initial pose, compute Revisit Success (binary), then combine with PSNR, LPIPS, SSIM:

$$S_{\text{rev}} = 100 \sqrt{ S_{\text{succ}} \cdot \frac{s_P + s_L + s_S}{3} }$$

Checklist-based evaluation (Object Interaction, Social Interaction, Physical Reaction, Goal Completion):

• Use GPT-5.5 as VLM judge on 10 uniformly sampled frames against case-specific checklists
• Score: $S_{\text{check}}(V, L) = \frac{100}{K} \sum_{k=1}^K \mathbb{I}[\ell_k \text{ is satisfied in } V]$

Image-space displacement alignment for camera-driven models: pre-generation calibration to normalize translation magnitudes across models.

Empirical Validation / Results

Static-Scene Track (Table 2)

• Camera-driven models (3D reprojection methods) excel at camera control and scene revisiting
• Action-driven WorldPlay achieves strong camera control (92.74) but lower scene revisit
• Language-driven models are less precise on composed camera controls

Dynamic-Interaction Track (Table 4)

• Action-driven models control subjects more precisely (LingBot-World: 55.47) but fail to induce world reactivity (Terrain Interaction: 24.33)
• Language-driven models perform better on interaction tasks but follow controls less faithfully
• Goal Completion: HappyHorse 1.0 (85.33) and Veo 3.1 (85.30) lead; Kling 2.5 scores only 48.25 despite highest General average

Cross-Task Diagnostic Analysis (Figure 8)

The capability split is clear:

• Camera-driven models: Strongest camera control and scene revisiting, no dynamic interaction support
• Action-driven models: More precise subject control, but terrain, objects, agents, and physics remain unresponsive
• Language-driven models: Better on interaction and goal-directed tasks, but less faithful on composed controls

No model combines broad coverage with consistently strong performance.

Human Alignment (Table 5)

Strong agreement between VLM judge and human evaluation.

Backend Stability (DA3)

Mean absolute relative change in Overall score: 3.09% (static-scene), 0.57% (dynamic-interaction). Rankings preserved within paradigms.

Theoretical and Practical Implications

• Theoretical: The paper formalizes the distinction between explicit instruction fulfillment and inherent reactivity, providing a framework for evaluating world models beyond surface-level generation quality. The four-level hierarchy (Visual Quality → Control Adherence → Spatial Consistency → World Reactivity) captures distinct capabilities that do not necessarily improve together.
• Practical implications:
  • Current video world models have complementary but incomplete capabilities. No single paradigm yet achieves both precise control and rich world reactivity.
  • High visual quality and explicit instruction fulfillment do not guarantee that a model can infer scene-conditioned consequences or execute goal-directed behaviors.
  • The benchmark provides a diagnostic tool for developers to localize failures (e.g., camera control vs. terrain adaptation vs. social interaction).
  • The interface adaptation methodology enables fair comparison across heterogeneous model paradigms.
• Limitations: Dynamic-interaction evaluation requires reliable third-person subject control; Goal Completion is limited to language-driven models. Metrics assess observable end-to-end behavior, not internal causal representations.

Conclusion

WorldExam introduces a unified hierarchical benchmark for diagnosing video world models beyond visual quality and explicit instruction fulfillment. It distinguishes direct fulfillment of specified controls from inherent reactivity — scene-conditioned reactions and goal-directed execution that must be inferred from the initial scene. The benchmark comprises 1,474 cases across eight tasks, with interface adaptation for camera-, action-, and language-driven models, and separate tracks to avoid penalizing unsupported capabilities.

Evaluation of 20 representative models reveals a clear capability split:

• Camera-driven models excel at camera control and scene revisiting
• Action-driven models control subjects more precisely but leave the world unresponsive
• Language-driven models perform better on interaction but follow composed controls less faithfully

No model combines broad task coverage with consistently strong performance, showing that high visual quality and explicit instruction fulfillment do not guarantee inherent reactivity. Strong human–VLM agreement and stable rankings under alternative reconstruction backends support the reliability of these findings.

Future directions: Broader interfaces, longer-horizon interactions, and more intervention-based settings would provide stronger tests of persistent world understanding. The benchmark data and evaluation toolkit will be publicly released to facilitate systematic evaluation and progress in the video world model community.