PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives

Summary (Overview)

• PlayWorld introduces a novel benchmark for evaluating interactive video world models using a multi-modal Agent Player that adaptively executes actions toward shared long-horizon objectives, rather than following fixed predefined trajectories.
• The benchmark comprises 171 human-annotated cases spanning 50 action patterns across four evaluation dimensions: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution, producing 10–60-second rollouts.
• A VQA rubric verifier (Gemini 3.1 Pro) assesses each rollout using sample-specific Yes/No questions, with dimension-specific validation gates before rubric scoring.
• Experiments across nine state-of-the-art world models (including Genie 3, HappyOyster, LingBot-World, HY-World2, and others) reveal that persistent state evolution and global spatial consistency remain the primary bottlenecks for current models.
• Human validation confirms that PlayWorld's VQA metrics closely align with human preference rankings (positive Spearman's ρ across all dimensions).

Introduction and Theoretical Foundation

Background and Motivation

Video world models aim to simulate interactive environments that respond coherently to user actions. While recent systems demonstrate impressive video consistency and action controllability over long sequences (e.g., Genie 3, HappyOyster, Matrix-Game), fair comparison remains challenging because:

1. Action granularity varies across models — the same low-level control input may produce different movement magnitudes in different models.
2. Predefined trajectories cannot adapt — fixed action sequences may overshoot or undershoot the intended state, conflating geometric inconsistency with trajectory failure.
3. Closed-source systems (e.g., Genie 3, HappyOyster) require direct human interaction through web interfaces, making evaluation labor-intensive and prone to bias.

Core Insight

The authors argue that a human player evaluates a world model by pursuing a high-level objective through interaction, not by checking whether a predefined action trajectory is followed. For example, a user may turn 360 degrees to check environmental consistency or walk into water to observe physical responses. The action sequence required to achieve the same objective may vary substantially across models.

Existing Benchmarks and Their Limitations

Existing benchmarks predominantly drive world models with predefined low-level controls, which cannot be flexibly composed into diverse long-horizon trajectories or adjusted online when the generated observation deviates from the expected state.

Methodology

Agent Player Design

The Agent Player consists of two components:

1. Agent model (replaceable multi-modal model, e.g., Claude or Gemini): Observes generated frames, previously executed actions, scene description, and the objective. Decides whether to:

   • Keep — retain the planned action
   • Stop — terminate the active action early when the intended visual state is reached
   • Extend — increase action hold duration when additional movement is required
   • Correct — revise or skip the next planned action when the observed state no longer matches the reference
   • End — terminate the evaluation case when the long-horizon objective is completed

2. Agent interface: Executes decisions on the evaluated world model, captures resulting observations, and returns them to the agent model, forming a closed interaction loop.

Key design choice: A human-annotated basic action sequence serves as a shared initial reference. This preserves consistent evaluation intent across models, reduces sequential planning burden, and still allows observation-conditioned adaptation. For interaction fidelity cases involving obstacles, reaching the obstacle does not trigger End — the Agent Player continues forward to reveal whether collision or penetration occurs.

Benchmark Construction

The construction pipeline involves:

1. Collecting candidate images from Pexels and Google Images (natural, urban, fantasy environments)
2. Screening for clear scene structure and selecting first-person or third-person views
3. Generating environment captions with Gemini 3.1 Pro, verified by human annotators
4. Defining long-horizon objectives based on scenario-specific attributes with visually verifiable completion conditions
5. Writing basic action sequences (W/A/S/D, ↑/↓/←/→, WAIT)
6. Constructing sample-specific VQA rubrics for the target evaluation dimension

VQA Rubric Verifier

The verifier uses Gemini 3.1 Pro to answer sample-specific Yes/No questions. Each rubric assigns greater importance to questions more relevant to the target capability. Dimension-specific validation acts as a gate before rubric scoring:

Weighted Yes/No outcomes are aggregated into a case-level score on a 1–5 scale (higher is better).

Basic Ability Evaluation

Automatic metrics complement the VQA rubric:

• Video Quality (normalized to [0,1] as percentages):

  • Aesthetic Quality, Imaging Quality, Motion Smoothness, Temporal Flickering (from VBench)
  • Temporal Consistency (from Omni-WorldBench)
  • Depth Stability (from MemoBench)
  • Subject Consistency (from HyDRA)

• Action Controllability:

  • Translation Pass Rate and Rotation Pass Rate (following WorldMark)
  • Camera poses estimated by VGGT from uniformly sampled frames
  • Target trajectories defined by executed actions including online adjustments

• Basic Ability Score: Converts the average rank across all nine metrics into a percentage.

Empirical Validation / Results

Main Evaluation Results

Rubric-based VQA evaluation (scores 1–5, higher is better):

Trajectory-validation pass rates:

Agent Player Analysis

Control strategy comparison (on 25 cases for Genie 3 and HappyOyster):

Agent model selection (on Genie 3):

Basic Ability Results

Key Observations

1. Sustained world evolution remains the primary bottleneck — out-of-sight evolution and insight evolution consistently receive lower scores across all models.

2. Long-horizon revisitation exposes global spatial inconsistency — landmarks are repeatedly regenerated at new viewpoints rather than anchored to unique spatial locations (e.g., the Taj Mahal appearing at multiple positions during a 360-degree orbit).

3. Interaction fidelity remains limited beyond simple collision responses — walking into water often fails to produce expected physical/visual responses; third-person models do not consistently bind input actions to the controlled character.

4. Trajectory control and world-model capability can diverge — SANA-WM achieves a relatively high validation pass rate (80.4%) but modest rubric scores, indicating it can follow trajectories without maintaining memory or spatial consistency.

5. Automatic metrics only test basic ability — high Depth Stability and Subject Consistency can occur when a model produces little or no camera motion, even though the intended objective is not completed.

Human Validation

• 600 valid pairwise judgments on video pairs evenly sampled across the four dimensions
• Consistently positive Spearman's ρ correlations between VQA metrics and human preference rankings across all dimensions and Overall score
• Confirms that the VQA metrics closely preserve human preference rankings

Theoretical and Practical Implications

For World Model Evaluation

• Objective-based evaluation with adaptive action execution provides more consistent cross-model comparison than fixed trajectory protocols
• VQA rubrics with validation gates prevent instruction-following failures from making memory evaluation unreliable
• Multiple evaluation dimensions (geometry, interaction, out-of-sight, insight) capture complementary capabilities that automatic metrics miss

For World Model Development

• Persistent state evolution across time and occlusion is identified as the central challenge — models need to maintain semantic state changes during hidden intervals
• Global spatial consistency requires moving beyond local appearance continuity and short-range motion cues toward persistent global representations of generated 3D worlds
• Interaction fidelity beyond simple collision responses (e.g., water interaction) remains an open problem

Limitations of Basic Metrics

• High scores on automatic metrics (e.g., Depth Stability, Subject Consistency) can be misleading when models produce little motion
• Action-controllability thresholds do not guarantee that rollouts reach objective-specific target states

Conclusion

PlayWorld establishes a practical basis for more precise evaluation of interactive video world models through:

1. Multi-modal Agent Players that simulate human players by adaptively executing actions toward shared long-horizon objectives
2. 171 human-annotated cases spanning 50 action patterns across four evaluation dimensions
3. End-to-end automated evaluation combining VQA rubric verification with basic ability metrics
4. Human-validated results confirming alignment between automated metrics and human perception

The benchmark reveals that while current world models can follow camera or action controls, they still struggle to simulate coherent and realistic world dynamics over long horizons, particularly in maintaining persistent state evolution and global spatial consistency. Future directions include upgrading agent models as stronger multi-modal models emerge and extending the benchmark to additional scenario types and evaluation dimensions.