Summary (Overview)

• WROP (World Reasoning with Object Permanence) is a large-scale 3D synthetic benchmark and training corpus designed to evaluate and train object permanence (OP) and object solidity (OS) in video generation models.
• The dataset comprises 150 hand-designed Blender generators across six cognitively grounded task families (three probing OP, three probing OS), yielding 1.5 million training samples (10,000 per generator) and a fixed 300-question evaluation exam.
• PWM-WROP, a 16B fine-tuned continuation model trained on WROP data, ranks first among true-continuation models (Elo 1679.5) and third overall in a blind pairwise human study, behind only a statistical tie between two commercial reference-to-video models (Wan 3.0 Prime and MiniMax H3 at Elo 1723.6).
• The paper releases the full data, exam, model answers, scores, weights, and PWM, a native-PyTorch training stack for AWS Trainium2, enabling reproduction and extension.
• Key finding: fine-tuning on cognitively principled synthetic data substantially improves object permanence and solidity reasoning, with PWM-WROP leading the next-best continuation model by 224 Elo points.

---

Introduction and Theoretical Foundation

Background and Motivation

Video generation models are increasingly regarded as world models capable of simulating physical environments. However, they exhibit characteristic failures in foundational physical reasoning:

• Objects vanish behind occluders and re-emerge at impossible positions
• Objects pass through solid barriers undeflected

These failures concern two foundational aspects of human physical intelligence:

1. Object Permanence (OP): The principle that objects persist through time and space independently of observation
2. Object Solidity (OS): The principle that solid objects cannot pass through one another

Cognitive Science Foundation

The paper grounds its work in developmental psychology and core knowledge theory:

• Kant (1929) identified the continued existence of objects as a formal precondition of experience
• Piaget (1954) treated object permanence as the defining cognitive achievement of the sensorimotor stage
• Baillargeon (1986) established via violation-of-expectation (VoE) paradigms that infants represent occluded objects by 3.5 months
• Baillargeon et al. (1985) and Hespos & VanMarle (2012) showed infants register solidity violations within the first half-year of life
• Spelke & Kinzler (2007) classified both OP and OS as part of core knowledge: domain-specific representational systems operational early in development

"A model that permits interpenetration cannot produce physically valid collision or support-removal events, and any higher-level scene construction or causal reasoning is likely to inherit these errors."

Gaps in Existing Benchmarks

The paper identifies three structural limitations in existing video reasoning benchmarks:

1. Coverage predominantly limited to two-dimensional environments
2. Evaluations limited to image-to-video, not covering video-to-video (V2V) settings
3. No dedicated evaluation of structured physical inference about object identity and constraints

Critically, existing benchmarks rely on VLM-based scoring, which is problematic because multimodal language models exhibit systematic core knowledge deficits and cannot reliably judge physical reasoning.

---

Methodology

Cognitive Taxonomy

WROP organizes 150 task generators into six families:

Parameter Design

Each generator partitions parameters into two sets:

• Structural parameters: object count, geometry, trajectory, occlusion configuration, aperture size, contact timing—define the physical challenge
• Surface-level parameters: color, material, lighting, camera viewpoint—randomized to maximize visual diversity without altering physics

Data Generation Pipeline

1. Task-specific generator implementation: Each of 150 tasks is a self-contained, parameterized Blender generator with keyframed motion (no rigid-body solver; trajectories authored analytically)
2. Sample generation: Each sample is a 120-frame animation rendered at 1280×720, 24 fps, split at frame 60 into a 60-frame input video and 60-frame target video, packaged as a five-tuple: input video, target video, prompt, trajectory, metadata
3. Large-scale validation: Automated checks for file completeness, schema validity, frame counts, and manual inspection of representative samples

Model Training (PWM-WROP)

• Base model: Cosmos3-Nano (16B parameters), architecture and tokenizer unchanged
• Training data: 1.5M samples from WROP, one epoch
• Training geometry: 117-frame packed clip at 320×192 (57 conditioning frames + 60 predicted frames)
• Training stack (PWM): Native-PyTorch implementation for AWS Trainium2, sharding the 36-layer model over a tensor-parallel × FSDP2 mesh of 64 NeuronCores, reaching 5.7 s per step at batch 16

Evaluation Protocol

14 models evaluated across three interface classes:

Human judgment: 20 crowdsourced raters, blind pairwise comparisons, Bradley–Terry model with Elo scaling, 1,000 rater-clustered bootstrap replicates for confidence intervals.

Automatic metrics: LPIPS, MS-SSIM, SSIM, PSNR, MSE, CLIP similarity, and FID against the target video (secondary measure).

---

Empirical Validation / Results

Human Preference Leaderboard

Performance by Task Family

PWM-WROP's family-level performance reveals a clear OP/OS asymmetry:

• OP families: Ranks 1st (OP-2), 3rd (OP-1), 3rd (OP-3)
• OS families: Ranks 5th (OS-1), 2nd (OS-2), 8th (OS-3)

This suggests occlusion tracking and contact-based dynamics may require different internal representations, and the fine-tuning regime is more effective for the former.

Automatic Metrics (at matched 320×192 resolution)

PWM-WROP achieves the best perceptual distance (LPIPS), structural similarity (MS-SSIM), and pixel error (MSE) at matched resolution, despite its lower native output resolution.

Qualitative Failure Analysis

Two recurring failure types identified across models:

1. Representation dropout: Model generates plausible scenes but fails to track objects present earlier

   • Balls not re-emerging in correct lanes
   • Objects suddenly appearing after occluder removal
   • Ball revealed under wrong cup
   • Ball passing through blocking barrier

2. Causal decoupling: Model generates reasonable individual events but fails to connect them through physical relationships

   • Ball not falling when support is removed
   • Collision changing object count

---

Theoretical and Practical Implications

Theoretical Significance

• Core knowledge framework: WROP operationalizes cognitive-scientific task paradigms (Baillargeonian occlusion, violation-of-expectation) into scalable training data, bridging developmental psychology and machine learning
• Evidence for trainability: The substantial improvement of PWM-WROP over other continuation models (224 Elo points) provides preliminary evidence that physical reasoning can be trained via cognitively principled synthetic data
• OP/OS dissociation: The differing performance profiles across OP and OS families support the view (Falck et al., 2020) that these may require distinct internal representations

Practical Implications

• Benchmarking infrastructure: WROP provides a human-grounded evaluation framework that circumvents VLM judge limitations
• Training resource: 1.5M samples with per-sample trajectories, scene-state metadata, and controlled nuisance parameters
• Reproducibility: Full release of data, weights, and the PWM training stack enables community extension
• Interface class insights: Reference-to-video models dominate overall but may sacrifice input fidelity; true-continuation models can be improved via targeted fine-tuning

---

Conclusion

WROP establishes a principled foundation for evaluating and training object permanence and solidity in video generation models. The key contributions are:

1. A cognitively grounded benchmark with 150 generators across six task families, 1.5M training samples, and a fixed 300-question exam
2. PWM-WROP, a fine-tuned continuation model ranking first among true-continuation models and competitive with frontier commercial systems
3. A comprehensive evaluation framework combining human preference (Bradley–Terry/Elo) with automatic target-fit metrics
4. A native-PyTorch training stack (PWM) for AWS Trainium2 with full engineering documentation

The results offer preliminary evidence that training on cognitively principled synthetic data is a viable path toward enabling physical reasoning in video generation models. Future work should address the observed OP/OS performance gap, potentially through task-specific training regimes or architectural innovations targeting contact-based dynamics.