# Training Object Permanence in World Models

> Fine-tuning a video generation model on 1.5M cognitively grounded synthetic samples yields a 224-Elo-point lead over all true-continuation competitors in object permanence and solidity reasoning.

- **Source:** [arXiv](https://arxiv.org/abs/2609.28654)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/1aBeo7
- **Whiteboard:** https://picx.dev/p/1aBeo7/image

## Summary

## Summary (Overview)

- **WROP (World Reasoning with Object Permanence)** is a large-scale 3D synthetic benchmark and training corpus designed to evaluate and train object permanence (OP) and object solidity (OS) in video generation models.
- The dataset comprises **150 hand-designed Blender generators** across **six cognitively grounded task families** (three probing OP, three probing OS), yielding **1.5 million training samples** (10,000 per generator) and a fixed **300-question evaluation exam**.
- **PWM-WROP**, a 16B fine-tuned continuation model trained on WROP data, ranks **first among true-continuation models** (Elo 1679.5) and third overall in a blind pairwise human study, behind only a statistical tie between two commercial reference-to-video models (Wan 3.0 Prime and MiniMax H3 at Elo 1723.6).
- The paper releases the full data, exam, model answers, scores, weights, and **PWM**, a native-PyTorch training stack for AWS Trainium2, enabling reproduction and extension.
- Key finding: fine-tuning on cognitively principled synthetic data substantially improves object permanence and solidity reasoning, with PWM-WROP leading the next-best continuation model by **224 Elo points**.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Video generation models are increasingly regarded as world models capable of simulating physical environments. However, they exhibit characteristic failures in foundational physical reasoning:

- **Objects vanish** behind occluders and re-emerge at impossible positions
- **Objects pass through solid barriers** undeflected

These failures concern two foundational aspects of human physical intelligence:

1. **Object Permanence (OP)**: The principle that objects persist through time and space independently of observation
2. **Object Solidity (OS)**: The principle that solid objects cannot pass through one another

### Cognitive Science Foundation

The paper grounds its work in developmental psychology and core knowledge theory:

- **Kant (1929)** identified the continued existence of objects as a formal precondition of experience
- **Piaget (1954)** treated object permanence as the defining cognitive achievement of the sensorimotor stage
- **Baillargeon (1986)** established via violation-of-expectation (VoE) paradigms that infants represent occluded objects by **3.5 months**
- **Baillargeon et al. (1985)** and **Hespos & VanMarle (2012)** showed infants register solidity violations within the first half-year of life
- **Spelke & Kinzler (2007)** classified both OP and OS as part of **core knowledge**: domain-specific representational systems operational early in development

> "A model that permits interpenetration cannot produce physically valid collision or support-removal events, and any higher-level scene construction or causal reasoning is likely to inherit these errors."

### Gaps in Existing Benchmarks

The paper identifies three structural limitations in existing video reasoning benchmarks:
1. Coverage predominantly limited to two-dimensional environments
2. Evaluations limited to image-to-video, not covering video-to-video (V2V) settings
3. No dedicated evaluation of structured physical inference about object identity and constraints

Critically, existing benchmarks rely on **VLM-based scoring**, which is problematic because multimodal language models exhibit systematic core knowledge deficits and cannot reliably judge physical reasoning.

---

## Methodology

### Cognitive Taxonomy

WROP organizes 150 task generators into six families:

| Family | Cognitive Dimension | Probe |
|--------|-------------------|-------|
| **OP-1: Baillargeonian Occlusion** | Object Permanence | Target object moves behind an occluder; model must generate re-emergence with identity, size, and motion direction intact |
| **OP-2: Object Static Occlusion** | Object Permanence | Moving occluder covers a static configuration; scene must be reinstated unchanged upon removal |
| **OP-3: Container Permanence** | Object Permanence | Object concealed in a container that moves/rotates/swaps; model must track object bound to container's new position |
| **OS-1: Baillargeonian Obstruction** | Object Solidity | Moving object approaches a barrier with aperture smaller/larger than object; model must generate physically correct outcome |
| **OS-2: Object Drop** | Object Solidity | Support surface withdrawn; object must fall; size-aperture filter determines passage or rest |
| **OS-3: Object Collision** | Object Solidity | Moving object strikes stationary configuration; post-collision trajectories must preserve count and identity |

### Parameter Design

Each generator partitions parameters into two sets:
- **Structural parameters**: object count, geometry, trajectory, occlusion configuration, aperture size, contact timing—define the physical challenge
- **Surface-level parameters**: color, material, lighting, camera viewpoint—randomized to maximize visual diversity without altering physics

### Data Generation Pipeline

1. **Task-specific generator implementation**: Each of 150 tasks is a self-contained, parameterized Blender generator with keyframed motion (no rigid-body solver; trajectories authored analytically)
2. **Sample generation**: Each sample is a 120-frame animation rendered at 1280×720, 24 fps, split at frame 60 into a 60-frame input video and 60-frame target video, packaged as a five-tuple: input video, target video, prompt, trajectory, metadata
3. **Large-scale validation**: Automated checks for file completeness, schema validity, frame counts, and manual inspection of representative samples

### Model Training (PWM-WROP)

- **Base model**: Cosmos3-Nano (16B parameters), architecture and tokenizer unchanged
- **Training data**: 1.5M samples from WROP, one epoch
- **Training geometry**: 117-frame packed clip at 320×192 (57 conditioning frames + 60 predicted frames)
- **Training stack (PWM)**: Native-PyTorch implementation for AWS Trainium2, sharding the 36-layer model over a tensor-parallel × FSDP2 mesh of 64 NeuronCores, reaching **5.7 s per step** at batch 16

### Evaluation Protocol

**14 models evaluated** across three interface classes:

| Interface Class | Models |
|----------------|--------|
| True continuation | PWM-WROP, MAGI-1 24B, LTX-2.3 Extend, Grok Imagine |
| Reference-to-video | Seedance 2.5, Wan 3.0 Prime, MiniMax H3 |
| Edit/transfer | Wan-VACE 14B, HY-OmniWeaving, LTX-2.3 Dev, Cosmos3 Super, Kling O3 Pro, Gemini Omni Flash 1.1, Runway Aleph 2 |

**Human judgment**: 20 crowdsourced raters, blind pairwise comparisons, Bradley–Terry model with Elo scaling, 1,000 rater-clustered bootstrap replicates for confidence intervals.

**Automatic metrics**: LPIPS, MS-SSIM, SSIM, PSNR, MSE, CLIP similarity, and FID against the target video (secondary measure).

---

## Empirical Validation / Results

### Human Preference Leaderboard

| Rank | Model | Class | Elo | 95% CI | Score rate |
|------|-------|-------|-----|--------|------------|
| 1 | Wan 3.0 Prime | Reference-to-video | 1723.6 | [1629.2, 1864.9] | 77.9% |
| 2 | MiniMax H3 | Reference-to-video | 1723.6 | [1644.5, 1837.6] | 77.9% |
| **3** | **PWM-WROP (ours)** | **True continuation** | **1679.5** | **[1603.5, 1781.5]** | **73.1%** |
| 4 | Seedance 2.5 | Reference-to-video | 1649.6 | [1554.2, 1751.5] | 69.6% |
| 5 | Runway Aleph 2 | Edit/transfer | 1518.3 | [1425.1, 1621.6] | 52.9% |
| 6 | Wan-VACE 14B | Edit/transfer | 1506.7 | [1429.9, 1590.8] | 51.0% |
| 7 | Gemini Omni Flash 1.1 | Edit/transfer | 1492.5 | [1404.1, 1572.9] | 49.0% |
| 8 | Kling O3 Pro | Edit/transfer | 1471.3 | [1404.0, 1534.0] | 46.2% |
| 9 | Grok Imagine (extend) | True continuation | 1457.0 | [1362.7, 1555.1] | 44.2% |
| 10 | LTX-2.3 Extend | True continuation | 1453.4 | [1363.6, 1545.8] | 43.1% |
| 11 | Cosmos3 Super | Edit/transfer | 1409.2 | [1318.2, 1488.7] | 37.3% |
| 12 | LTX-2.3 Dev | Edit/transfer | 1398.9 | [1297.7, 1482.3] | 36.5% |
| 13 | HY-OmniWeaving | Edit/transfer | 1268.5 | [1159.4, 1344.2] | 21.0% |
| 14 | MAGI-1 24B | True continuation | 1248.0 | [1137.2, 1315.6] | 19.2% |

### Performance by Task Family

PWM-WROP's family-level performance reveals a clear OP/OS asymmetry:
- **OP families**: Ranks 1st (OP-2), 3rd (OP-1), 3rd (OP-3)
- **OS families**: Ranks 5th (OS-1), 2nd (OS-2), 8th (OS-3)

This suggests occlusion tracking and contact-based dynamics may require different internal representations, and the fine-tuning regime is more effective for the former.

### Automatic Metrics (at matched 320×192 resolution)

| Model | LPIPS↓ | MS-SSIM↑ | SSIM↑ | PSNR↑ | MSE (×10⁻³)↓ | CLIP↑ | FID↓ |
|-------|--------|----------|-------|-------|--------------|-------|------|
| Wan 3.0 Prime | 0.115 | 0.861 | 0.942 | 27.15 | 4.48 | 0.948 | 15.1 |
| MiniMax H3 | 0.105 | 0.877 | 0.938 | 27.51 | 9.09 | 0.962 | 14.8 |
| **PWM-WROP (ours)** | **0.081** | **0.921** | 0.917 | 26.45 | **2.97** | 0.956 | 20.4 |

PWM-WROP achieves the best perceptual distance (LPIPS), structural similarity (MS-SSIM), and pixel error (MSE) at matched resolution, despite its lower native output resolution.

### Qualitative Failure Analysis

Two recurring failure types identified across models:

1. **Representation dropout**: Model generates plausible scenes but fails to track objects present earlier
   - Balls not re-emerging in correct lanes
   - Objects suddenly appearing after occluder removal
   - Ball revealed under wrong cup
   - Ball passing through blocking barrier

2. **Causal decoupling**: Model generates reasonable individual events but fails to connect them through physical relationships
   - Ball not falling when support is removed
   - Collision changing object count

---

## Theoretical and Practical Implications

### Theoretical Significance

- **Core knowledge framework**: WROP operationalizes cognitive-scientific task paradigms (Baillargeonian occlusion, violation-of-expectation) into scalable training data, bridging developmental psychology and machine learning
- **Evidence for trainability**: The substantial improvement of PWM-WROP over other continuation models (224 Elo points) provides preliminary evidence that physical reasoning can be trained via cognitively principled synthetic data
- **OP/OS dissociation**: The differing performance profiles across OP and OS families support the view (Falck et al., 2020) that these may require distinct internal representations

### Practical Implications

- **Benchmarking infrastructure**: WROP provides a human-grounded evaluation framework that circumvents VLM judge limitations
- **Training resource**: 1.5M samples with per-sample trajectories, scene-state metadata, and controlled nuisance parameters
- **Reproducibility**: Full release of data, weights, and the PWM training stack enables community extension
- **Interface class insights**: Reference-to-video models dominate overall but may sacrifice input fidelity; true-continuation models can be improved via targeted fine-tuning

---

## Conclusion

WROP establishes a principled foundation for evaluating and training object permanence and solidity in video generation models. The key contributions are:

1. A **cognitively grounded benchmark** with 150 generators across six task families, 1.5M training samples, and a fixed 300-question exam
2. **PWM-WROP**, a fine-tuned continuation model ranking first among true-continuation models and competitive with frontier commercial systems
3. A **comprehensive evaluation framework** combining human preference (Bradley–Terry/Elo) with automatic target-fit metrics
4. A **native-PyTorch training stack** (PWM) for AWS Trainium2 with full engineering documentation

The results offer preliminary evidence that training on cognitively principled synthetic data is a viable path toward enabling physical reasoning in video generation models. Future work should address the observed OP/OS performance gap, potentially through task-specific training regimes or architectural innovations targeting contact-based dynamics.

---

_Markdown view of https://picx.dev/p/1aBeo7, served by PicX — AI-generated visual whiteboard summaries of research papers._