Summary (Overview)

• RynnWorld-4D introduces a projective 4D representation (RGB, Depth, Optical Flow – RGB-DF) that makes geometry and motion explicit while staying compatible with large-scale video diffusion priors.
• The method extends a video diffusion transformer into a tri-branch architecture with joint cross-modal attention and 3D RoPE to generate synchronized, physically coherent 4D sequences from a single RGB-D image and language instruction.
• To train the model, the authors curate Rynn4DDataset 1.0, a massive hybrid dataset of 254.4 million frames from egocentric human and robotic manipulation videos, pseudo-annotated with depth and optical flow.
• The RynnWorld-4D-Policy head extracts actions directly from the model’s internal 4D features in a single forward pass, bypassing iterative denoising and enabling closed-loop control at ~9 Hz.
• Experiments show the model achieves state‑of‑the‑art success rates across six real‑world bimanual dexterous manipulation tasks, especially those requiring spatial precision and temporal coordination (e.g., 65.71% on Bowl Stacking and Lid Placement).

Introduction and Theoretical Foundation

Robotic manipulation in open-world environments requires not only visual recognition but also the ability to anticipate how the 3D structure of a scene will move under interaction. Current generative video models operate in 2D pixel space, losing critical spatial relationships and leading to temporal inconsistencies (e.g., fluctuating object scales). The authors argue that predicting synchronized RGB, depth, and optical flow (RGB-DF) provides a physically grounded representation that:

• Lifts each pixel to a 3D location (via depth),
• Tracks 3D motion over time (via depth + optical flow → 3D scene flow),
• Stays in a 2D‑aligned format, inheriting the scalability and rich generative priors of pretrained video diffusion models.

This representation bridges the gap between world prediction and policy learning, as the predicted 4D dynamics align more closely with the robot’s action space than raw 2D pixel changes.

Methodology

1. Rynn4DDataset 1.0 (Data Curation)

The dataset comprises over 254.4 million video frames from human egocentric (Epic-Kitchens, EgoVid) and robotic manipulation (RoboMIND, RDT-1B, Galaxea, RoboCoin, AgiBot) datasets. Each frame is annotated with:

• Video captions via Qwen3-VL (512 tokens, temperature 0.7).
• Optical flow via DPFlow (dense per-frame motion fields).
• Depth via Depth Anything 3 (DA3NESTED-GIANT-LARGE-1.1), with values clipped to [0.0, 5.0] meters and quantized to 8-bit grayscale.

2. 3D Scene Reconstruction from Multi-Modal Videos

The key geometric interpretability is given by:

Geometric Unprojection (Equation 1):

$$P_t = D_t(u, v) \cdot K^{-1} p_t$$

where $P_t$ is a 3D point from pixel $p_t = [u, v, 1]^\top$ at frame $t$, $D_t$ is the depth map, and $K$ the camera intrinsic matrix.

Metric Scene Flow (Equation 2):

$$P_{t+1} = D_{t+1}(u+\Delta u, v+\Delta v) \cdot K^{-1} (p_t + [\Delta u, \Delta v, 0]^\top)$$ $$f_{\text{3D}} = P_{t+1} - P_t$$

where $[\Delta u, \Delta v]^\top$ is the optical flow.

3. RynnWorld-4D (Generative Model)

The model extends the Wan 2.2-TI2V-5B diffusion transformer into a tri-branch architecture (RGB, depth, flow). Key components:

• Joint Cross-Modal Attention (JA): Inserted every 3 of 30 layers (10 modules total). Each branch $m$ produces queries $Q^m_l$ and shared keys/values $K^m_l, V^m_l$. Queries attend to the concatenated keys/values of the complementary modalities (Equation 5):

  $$A^m_l = \text{Attn}_{\text{RoPE}}(Q^m_l, K_{\text{cross}}^l, V_{\text{cross}}^l)$$

  with $K_{\text{cross}}^l = \text{concat}(\{K^j_l\}_{j \neq m})$, $V_{\text{cross}}^l = \text{concat}(\{V^j_l\}_{j \neq m})$.

• 3D Rotary Positional Embeddings (RoPE) applied frame-wise to maintain cross-modal spatial correspondence.

• Phased Training: Three stages – (1) independent branch adaptation, (2) frozen backbone + JA training, (3) full joint fine-tuning. A Branch Dropout mechanism randomly drops depth or flow latents to improve robustness.

• Training Objective (Equation 7):

  $$\mathcal{L}_{\text{total}} = \sum_{m \in \mathcal{M}} \lambda_m \, \mathbb{E}\left[\left\| v_\theta^m(z_t^m, t, c)_{[1:]} - (\epsilon^m - z_0^m)_{[1:]} \right\|_2^2\right]$$

  with flow-matching on frames $1,\dots,T-1$ and a shared noise sample $\epsilon^\text{rgb}=\epsilon^\text{depth}=\epsilon^\text{flow}$.

4. RynnWorld-4D-Policy (Action Decoder)

Extracts intermediate hidden states from the frozen RynnWorld-4D encoder (block 15, diffusion timestep $t=500$), concatenates RGB/depth/flow latents into $\mathbf{F}_p \in \mathbb{R}^{B \times T \times 3C \times H \times W}$, and compresses them via a Flow Former (spatial cross-attention + temporal self-attention). Actions are generated with a flow matching head (4-step Euler ODE), predicting chunks of $K=10$ 54-dimensional actions.

Empirical Validation / Results

4D World Modeling Quality (Table 4)

RynnWorld-4D achieves the best performance across depth and flow metrics among all baselines, while maintaining competitive visual generation quality.

Significant drops in the ablation “w/o 4D Pre‑training” confirm the importance of large‑scale data.

Real‑world Manipulation Tasks (Table 5)

Success rates (%) on six tasks evaluated over 35 trials each.

The full 4D representation (RGB‑DF) consistently outperforms single‑ or two‑modality variants, especially on tasks requiring spatial precision (Hand‑over, Lid Placement, Bowl Stacking).

Inference Latency (Table 1)

With action chunking ($K=10$), the effective control frequency is ~9 Hz.

Theoretical and Practical Implications

• The RGB-DF projective 4D representation provides a natural bridge between video diffusion priors and robotic action spaces, enabling explicit 3D geometry and motion without abandoning the scalability of 2D generative models.
• By generating synchronized depth and optical flow, the model enforces physical consistency (e.g., optical flow aligns with depth changes), reducing temporal artifacts common in RGB‑only generation.
• The RynnWorld-4D-Policy demonstrates that internal 4D latents can be used as a predictive vision encoder for closed‑loop control, achieving state‑of‑the‑art performance in dexterous bimanual tasks without requiring iterative decoding of full videos during action execution.
• Practical deployment on an RTX 5090 yields a 9 Hz effective control rate, sufficient for human‑scale manipulation, with robustness provided by 3D‑aware planning (wider capture range, motion anticipation).

Conclusion

RynnWorld-4D presents a complete framework for 4D embodied world modeling: from data curation (Rynn4DDataset 1.0) to a tri‑branch diffusion model that co‑generates RGB, depth, and optical flow, to a policy head that leverages these predictive 4D features for real‑time control. Extensive experiments validate that explicit geometry and motion are crucial for precise manipulation. The main limitations are computational overhead (~1.1 s per world model forward pass) and a focus on egocentric perspectives; future work may extend to multi‑view or multi‑robot setups. The work establishes a promising foundation for building general‑purpose embodied intelligence that understands and interacts with the 3D world.