# RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation

> A generative world model synthesizes real-time robot videos from human hand poses, enabling zero-shot real-world policy learning without physical robots.

- **Source:** [arXiv](https://arxiv.org/abs/2607.06558)
- **Published:** 2026-07-09
- **Permalink:** https://picx.dev/p/pZ2fOz
- **Whiteboard:** https://picx.dev/p/pZ2fOz/image

## Summary

## Summary (Overview)
- **Digital teleoperation paradigm**: Replaces physical robots with an action-conditioned generative world model that synthesizes egocentric robot videos from a human hand-pose stream and a single reference image, decoupling data collection from hardware constraints.
- **RynnWorld-Teleop system**: Integrates three core innovations—depth-aware skeletal conditioning (renders hand joints with depth-modulated color/size), progressive human-to-robot training (pretrain on large-scale egocentric human videos, then fine-tune on paired robot data), and streaming autoregressive distillation (bidirectional teacher → causal student) for real-time 40+ FPS generation on a single H100 GPU.
- **Zero-shot Sim2Real transfer**: Policies trained exclusively on RynnWorld-Teleop-generated data achieve competitive success rates (e.g., 82.86% on Block Pushing, 77.14% on Bimanual Lifting) without any real robot data.
- **Data augmentation boosts real policy performance**: Adding 300 synthetic episodes to 300 real episodes consistently improves success rates across four bimanual tasks (e.g., π₀.₅ Lid Placement rises 20% from 42.86% to 62.86%).
- **Generalization to out-of-distribution scenes**: The model maintains high-fidelity synthesis when reference images are edited with unseen objects or backgrounds, enabling unbounded scene instantiation from a single image.

## Introduction and Theoretical Foundation
### Background and Motivation
Scaling robot learning requires massive, diverse trajectory data, but physical teleoperation is bottlenecked by fixed hardware, manual environment resets, and limited object diversity. Existing approaches fall short:
- **Human-to-robot video translation** (e.g., Phantom, Masquerade, H2R) bridges the visual gap but is passive and observation-only—it cannot produce action signals or support closed-loop interaction.
- **Action-conditioned egocentric world models** (e.g., Hand2World, GeneratedReality, CosHand) synthesize future frames from human action signals but remain human-centric (still render a human hand), leaving the embodiment gap unbridged.

### Three Requirements for Digital Teleoperation
A genuine digital teleoperation system must satisfy:
1. **Robot-centric** – the operator teleoperates a robot, not a human hand.
2. **Action-grounded** – every generated frame is tied to a recoverable joint-level action signal.
3. **Real-time** – the operator stays in the control loop (frame rates ≥ 30 Hz).

### Key Definitions
- **Digital teleoperation**: An operator’s real-time hand-pose stream drives a generative world model that, conditioned on a single reference image $I_{\text{ref}}$, synthesizes the egocentric video the robot *would have* produced. The hand-pose stream itself serves as an embodiment-agnostic action label transferable to any target robot via retargeting.
- **Action-conditioned world model**: Given reference image $I_{\text{ref}}$ and hand-gesture sequence $P = \{p_1, \dots, p_T\}$, synthesizes robotic egocentric videos $V = \{v_1, \dots, v_T\}$.

## Methodology
### 3.1 Preliminaries
Base model: Wan-I2V architecture (3D VAE + Transformer denoiser $F_\Theta$). Training follows conditional flow matching (CFM):

$$
z_t = (1 - t) z_0 + t\epsilon, \quad \epsilon \sim \mathcal{N}(0, I)
$$

$$

\mathcal{L}_{\text{CFM}} = \mathbb{E}_{t, z_0, \epsilon} \left[ \| v_\Theta(z_t, t, z_{\text{ref}}, c) - (\epsilon - z_0) \|_2^2 \right]
$$

where $z_0 = \mathcal{E}(I_V)$, $z_{\text{ref}} = \mathcal{E}(I_{\text{ref}})$, $c$ is the depth-aware skeletal control latent, and $v_\Theta$ predicts the velocity field.

### 3.2 Depth-Aware Action Representation
Hand poses are rendered as 2D skeletal videos from 21-joint tracking, with **depth-modulated rendering**: joint/bone color and diameter are dynamically scaled by camera-space depth (Fig. 2). This resolves depth ambiguity critical for modeling precise hand-object interactions. The rendered pose video is VAE-encoded to obtain control latent $c \in \mathbb{R}^{C \times T \times H \times W}$.

### 3.3 Action-Conditioned Video Generation
**Additive patch embedding with distribution alignment** (Eq. 3):
- A dedicated control patch-embedding layer $\text{PatchEmbed}^c_{C \to D}$ runs in parallel to the original $\text{PatchEmbed}^z_{C \to D}$.
- Outputs are fused via a learnable scalar gate $\alpha$:

$$
x = \text{PatchEmbed}^z_{C \to D}(z_t) + \alpha \cdot \text{PatchEmbed}^c_{C \to D}(e_c), \quad e_c = \frac{c - \mu_c}{\sigma_c} \cdot \sigma_z + \mu_z
$$

- $\alpha$ is initialized small (0.1) and $\text{PatchEmbed}^c$ is zero-initialized to preserve pretrained priors.

### 3.4 Progressive Cross-Domain Training
- **Stage 1: Egocentric Human Pretraining** – on EgoDex (81-frame clips) and VITRA (25-frame clips) to learn hand-object interaction dynamics. Data: ~105M frames → ~2.14M slices (Table 1).
- **Stage 2: Robotic Domain Adaptation** – fine-tune on 1,800 paired teleoperation episodes (four bimanual tasks, Table 2) with human gestures mapped to robot actions via inverse kinematics.

| Dataset | Type | Seq. Length | Total Frames | Total Slices |
|---------|------|-------------|--------------|--------------|
| VITRA (Li et al., 2025c) | Human | 25 | 30.7M | 1.23M |
| EgoDex (Hoque et al., 2025) | Human | 81 | 74.0M | 0.91M |
| Ours (Real-Robot) | Robot | 81 | 0.43M | 5.3K |

**Table 1:** Statistics of datasets used for pretraining and adaptation.

### 3.5 Autoregressive Distillation
Two stages to convert the bidirectional teacher into a causal student for real-time inference:
1. **Causal Flow-Matching Warm-up** – train student with causal temporal mask and KV cache using MSE loss on velocity prediction.
2. **Distribution Matching Distillation (DMD)** – 4-step sampling with a critic and frozen teacher, with gradient backpropagation through persisted KV cache across chunks to minimize boundary artifacts.

### 4. Digital Teleoperation System
**Pipeline** (Sec. 4.1):
1. **Retargeting**: Raw 6-DoF Vive tracker poses → damped least-squares IK → 54-D robot action vector (dual 7-DoF arms + dual 20-DoF dexterous hands).
2. **Skeletal-Conditioned Synthesis**: Render hand-pose stream into depth-aware skeletal video $S_{1:T}$ at 16 FPS, condition RynnWorld-Teleop on $I_{\text{ref}}$ and $S_{1:T}$ to generate robot execution video.
3. **Chunked Re-anchoring**: Generate in 81-frame chunks; for each subsequent chunk, re-anchor using the actual egocentric frame from the robot camera as new $I_{\text{ref}}$ to mitigate drift.

**Advantages over traditional simulation** (Sec. 4.2): zero 3D asset overhead, no visual domain gap (synthesizes real-world pixel distribution), implicit physics from human pretraining.

## Empirical Validation / Results
### Experiment Setup
- **Hardware**: TIANJI M6 mobile robot with dual arms + dual WUJI dexterous hands, egocentric RealSense D435i.
- **Tasks** (Fig. 4): Dual Picking, Block Pushing, Bimanual Lifting, Lid Placement – challenging bimanual coordination.
- **Evaluation**: Success rate over 35 real-world trials per task, with randomized object poses.

### 5.2 Digital Teleoperation System Evaluation
**Policy learning with generative data scaling** (Table 3):

| Method | Data Source | Dual Picking | Block Pushing | Bimanual Lifting | Lid Placement |
|--------|-------------|--------------|---------------|------------------|---------------|
| DP | 300 real | 82.86 | 85.71 | 88.57 | 57.14 |
| DP | 300 real + 300 RynnWorld-Teleop | **88.57** | **88.57** | **94.29** | **65.71** |
| π₀.₅ | 300 real | 94.29 | 100.00 | 94.29 | 42.86 |
| π₀.₅ | 300 real + 300 RynnWorld-Teleop | **97.14** | 97.14 | **100.00** | **62.86** |
| π₀ | 300 real | 88.57 | 94.29 | 91.43 | 34.29 |
| π₀ | 0 real + 300 RynnWorld-Teleop | 68.57 | 82.86 | 77.14 | 28.57 |
| π₀ | 300 real + 300 RynnWorld-Teleop | **94.29** | **100.00** | **97.14** | **54.29** |

**Table 3:** Augmenting real data with RynnWorld-Teleop-generated data consistently improves success rates across all tasks and policies. π₀ trained solely on synthetic data achieves competitive results.

**t-SNE analysis** (Fig. 5): High-dimensional feature embeddings of real and synthetic trajectories show significant overlap, confirming semantic alignment.

**Latency analysis**: Distilled causal student achieves 40.0 fps at 480×832 resolution on single H100 GPU, with per-frame latency ~25 ms breakdown: skeletal encoding (5%), causal DiT denoising (72%), VAE decoding (23%).

### 5.3 Action-Conditioned World Model Evaluation
**Quantitative results** (Table 4):

| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ | FPS ↑ |
|--------|--------|--------|---------|-------|-------|
| **Text-conditioned I2V** | | | | | |
| Wan-2.2-TI2V-5B | 18.61 | 0.772 | 0.373 | 1998 | 2.8 |
| **Action-conditioned** | | | | | |
| InterDyn | 21.47 | 0.831 | 0.279 | 655 | 2.9 |
| **RynnWorld-Teleop (SFT)** | **26.78** | **0.887** | **0.119** | **550** | 2.8 |
| **RynnWorld-Teleop-Causal** | 22.25 | 0.830 | 0.207 | 1226 | **40.0** |
| **Ablation** | | | | | |
| w/o Human Pre-training | 17.81 | 0.763 | 0.453 | 2598 | 2.8 |
| w/o DMD (Causal) | 19.25 | 0.777 | 0.244 | 1338 | 40.0 |

**Table 4:** RynnWorld-Teleop significantly outperforms all baselines in visual quality. The causal student (40 fps) maintains high quality while enabling real-time interaction. Pretraining and sequential distillation are critical.

**Key findings**:
- RynnWorld-Teleop reduces FVD from 1998 (Wan baseline) to 550, and from 655 (InterDyn) to 550.
- LoRA variant achieves 26.08 PSNR, 585 FVD – strong even with parameter-efficient tuning.
- Generalization to OOD visual states (Fig. 7): maintains temporal consistency under unseen objects and backgrounds, validating the claim that digital teleoperation can instantiate arbitrary scenes from a single reference image.

### 5.4 Ablation Study
- **Additive vs. concatenation fusion**: Concatenation collapses performance (FVD 1191 vs. 585) by disrupting pretrained latent distribution (Fig. 8).
- **Value of human pretraining**: Without it, FVD jumps to 2598; the model fails to generate coherent hand-object interactions (Fig. 9).
- **Sequential distillation**: Both causal warm-up and DMD are necessary; omitting causal warm-up leads to training instability and blur.

## Theoretical and Practical Implications
- **Paradigm shift**: Digital teleoperation replaces physical hardware with a generative world model, decoupling data collection from hardware availability, workspace constraints, and manual reset overhead. This could unlock orders-of-magnitude scaling in robot data acquisition.
- **Embodiment-agnostic actions**: The hand-pose stream serves as a universal action label, transferable to any robot via retargeting, enabling cross-embodiment data reuse.
- **Bridging human and robot domains**: Progressive training (human → robot) effectively transfers dexterous manipulation priors from large-scale human video datasets to robot-specific embodiments, addressing the scarcity of robot data.
- **Practical impact**: The system already works for complex bimanual tasks and achieves zero-shot Sim2Real. As a data augmentation tool, it consistently improves real policy performance by up to +20% on precision tasks like lid placement.
- **Limitations**:
  - Struggles with fine-grained physics (liquids, highly deformable objects).
  - Per-platform fine-tuning required; cross-embodiment foundation world models (conditioned on kinematic descriptors) are a promising direction.

## Conclusion
RynnWorld-Teleop is the first system to deliver **digital teleoperation** as a scalable data engine for robot learning. By combining depth-aware skeletal conditioning, progressive human-to-robot training, and streaming autoregressive distillation, it generates high-fidelity, action-consistent robot videos at 40+ FPS from a single reference image. Policies trained on its output achieve competitive zero-shot Sim2Real transfer, and augmenting real data with synthetic data consistently improves success rates. This work demonstrates that digital teleoperation can become a standard ingredient in scalable robot data pipelines, leveraging human intuition as a digital control signal to train general-purpose robotic agents without the constraints of physical infrastructure.

---

_Markdown view of https://picx.dev/p/pZ2fOz, served by PicX — AI-generated visual whiteboard summaries of research papers._