Summary (Overview)

• Digital teleoperation paradigm: Replaces physical robots with an action-conditioned generative world model that synthesizes egocentric robot videos from a human hand-pose stream and a single reference image, decoupling data collection from hardware constraints.
• RynnWorld-Teleop system: Integrates three core innovations—depth-aware skeletal conditioning (renders hand joints with depth-modulated color/size), progressive human-to-robot training (pretrain on large-scale egocentric human videos, then fine-tune on paired robot data), and streaming autoregressive distillation (bidirectional teacher → causal student) for real-time 40+ FPS generation on a single H100 GPU.
• Zero-shot Sim2Real transfer: Policies trained exclusively on RynnWorld-Teleop-generated data achieve competitive success rates (e.g., 82.86% on Block Pushing, 77.14% on Bimanual Lifting) without any real robot data.
• Data augmentation boosts real policy performance: Adding 300 synthetic episodes to 300 real episodes consistently improves success rates across four bimanual tasks (e.g., π₀.₅ Lid Placement rises 20% from 42.86% to 62.86%).
• Generalization to out-of-distribution scenes: The model maintains high-fidelity synthesis when reference images are edited with unseen objects or backgrounds, enabling unbounded scene instantiation from a single image.

Introduction and Theoretical Foundation

Background and Motivation

Scaling robot learning requires massive, diverse trajectory data, but physical teleoperation is bottlenecked by fixed hardware, manual environment resets, and limited object diversity. Existing approaches fall short:

• Human-to-robot video translation (e.g., Phantom, Masquerade, H2R) bridges the visual gap but is passive and observation-only—it cannot produce action signals or support closed-loop interaction.
• Action-conditioned egocentric world models (e.g., Hand2World, GeneratedReality, CosHand) synthesize future frames from human action signals but remain human-centric (still render a human hand), leaving the embodiment gap unbridged.

Three Requirements for Digital Teleoperation

A genuine digital teleoperation system must satisfy:

1. Robot-centric – the operator teleoperates a robot, not a human hand.
2. Action-grounded – every generated frame is tied to a recoverable joint-level action signal.
3. Real-time – the operator stays in the control loop (frame rates ≥ 30 Hz).

Key Definitions

• Digital teleoperation: An operator’s real-time hand-pose stream drives a generative world model that, conditioned on a single reference image $I_{\text{ref}}$, synthesizes the egocentric video the robot would have produced. The hand-pose stream itself serves as an embodiment-agnostic action label transferable to any target robot via retargeting.
• Action-conditioned world model: Given reference image $I_{\text{ref}}$ and hand-gesture sequence $P = \{p_1, \dots, p_T\}$, synthesizes robotic egocentric videos $V = \{v_1, \dots, v_T\}$.

Methodology

3.1 Preliminaries

Base model: Wan-I2V architecture (3D VAE + Transformer denoiser $F_\Theta$). Training follows conditional flow matching (CFM):

$$z_t = (1 - t) z_0 + t\epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$ $$\mathcal{L}_{\text{CFM}} = \mathbb{E}_{t, z_0, \epsilon} \left[ \| v_\Theta(z_t, t, z_{\text{ref}}, c) - (\epsilon - z_0) \|_2^2 \right]$$

where $z_0 = \mathcal{E}(I_V)$, $z_{\text{ref}} = \mathcal{E}(I_{\text{ref}})$, $c$ is the depth-aware skeletal control latent, and $v_\Theta$ predicts the velocity field.

3.2 Depth-Aware Action Representation

Hand poses are rendered as 2D skeletal videos from 21-joint tracking, with depth-modulated rendering: joint/bone color and diameter are dynamically scaled by camera-space depth (Fig. 2). This resolves depth ambiguity critical for modeling precise hand-object interactions. The rendered pose video is VAE-encoded to obtain control latent $c \in \mathbb{R}^{C \times T \times H \times W}$.

3.3 Action-Conditioned Video Generation

Additive patch embedding with distribution alignment (Eq. 3):

• A dedicated control patch-embedding layer $\text{PatchEmbed}^c_{C \to D}$ runs in parallel to the original $\text{PatchEmbed}^z_{C \to D}$.
• Outputs are fused via a learnable scalar gate $\alpha$:

$$x = \text{PatchEmbed}^z_{C \to D}(z_t) + \alpha \cdot \text{PatchEmbed}^c_{C \to D}(e_c), \quad e_c = \frac{c - \mu_c}{\sigma_c} \cdot \sigma_z + \mu_z$$

• $\alpha$ is initialized small (0.1) and $\text{PatchEmbed}^c$ is zero-initialized to preserve pretrained priors.

3.4 Progressive Cross-Domain Training

• Stage 1: Egocentric Human Pretraining – on EgoDex (81-frame clips) and VITRA (25-frame clips) to learn hand-object interaction dynamics. Data: ~105M frames → ~2.14M slices (Table 1).
• Stage 2: Robotic Domain Adaptation – fine-tune on 1,800 paired teleoperation episodes (four bimanual tasks, Table 2) with human gestures mapped to robot actions via inverse kinematics.

Table 1: Statistics of datasets used for pretraining and adaptation.

3.5 Autoregressive Distillation

Two stages to convert the bidirectional teacher into a causal student for real-time inference:

1. Causal Flow-Matching Warm-up – train student with causal temporal mask and KV cache using MSE loss on velocity prediction.
2. Distribution Matching Distillation (DMD) – 4-step sampling with a critic and frozen teacher, with gradient backpropagation through persisted KV cache across chunks to minimize boundary artifacts.

4. Digital Teleoperation System

Pipeline (Sec. 4.1):

1. Retargeting: Raw 6-DoF Vive tracker poses → damped least-squares IK → 54-D robot action vector (dual 7-DoF arms + dual 20-DoF dexterous hands).
2. Skeletal-Conditioned Synthesis: Render hand-pose stream into depth-aware skeletal video $S_{1:T}$ at 16 FPS, condition RynnWorld-Teleop on $I_{\text{ref}}$ and $S_{1:T}$ to generate robot execution video.
3. Chunked Re-anchoring: Generate in 81-frame chunks; for each subsequent chunk, re-anchor using the actual egocentric frame from the robot camera as new $I_{\text{ref}}$ to mitigate drift.

Advantages over traditional simulation (Sec. 4.2): zero 3D asset overhead, no visual domain gap (synthesizes real-world pixel distribution), implicit physics from human pretraining.

Empirical Validation / Results

Experiment Setup

• Hardware: TIANJI M6 mobile robot with dual arms + dual WUJI dexterous hands, egocentric RealSense D435i.
• Tasks (Fig. 4): Dual Picking, Block Pushing, Bimanual Lifting, Lid Placement – challenging bimanual coordination.
• Evaluation: Success rate over 35 real-world trials per task, with randomized object poses.

5.2 Digital Teleoperation System Evaluation

Policy learning with generative data scaling (Table 3):

Table 3: Augmenting real data with RynnWorld-Teleop-generated data consistently improves success rates across all tasks and policies. π₀ trained solely on synthetic data achieves competitive results.

t-SNE analysis (Fig. 5): High-dimensional feature embeddings of real and synthetic trajectories show significant overlap, confirming semantic alignment.

Latency analysis: Distilled causal student achieves 40.0 fps at 480×832 resolution on single H100 GPU, with per-frame latency ~25 ms breakdown: skeletal encoding (5%), causal DiT denoising (72%), VAE decoding (23%).

5.3 Action-Conditioned World Model Evaluation

Quantitative results (Table 4):

Table 4: RynnWorld-Teleop significantly outperforms all baselines in visual quality. The causal student (40 fps) maintains high quality while enabling real-time interaction. Pretraining and sequential distillation are critical.

Key findings:

• RynnWorld-Teleop reduces FVD from 1998 (Wan baseline) to 550, and from 655 (InterDyn) to 550.
• LoRA variant achieves 26.08 PSNR, 585 FVD – strong even with parameter-efficient tuning.
• Generalization to OOD visual states (Fig. 7): maintains temporal consistency under unseen objects and backgrounds, validating the claim that digital teleoperation can instantiate arbitrary scenes from a single reference image.

5.4 Ablation Study

• Additive vs. concatenation fusion: Concatenation collapses performance (FVD 1191 vs. 585) by disrupting pretrained latent distribution (Fig. 8).
• Value of human pretraining: Without it, FVD jumps to 2598; the model fails to generate coherent hand-object interactions (Fig. 9).
• Sequential distillation: Both causal warm-up and DMD are necessary; omitting causal warm-up leads to training instability and blur.

Theoretical and Practical Implications

• Paradigm shift: Digital teleoperation replaces physical hardware with a generative world model, decoupling data collection from hardware availability, workspace constraints, and manual reset overhead. This could unlock orders-of-magnitude scaling in robot data acquisition.
• Embodiment-agnostic actions: The hand-pose stream serves as a universal action label, transferable to any robot via retargeting, enabling cross-embodiment data reuse.
• Bridging human and robot domains: Progressive training (human → robot) effectively transfers dexterous manipulation priors from large-scale human video datasets to robot-specific embodiments, addressing the scarcity of robot data.
• Practical impact: The system already works for complex bimanual tasks and achieves zero-shot Sim2Real. As a data augmentation tool, it consistently improves real policy performance by up to +20% on precision tasks like lid placement.
• Limitations:
  • Struggles with fine-grained physics (liquids, highly deformable objects).
  • Per-platform fine-tuning required; cross-embodiment foundation world models (conditioned on kinematic descriptors) are a promising direction.

Conclusion

RynnWorld-Teleop is the first system to deliver digital teleoperation as a scalable data engine for robot learning. By combining depth-aware skeletal conditioning, progressive human-to-robot training, and streaming autoregressive distillation, it generates high-fidelity, action-consistent robot videos at 40+ FPS from a single reference image. Policies trained on its output achieve competitive zero-shot Sim2Real transfer, and augmenting real data with synthetic data consistently improves success rates. This work demonstrates that digital teleoperation can become a standard ingredient in scalable robot data pipelines, leveraging human intuition as a digital control signal to train general-purpose robotic agents without the constraints of physical infrastructure.