Summary (Overview)

• LingBot-Video is the first large-scale, open-source Mixture-of-Experts (MoE) video foundation model specifically tailored for embodied intelligence, bridging the gap between digital video generation and physical actuation.
• It adopts a sparse MoE Diffusion Transformer architecture to achieve a superior trade-off between modeling capacity and inference efficiency, scaling up to 120B total parameters.
• A dedicated data profiling engine systematically integrates internet-scale videos with robot-oriented footage (manipulation, navigation, egocentric) to inject embodiment priors.
• A multi-dimensional reward system enforces physical rationality, task completion, and action fidelity, extending beyond conventional aesthetic and text-alignment criteria.
• Comprehensive evaluations on internal benchmarks, RBench, Physics-IQ Verified, and user studies show state-of-the-art performance in text-and-image-to-video (TI2V) tasks and competitive results on embodied/physical domains, outperforming open-source baselines and rivaling commercial models.

Introduction and Theoretical Foundation

Despite the success of diffusion-based and autoregressive video models in content creation, they suffer from a fundamental domain mismatch when applied to embodied intelligence: they are optimized for perceptual quality (realism, aesthetics) rather than physical correctness, contact stability, or long-horizon consistency. Internet-scale video data lacks robot embodiment priors and precise interaction dynamics.

Existing efforts to bridge this gap fall short along three tightly coupled dimensions:

• Architecture: Dense computation in video transformers leads to prohibitive inference costs and limited scalability. Sparse Mixture-of-Experts (MoE) formulations, proven in LLMs, are under-explored for video generation.
• Data: Training corpora are dominated by internet videos without robot interaction dynamics.
• Training objectives: Current alignment strategies optimize aesthetic quality without explicit physical feasibility, task completion, or long-horizon reward signals.

LingBot-Video addresses these gaps through an integrated approach: a sparse MoE video diffusion framework for capacity-compute decoupling, a robot-augmented pretraining corpus for embodiment grounding, and a multi-dimensional reward system that incorporates physical plausibility and task-oriented success signals.

Methodology

Architecture: Task-Unified Single-Stream Diffusion Transformer

• Unified Input Formulation: Visual latent patches and condition tokens (from Qwen3-VL-4B) are projected to the same hidden dimension and concatenated along the sequence dimension, handling T2I, T2V, and TI2V tasks within a single framework.
• Single-Stream DiT: All tokens share the same transformer blocks, maximizing parameter reuse and enabling dense cross-modal interactions. Uses Multi-Modal 3D Rotary Position Embedding (3D RoPE) to separate condition and visual tokens while preserving video geometry. Implements QK-Norm (per-head RMSNorm) for training stability, and AdaLN-Single modulation for reduced overhead.
• Cascaded Refiner: A second-stage refiner upsamples generated video from 480p to 1080p using a conditional rectified flow, learning to restore high-frequency details from degraded low-resolution inputs.

Sparse Mixture-of-Experts (MoE) Scaling

• Each transformer block replaces its dense FFN with a token-choice sparse MoE layer comprising shared experts (always active, capturing general priors) and routed experts (selected per token via sigmoid router).
• Group-limited routing (inspired by DeepSeekMoE) divides routed experts into groups, selecting top-K groups then top-K experts within those groups to control communication cost.
• Online bias correction maintains load balance without auxiliary loss: bias $b_j$ is updated at each step: $b_j \leftarrow b_j - \eta\,\text{sign}(n_j - \bar{n})$, with optional mean-centering.
• Sequence-wise auxiliary loss encourages balanced expert usage within each packed video sequence:

$$L_{\text{seq}} = \frac{1}{S}\sum_{s=1}^{S}\sum_{j=1}^{N_r} f^{(s)}_j P^{(s)}_j$$

where $P^{(s)}_j = \frac{1}{T_s}\sum_{t=1}^{T_s} p_{t,j}$ and $f^{(s)}_j = \frac{N_r}{K_r T_s}c^{(s)}_j$.

• Recipe Exploration: Experiments show that scaling expert count (64→128→256) under fixed active compute (1.4B) improves loss, and fine-grained routing (8/128 experts) outperforms coarse routing (4/64 experts) under the same total parameter budget.

Data Infrastructure

• Data Profiling Engine: Extracts multi-dimensional attributes (structural, semantic, motion, camera, quality) into structured records using VLMs and specialized detectors (TransNetV2, LocoTrack, HPSv3, OmniAID).
• World-Knowledge Topological Graph: Organizes samples along a semantic concept tree (50K leaf concepts, 1K intermediate, 25 top-level groups) and an action tree (several hundred canonical action nodes). Enables distribution-aware sampling and data diagnosis.
• Dense Structured Captioning: All training data is annotated with JSON captions covering images, videos, VLA (robot manipulation), and egocentric footage. Captions include comprehensive scene descriptions, camera attributes, timestamped actions for each element.
• Caption Rewriter: A two-stage pipeline (Expand–Map) at inference time converts brief user prompts into the structured caption format, closing the train–inference distribution gap.
• Data Curriculum: Five-stage progressive training evolving from 192p images → 192p video+image → 480p → 480p challenge-focused → 1080p refinement. Injection of >70,000 hours of embodiment footage (VLA, navigation, egocentric) occurs in Stage 2.

Training

• Progressive Pre-Training: Five stages gradually introducing complexity (image priors → temporal learning → multi-task conditioning → distribution harmonization → high-resolution refinement).
• Post-Training (RL): On-policy GRPO with multi-aspect rewards (Vision Quality, Text-Video Alignment, Dynamic Degree, Motion Coherence, Human-Motion Consistency, Physical Plausibility). Uses single-step exploration with Coefficients-Preserving Sampling (CPS) and timestep-balanced gradient reweighting.
• Negative-Aware Finetuning: Pairs real videos (chosen) with generated videos (rejected) using a DiffusionNFT-style forward-process optimization to mitigate reward hacking.
• Action-to-Video Post-Training (LingBot-Video-A2V): Adapts the model for action-conditioned world modeling. Actions are converted to relative commands, embedded, and injected as residual signals into transformer blocks.
• Distillation: Distills the model into a few-step generator using DMD2 (distribution matching + GAN objective).

Empirical Validation / Results

Internal Benchmark

Evaluates across General Quality (Motion Quality, Prompt Following, Visual Consistency, Aesthetic Quality) and Embodied Domain (Human Interaction, Physical Simulation, Robotics, Egocentric Perspective, Navigation). Two settings: T2V and TI2V. LingBot-Video achieves state-of-the-art in TI2V on both general and embodied scores, and is competitive in T2V, particularly excelling in embodied domain (second overall but above Cosmos).

Public Benchmarks

RBench (650 robot-centric prompts, task-oriented and embodiment-specific categories):

Table 1: RBench evaluation results (full table with sub-dimensions in paper). LingBot-Video ranks first among all models on average score.

Physics-IQ Verified (66 controlled physical experiments, I2V setting): LingBot-Video scores 40.4, ranking first among open-source models, ahead of Cosmos 3 (39.5), Hunyuan Video 1.5 (33.4), and Wan 2.2 A14B (32.2).

User Study

Good-Same-Bad (GSB) pairwise evaluation against 6 open-source and 4 commercial models. For T2V: LingBot-Video outperforms Wan 2.2 5B/14B, LongCat-Video, LTX-2.3; competitive with HunyuanVideo 1.5 and Cosmos 3. For TI2V: Consistently higher Good than Bad rates against all open-source baselines, with large margins over Wan 2.2 5B and clear gains over Cosmos 3, LTX-2.3, Wan 2.2 14B, LongCat-Video, HunyuanVideo 1.5.

Action-to-Video Post-Training

LingBot-Video-A2V demonstrates better adherence to physical laws and action following on out-of-distribution evaluation (EgoDex Eval, DreamDojo-HV Eval) compared to baseline DreamDojo.

Theoretical and Practical Implications

• Capacity-Compute Decoupling: The MoE architecture empirically demonstrates that scaling total parameters (up to 120B) under a fixed active compute budget consistently reduces training loss, validating that sparse routing provides a larger repository for physical-world priors without inflating inference cost. Fine-grained expert specialization outperforms coarse routing.
• Data-Centric Embodiment Grounding: The integrated data pipeline (profiling, topological graph, structured captions, curriculum) effectively injects robot interaction priors into a video foundation model, enabling physically plausible generation in manipulation, navigation, and egocentric domains.
• Multi-Dimensional RL Post-Training: The decoupled reward system (covering visual quality, alignment, dynamics, coherence, human motion, physics) and on-policy GRPO with single-step exploration provide fine-grained optimization signals that improve temporal consistency and physical realism, as validated by user studies and Physics-IQ scores.
• Practical Impact for Robotics: LingBot-Video is positioned as a video simulator serving three roles: synthetic data engine, safe policy evaluator, and action planner. The open-source release (checkpoints, code, Diffusers-compatible package) lowers the barrier for the robotics community to adopt video foundation models for embodied AI.

Conclusion

LingBot-Video is a pioneering open-source MoE video foundation model designed for embodied intelligence, successfully bridging digital creativity and physical actuation. Key achievements:

• Scalable sparse MoE DiT architecture achieving superior capacity-efficiency trade-off.
• Comprehensive data infrastructure enabling integration of internet-scale and robot-oriented footage.
• Novel post-training with multi-aspect RL and negative-aware finetuning.

The paper provides a complete pipeline from architecture design through progressive pre-training, post-training, and distillation, validated on multiple benchmarks and user studies. Future directions include strengthening the model as a world simulator for robotics (data engine, policy evaluator, action planner), and collaborative community development to push the boundaries of embodied physical engines and next-generation robot brains. The model is fully open-sourced at https://github.com/robbyant/lingbot-video.