# Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence

> LingBot-Video is the first open-source MoE video foundation model achieving state-of-the-art embodied video generation via robot-augmented data and multi-dimensional RL.

- **Source:** [arXiv](https://arxiv.org/abs/2607.07675)
- **Published:** 2026-07-10
- **Permalink:** https://picx.dev/p/ydBeEB
- **Whiteboard:** https://picx.dev/p/ydBeEB/image

## Summary

## Summary (Overview)

- **LingBot-Video** is the first large-scale, open-source Mixture-of-Experts (MoE) video foundation model specifically tailored for embodied intelligence, bridging the gap between digital video generation and physical actuation.
- It adopts a **sparse MoE Diffusion Transformer** architecture to achieve a superior trade-off between modeling capacity and inference efficiency, scaling up to 120B total parameters.
- A dedicated **data profiling engine** systematically integrates internet-scale videos with robot-oriented footage (manipulation, navigation, egocentric) to inject embodiment priors.
- A **multi-dimensional reward system** enforces physical rationality, task completion, and action fidelity, extending beyond conventional aesthetic and text-alignment criteria.
- Comprehensive evaluations on internal benchmarks, RBench, Physics-IQ Verified, and user studies show **state-of-the-art performance in text-and-image-to-video (TI2V) tasks and competitive results on embodied/physical domains**, outperforming open-source baselines and rivaling commercial models.

## Introduction and Theoretical Foundation

Despite the success of diffusion-based and autoregressive video models in content creation, they suffer from a **fundamental domain mismatch** when applied to embodied intelligence: they are optimized for perceptual quality (realism, aesthetics) rather than physical correctness, contact stability, or long-horizon consistency. Internet-scale video data lacks robot embodiment priors and precise interaction dynamics.

Existing efforts to bridge this gap fall short along three tightly coupled dimensions:
- **Architecture**: Dense computation in video transformers leads to prohibitive inference costs and limited scalability. Sparse Mixture-of-Experts (MoE) formulations, proven in LLMs, are under-explored for video generation.
- **Data**: Training corpora are dominated by internet videos without robot interaction dynamics.
- **Training objectives**: Current alignment strategies optimize aesthetic quality without explicit physical feasibility, task completion, or long-horizon reward signals.

LingBot-Video addresses these gaps through an integrated approach: a **sparse MoE video diffusion framework** for capacity-compute decoupling, a **robot-augmented pretraining corpus** for embodiment grounding, and a **multi-dimensional reward system** that incorporates physical plausibility and task-oriented success signals.

## Methodology

### Architecture: Task-Unified Single-Stream Diffusion Transformer

- **Unified Input Formulation**: Visual latent patches and condition tokens (from Qwen3-VL-4B) are projected to the same hidden dimension and concatenated along the sequence dimension, handling T2I, T2V, and TI2V tasks within a single framework.
- **Single-Stream DiT**: All tokens share the same transformer blocks, maximizing parameter reuse and enabling dense cross-modal interactions. Uses **Multi-Modal 3D Rotary Position Embedding (3D RoPE)** to separate condition and visual tokens while preserving video geometry. Implements **QK-Norm** (per-head RMSNorm) for training stability, and **AdaLN-Single** modulation for reduced overhead.
- **Cascaded Refiner**: A second-stage refiner upsamples generated video from 480p to 1080p using a conditional rectified flow, learning to restore high-frequency details from degraded low-resolution inputs.

### Sparse Mixture-of-Experts (MoE) Scaling

- Each transformer block replaces its dense FFN with a **token-choice sparse MoE layer** comprising **shared experts** (always active, capturing general priors) and **routed experts** (selected per token via sigmoid router).
- **Group-limited routing** (inspired by DeepSeekMoE) divides routed experts into groups, selecting top-K groups then top-K experts within those groups to control communication cost.
- **Online bias correction** maintains load balance without auxiliary loss: bias $b_j$ is updated at each step: $b_j \leftarrow b_j - \eta\,\text{sign}(n_j - \bar{n})$, with optional mean-centering.
- **Sequence-wise auxiliary loss** encourages balanced expert usage within each packed video sequence:
$$L_{\text{seq}} = \frac{1}{S}\sum_{s=1}^{S}\sum_{j=1}^{N_r} f^{(s)}_j P^{(s)}_j$$
where $P^{(s)}_j = \frac{1}{T_s}\sum_{t=1}^{T_s} p_{t,j}$ and $f^{(s)}_j = \frac{N_r}{K_r T_s}c^{(s)}_j$.
- **Recipe Exploration**: Experiments show that scaling expert count (64→128→256) under fixed active compute (1.4B) improves loss, and **fine-grained routing** (8/128 experts) outperforms coarse routing (4/64 experts) under the same total parameter budget.

### Data Infrastructure

- **Data Profiling Engine**: Extracts multi-dimensional attributes (structural, semantic, motion, camera, quality) into structured records using VLMs and specialized detectors (TransNetV2, LocoTrack, HPSv3, OmniAID).
- **World-Knowledge Topological Graph**: Organizes samples along a semantic concept tree (50K leaf concepts, 1K intermediate, 25 top-level groups) and an action tree (several hundred canonical action nodes). Enables distribution-aware sampling and data diagnosis.
- **Dense Structured Captioning**: All training data is annotated with JSON captions covering images, videos, VLA (robot manipulation), and egocentric footage. Captions include comprehensive scene descriptions, camera attributes, timestamped actions for each element.
- **Caption Rewriter**: A two-stage pipeline (Expand–Map) at inference time converts brief user prompts into the structured caption format, closing the train–inference distribution gap.
- **Data Curriculum**: Five-stage progressive training evolving from 192p images → 192p video+image → 480p → 480p challenge-focused → 1080p refinement. Injection of >70,000 hours of embodiment footage (VLA, navigation, egocentric) occurs in Stage 2.

### Training

- **Progressive Pre-Training**: Five stages gradually introducing complexity (image priors → temporal learning → multi-task conditioning → distribution harmonization → high-resolution refinement).
- **Post-Training (RL)**: On-policy GRPO with **multi-aspect rewards** (Vision Quality, Text-Video Alignment, Dynamic Degree, Motion Coherence, Human-Motion Consistency, Physical Plausibility). Uses **single-step exploration** with Coefficients-Preserving Sampling (CPS) and timestep-balanced gradient reweighting.
- **Negative-Aware Finetuning**: Pairs real videos (chosen) with generated videos (rejected) using a DiffusionNFT-style forward-process optimization to mitigate reward hacking.
- **Action-to-Video Post-Training (LingBot-Video-A2V)**: Adapts the model for action-conditioned world modeling. Actions are converted to relative commands, embedded, and injected as residual signals into transformer blocks.
- **Distillation**: Distills the model into a few-step generator using DMD2 (distribution matching + GAN objective).

## Empirical Validation / Results

### Internal Benchmark

Evaluates across **General Quality** (Motion Quality, Prompt Following, Visual Consistency, Aesthetic Quality) and **Embodied Domain** (Human Interaction, Physical Simulation, Robotics, Egocentric Perspective, Navigation). Two settings: T2V and TI2V. LingBot-Video achieves **state-of-the-art in TI2V** on both general and embodied scores, and is **competitive in T2V**, particularly excelling in embodied domain (second overall but above Cosmos).

### Public Benchmarks

**RBench** (650 robot-centric prompts, task-oriented and embodiment-specific categories):

| Model | Type | Avg. | Tasks | Embodiments |
|-------|------|------|-------|-------------|
| LingBot-Video | open-source | **0.620** | 0.578 | 0.643 |
| Cosmos3 Super | open-source | 0.581 | 0.487 | 0.642 |
| Wan 2.2 A14B | open-source | 0.507 | 0.381 | 0.454 |
| HunyuanVideo 1.5 | open-source | 0.460 | 0.442 | 0.316 |
| Wan 2.6 | closed-source | 0.607 | 0.546 | 0.656 |
| Seedance 1.5 pro | closed-source | 0.584 | 0.577 | 0.495 |
| Veo 3 | closed-source | 0.563 | 0.521 | 0.508 |

*Table 1: RBench evaluation results (full table with sub-dimensions in paper). LingBot-Video ranks first among all models on average score.*

**Physics-IQ Verified** (66 controlled physical experiments, I2V setting): LingBot-Video scores **40.4**, ranking first among open-source models, ahead of Cosmos 3 (39.5), Hunyuan Video 1.5 (33.4), and Wan 2.2 A14B (32.2).

### User Study

Good-Same-Bad (GSB) pairwise evaluation against 6 open-source and 4 commercial models. For **T2V**: LingBot-Video outperforms Wan 2.2 5B/14B, LongCat-Video, LTX-2.3; competitive with HunyuanVideo 1.5 and Cosmos 3. For **TI2V**: **Consistently higher Good than Bad rates against all open-source baselines**, with large margins over Wan 2.2 5B and clear gains over Cosmos 3, LTX-2.3, Wan 2.2 14B, LongCat-Video, HunyuanVideo 1.5.

### Action-to-Video Post-Training

LingBot-Video-A2V demonstrates better adherence to physical laws and action following on out-of-distribution evaluation (EgoDex Eval, DreamDojo-HV Eval) compared to baseline DreamDojo.

## Theoretical and Practical Implications

- **Capacity-Compute Decoupling**: The MoE architecture empirically demonstrates that scaling total parameters (up to 120B) under a fixed active compute budget consistently reduces training loss, validating that sparse routing provides a larger repository for physical-world priors without inflating inference cost. Fine-grained expert specialization outperforms coarse routing.
- **Data-Centric Embodiment Grounding**: The integrated data pipeline (profiling, topological graph, structured captions, curriculum) effectively injects robot interaction priors into a video foundation model, enabling physically plausible generation in manipulation, navigation, and egocentric domains.
- **Multi-Dimensional RL Post-Training**: The decoupled reward system (covering visual quality, alignment, dynamics, coherence, human motion, physics) and on-policy GRPO with single-step exploration provide fine-grained optimization signals that improve temporal consistency and physical realism, as validated by user studies and Physics-IQ scores.
- **Practical Impact for Robotics**: LingBot-Video is positioned as a **video simulator** serving three roles: synthetic data engine, safe policy evaluator, and action planner. The open-source release (checkpoints, code, Diffusers-compatible package) lowers the barrier for the robotics community to adopt video foundation models for embodied AI.

## Conclusion

LingBot-Video is a pioneering open-source MoE video foundation model designed for embodied intelligence, successfully bridging digital creativity and physical actuation. Key achievements:
- Scalable sparse MoE DiT architecture achieving superior capacity-efficiency trade-off.
- Comprehensive data infrastructure enabling integration of internet-scale and robot-oriented footage.
- Novel post-training with multi-aspect RL and negative-aware finetuning.

The paper provides a complete pipeline from architecture design through progressive pre-training, post-training, and distillation, validated on multiple benchmarks and user studies. Future directions include strengthening the model as a world simulator for robotics (data engine, policy evaluator, action planner), and collaborative community development to push the boundaries of embodied physical engines and next-generation robot brains. The model is fully open-sourced at [https://github.com/robbyant/lingbot-video](https://github.com/robbyant/lingbot-video).

---

_Markdown view of https://picx.dev/p/ydBeEB, served by PicX — AI-generated visual whiteboard summaries of research papers._