Back to News

Zhejiang University and Unitree Open-Source Movens-5M Robot Pretraining Dataset

#robotics#dataset#pretraining#unitree

Zhejiang University and Unitree's joint lab released Movens-5M, a pretraining dataset of 5 million robot trajectories derived from human videos, totaling over 10,000 hours and covering 297 actions across 16 scenarios. The team also introduced Movens-Zero, a model trained on this data that generates whole-body robot motions from text, video, speech, and music inputs, demonstrated on the Unitree G1. Scaling experiments suggest the dataset supports continued model improvement with more data and larger modules.

Coverage timeline

  1. 机器之心机器之心

    浙江大学与宇树科技具身智能联合实验室 等团队从海量互联网人类视频中构建并开源了预训练数据集 Movens-5M ,收录 500 万条经仿真执行验证的机器人动作,总时长超过 1 万小时 。 数据集覆盖行走、深蹲、拖地、拳击等 297 种细分动作,涵盖居家生活、健身训练、户外休闲等 16 类场景。 团队将文字指令、视频示范、语音和音乐与机器人动作配对,实现 多模态对齐 ,并提供分时段、分身体部位的动作说明。研究者可以用这些数据训练模型,让模型学习不同形式的输入分别对应怎样的动作。 基于这套数据训练的动作模型 Movens-Zero ,能根据上述四种输入生成机器人全身动作。团队在宇树 G1 上展示了真机执行效果,并通过 Scaling(规模扩展)实验检验:数据更多、动作生成模块更大,模型能否持续进步。 Movens-Zero 研究概览。图源:论文首页与图1。 人类擦门视频,怎样成为机器人训练数据? 一条能用于模型训练的数据,除了视频,还需要包含什么?可以先看一段清洁视频。 画面中,人物俯身擦拭门板中下部,随后逐渐起身,将手臂抬向高处。这条数据记录通过四路画面,展示了 原始视频、从视频中还原的人体三维动作、转换成适合 G1 的动作,以及模拟环境中的 G1 执行结果 。 清洁数据案例。四列依次为原始视频(Original Video)、人体动作(Recovered Motion)、G1参考动作(G1 Reference)、G1模拟执行(G1 Tracking)。 系统先从视频中估计人的三维动作,再按机器人的身体结构调整这些动作。随后,系统让模拟环境中的 G1 执行调整后的动作,检查机器人能否稳定完成。 数据记录还配有文字说明,描述不同时间段内各个身体部位的动作 。这段 7.25 秒的视频可分为三个阶段: 0–4 秒 :人物俯身,左手持绿色清洁布往返擦拭门板中下部,右臂弯曲,放在身体后侧。 4–6 秒 :人物逐渐直起上身,左臂随擦拭位置抬高。 6–7.25 秒 :双臂向上抬起,伸向门板较高的位置。 这些文字就像视频的分步说明,记录了人物什么时候弯腰、哪只手在擦拭、什么时候起身抬臂。每一步的说明都有对应的画面和机器人动作,帮助模型学习根据文字要求生成动作的能力。 数据中也有与物体交互的动作,哑铃训练和杠铃硬拉就是其中的例子。哑铃训练包含弯举和俯身;杠铃硬拉包含上身前倾、慢慢放低杠

Related stories

  • MIT Han Lab's PEAR Trains Qwen3.8-27B for Real Robot Arms via Sim RL, Open-Sourced

    MIT Han Lab introduced PEAR (Post-training Engine for Agentic Robotics), a post-training project that uses reinforcement learning in simulation to train the open-source Qwen3.8-27B model to operate a real dual-arm robot, requiring no human teleoperation demonstrations or real-robot data. On an unseen YAM robot, PEAR achieved 18 successes in 30 tests (up from 0 before training), runs at over 220 tokens per second on an RTX 5090, and maintains performance on nine general benchmarks. The project is open-sourced with code, model weights, and a paper available online.

    1 reports · 1 sources
  • Shanghai AI4S open-sources REMORY, a 1.9B memory module for recursive self-improvement

    The Shanghai Academy of AI for Science, with Fudan University and Shanghai Chuangzhi Institute, open-sourced REMORY, a 1.9B-parameter memory module that enables small models to run independently for days via recursive self-improvement, achieving results near GPT-6.1-Terra. Trained fully self-supervised on two H800 GPUs, REMORY leads several key benchmarks and is available on arXiv, Hugging Face, GitHub, and the Xinghe Qizhi platform.

    1 reports · 1 sources
  • Study suggests GPT-6 Astra's robot skills come from specialized training data, not emergent general intelligence

    A technical analysis speculates that GPT-6 Astra's robot-manipulation abilities may stem from specialized robot training data rather than emergent general intelligence. The study points to demos with NVIDIA and Unitree, and notes that OpenAI has not disclosed whether pre-training included robot-specific data. Similar training is suggested for GPT-6.1 Sol and Gemini Argon.

    1 reports · 1 sources
  • Q Labs proposes Dust, a zeroth-order optimizer that rivals backprop in Transformer pretraining up to 243M params

    Q Labs, a small research lab backed by Jeff Dean, published a paper introducing Dust, a zeroth-order optimization method that adds Gaussian noise to activations per token and estimates gradients via a virtual population. They claim it is the first such method to compete with backpropagation in Transformer pretraining, with experiments up to 243M parameters and 20M tokens. Third-party replication from SOTAAZ Lab confirmed basic results but found backprop leads when given better optimizers or more epochs, and Dust is far more computationally expensive, so its significance lies in demonstrating feasibility rather than replacing backprop.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe