Summary (Overview)

• PhysBrain 1.5 is a unified 8B-parameter physical foundation model that jointly learns embodied understanding, action generation, and future-state prediction through a shared autoregressive framework built on Qwen3-VL-Instruct.
• The model represents language responses, end-effector trajectories, and dense visual targets (RGB, depth, robot masks) as discrete tokens, optimized via masked next-token prediction under a common vocabulary and output head.
• Pre-training derives all embodied supervision from human interaction videos (egocentric, ego–exocentric, and panoramic sources), while supervised fine-tuning incorporates human, real-robot, and simulated interaction data.
• On 28 embodied understanding benchmarks, PhysBrain 1.5 achieves an average score of 72.5, surpassing all open-source baselines (best prior: Hy-Embodied-VLM-1.0 at 66.0) and approaching proprietary models such as GPT-6-Astra (73.3) and Gemini 3.6 Flash (73.0).
• The model retains general multimodal capabilities comparable to its base model, and qualitative results demonstrate end-effector trajectory generation and spatially coherent future-frame prediction across diverse embodiments.

---

Introduction and Theoretical Foundation

The Physical Loop

The paper frames its contribution around the physical loop: an agent observes its surroundings, interprets spatial relationships and task goals, anticipates action outcomes, and acts on the environment. The changed world then provides new observations that inform subsequent understanding and behavior. A physical foundation model must provide reusable capabilities for understanding, acting, and predicting within this loop.

Limitations of Existing Approaches

• General vision–language models (VLMs) provide visual understanding, semantic knowledge, and instruction following, but lack action generation and state-transition modeling.
• System-level approaches (SayCan, Code as Policies, VoxPoser) connect language reasoning with robot capabilities through skills, programs, or spatial control representations, but rely on specialized components rather than unified learning.
• Recent embodied foundation models (UniVLA, RynnVLA-002, Cosmos 3) explore shared learning of understanding, action generation, and visual prediction, but do not fully unify all three capabilities in a single framework.

Core Design Principle

PhysBrain 1.5 unifies these capabilities by expressing language responses, end-effector motion, and dense visual targets as discrete sequences learned through a common autoregressive interface. This enables joint training where:

• Perceptual supervision provides context for predicting interactions and their consequences.
• Action and state-transition supervision introduces physical priors into the shared backbone.

The approach extends the authors' earlier work, Phys-Brain 1.0 [36], which converted human egocentric video into physical-commonsense supervision; PhysBrain 1.5 extends this understanding-first approach to joint learning across the full physical loop.

---

Methodology

2.1 Unified Architecture

PhysBrain 1.5 is built from the Qwen3-VL-Instruct (8B) family. The central design:

1. Unified vocabulary: Language tokens, action tokens (from ActionPiece), and visual-state tokens share a common embedding table and language-model output head.
2. Task-specific loss masks: Depending on the task, the model is optimized with masked next-token prediction to answer in language, produce an action trajectory, or generate a visual-state sequence.
3. No separate perception head: Embodied understanding retains the base VLM's general interface, enriched through embodied supervision.

2.2 Action Representation

Actions are modeled as short end-effector trajectories:

• Human-derived and robot action data share the same 10D end-effector representation and ActionPiece vocabulary.
• No single globally canonicalized coordinate frame is imposed; instead, the action immediately preceding the current observation serves as local motion context.

2.3 Future-State Prediction

Given the current RGB observation and task instruction, the model predicts the future physical state $S_{t+\Delta}$:

$$S_{t+\Delta} = f(\text{RGB}_t, \text{instruction}, \text{action history})$$

The predicted state includes:

• Future RGB images
• Depth maps
• Robot masks (segmentation targets covering the robot body)

All annotations describe the physical state at a common timestamp in the same image coordinate system.

2.4 Training Stages

Two-stage training with ms-swift (Megatron-Core backend):

Stage 1: Physical-Aware Pre-Training

• Derived entirely from human interaction videos.
• Physical perception data: structured annotations, captions, and QA pairs from sampled frames/clips; segment-level captions align fine-grained actions with manipulated objects.
• Human action data: motion trajectories recovered via the Human-as-Humanoid pipeline [38], with wrist end-effector trajectories extracted from the kinematic chain.
• Future-state data: pairs of pre-interaction scene context with subsequent physical states, preserving temporal correspondences between action segments and observations.
• Includes 14.9M general language and vision-language instruction samples.

Stage 2: Embodied Supervised Fine-Tuning

• Combines high-quality human interaction data, real-robot trajectories, and simulated interactions.
• Includes 1M general instruction samples to retain broad capabilities.
• Native annotations converted into instruction-formatted language or spatial targets.
• Grounding and visual-trajectory data use a shared normalized spatial format for points, boxes, regions, and image-space waypoints.

---

Empirical Validation / Results

3.1 Embodied Understanding Benchmarks (28 benchmarks, 5 groups)

Key findings:

• Best open-source result on 14 benchmarks.
• Surpasses the strongest open-source baseline (Hy-Embodied-VLM-1.0) by 6.5 points.
• Outperforms the base model Qwen3-VL-Instruct (8B) on all 28 benchmarks.
• Approaches leading proprietary models: within 0.8 points of GPT-6-Astra and 0.5 of Gemini 3.6 Flash.
• Outperforms Claude Opus 5 by 4.6 points despite being much smaller.

Benchmark groups evaluated:

1. Visual-spatial perception
2. Spatial and multi-view understanding
3. Embodied cognition and planning
4. Spatial grounding
5. Visual trace and trajectory reasoning (ShareRobot-Trajectory [90], VABench-Visual-Trace [91])

3.2 General Multimodal Understanding

• Evaluated against Qwen3-VL-Instruct (8B) as reference.
• Performance across the general suite is comparable to the base model, indicating retained broad perception, reasoning, and commonsense capabilities.

3.3 Qualitative Results

• Action generation: Given current RGB observation, instruction, and preceding 16-step ground-truth action chunk, the model autoregressively generates the next 16 action tokens. Predicted and ground-truth end-effector positions are overlaid on observations (solid = ground truth, dashed = predictions).
• Future visual prediction: The model generates future RGB images, depth maps, and robot masks that are spatially coherent and task-relevant, illustrating joint modeling of environment and robot evolution.

---

Theoretical and Practical Implications

Theoretical Significance

• Demonstrates that understanding, action, and prediction can be jointly learned through a single autoregressive objective, challenging the traditional separation of perception and control modules.
• Validates the understanding-first approach: human interaction videos provide sufficient embodied supervision for learning physical priors that transfer to robot embodiments.
• Shows that task-centered episodes coupling semantic/spatial annotations with recovered motion and subsequent observations are an effective data organization principle.

Practical Implications

• Open-source accessibility: An 8B model achieving near-proprietary performance makes state-of-the-art embodied understanding available to the broader research community.
• Unified interface: The same model handles language responses, spatial coordinates, waypoint sequences, action trajectories, and visual-state generation, simplifying deployment.
• Data efficiency: Human interaction videos serve as a scalable, low-cost source of embodied supervision, reducing reliance on expensive robot data collection.

---

Conclusion

PhysBrain 1.5 demonstrates that a compact 8B model can achieve state-of-the-art open-source performance in embodied understanding while retaining general multimodal capabilities and supporting action generation and future-state prediction. The key contributions are:

1. Unified tokenization of language, action, and visual-state outputs under a shared autoregressive objective.
2. Human-interaction-driven pre-training that derives all embodied supervision from videos.
3. SFT with diverse data sources (human, real-robot, simulated) for embodiment generalization.

Future directions identified by the authors:

• Scaling the volume of training data further.
• Expanding modality coverage (e.g., richer sensor inputs).
• Incorporating richer interaction experience (e.g., more diverse tasks and embodiments).

The results support the thesis that learning from interaction experience is an effective route toward developing the capabilities needed for the physical loop.