PDRVLA: A Vision–Language–Action Model with Physical Dynamics Representations

Summary (Overview)

• PDRVLA is a Vision-Language-Action (VLA) model that introduces a lightweight Physical Dynamics Module to extract compact "physical tokens" from intermediate VLM features, conditioning action generation without requiring online video generation during deployment.
• The model learns physical representations through direct prediction of future RGB, depth, and optical-flow latent features, with auxiliary supervision from a frozen pretrained video world model, avoiding latency issues of in-the-loop video generation.
• The training corpus is entirely built from publicly available datasets, enriched through egocentric human-to-robot synthesis, subtask annotation, and embodied chain-of-thought (ECoT) annotation, totaling over 4.35M trajectories and 31.7M VL examples.
• PDRVLA achieves state-of-the-art success rates on four simulation benchmarks: 99.3% on LIBERO, 89.9% on LIBERO-Plus, 93.6% on RoboTwin 2.0, and 81.5% on SimplerEnv, surpassing prior VLA models including Qwen-RobotManip and InternVLA-A1.5.
• Ablation studies demonstrate that world-model-assisted supervision improves future prediction (especially RGB and depth), and characterize the effects of subtask conditioning, future object-pose supervision, and ECoT annotations on policy performance.

---

Introduction and Theoretical Foundation

Vision–Language–Action (VLA) models connect visual observations and language instructions to robot control, leveraging pretrained vision-language models (VLMs) and heterogeneous robot demonstrations. However, their understanding of physical dynamics remains limited:

"A recent analysis reports weak encoding of object physical properties across several VLA policies and unreliable adaptation of their motions to changes in object mass" (Xu, 2026).

For manipulation, recognizing the current scene is only part of the problem—grasping, moving, and placing objects change their configurations and spatial relationships. Anticipating how a scene evolves during interaction can provide useful context for action selection.

Two competing approaches to physical dynamics:

1. Pretrained video world models predict how scenes change over time, offering complementary dynamics learning. However, generating future videos within the control loop adds significant latency.
2. Video prediction during training only—shaping compact representations that condition a policy without online video generation (InternRobotics, 2026; Tang et al., 2026).

PDRVLA's approach follows the second direction: a lightweight Physical Dynamics Module extracts a fixed set of physical tokens from intermediate VLM features. These tokens are trained via direct supervision on future RGB, depth, and optical-flow features, with auxiliary supervision from a frozen video world model. At deployment, the policy uses the learned physical tokens without running the video model or prediction heads.

Layer-wise motivation: Later VLM layers may suppress visual information useful for predicting physical change. Studies have found stronger perceptual representations at intermediate layers of large language models (Jain et al., 2025; Kaduri et al., 2025), motivating the use of layer-$l=8$ intermediate features (out of 32 decoder blocks).

---

Methodology

Data Corpus

The training corpus is organized into five categories:

Data Augmentations:

Data sources include:

• Real-robot: RoboSet, DROID, BridgeData V2, BC-Z, RH20T, RoboMIND V1/V2, RoboCOIN, ABC-130K, Humanoid Everyday, Unitree UniFoLM, AgiBot World Colosseo, REASSEMBLE, FuSe, MolmoAct
• UMI: UMI, FastUMI, FastUMI-100K, Hy-Embodied-0.5, HiFi-UMI-2K, RealOmni-Open Data
• Simulation: ManiSkill2, Meta-World, RoboCasa, InternData-A1, GR00T-X-Embodiment-Sim, InternVLA-M1, RoboTwin 2.0, DexMimicGen
• VL: Cambrian-1, COCO-Caption, VQAv2, OneThinker, SpaceThinker, RoboPoint, Robo2VLM, RoboVQA, EO-1, Cosmos-Reason1, ProcCorpus-60M

Data Processing

Action Representation. End-effector (EEF) poses are standardized: positions in meters, orientations as rotation matrices, right-handed coordinates ($+x$ forward, $+y$ left, $+z$ up). The EEF action consists of translation and rotation deltas expressed in reference frame $r$:

$$\Delta a_t^{\mathrm{eef}} = (\Delta \mathbf{p}_{r,t}, \Delta R_{r,t}), \qquad \Delta \mathbf{p}_{r,t} = \mathbf{p}_{r,t}^{*} - \mathbf{p}_{r,t}, \quad \Delta R_{r,t} = R_{re,t}^{*} R_{re,t}^{\mathsf{T}}$$

where $\mathbf{p}_{r,t}, \mathbf{p}_{r,t}^{*} \in \mathbb{R}^3$ are current/target positions, and $R_{re,t}, R_{re,t}^{*} \in SO(3)$ are current/target orientations mapping EEF frame $e$ into reference frame $r$. Gripper openings are normalized to $[0,1]$ (0 = closed, 1 = open).

Cross-Embodiment Action Vector. A fixed 58-dimensional vector with separate left/right manipulator slots, each reserving dimensions for: arm-joint positions (7D), EEF pose (9D reserved, 6D used), dexterous-hand joints (12D), and gripper opening (1D). Unsupported dimensions are padded and masked from loss.

Model Architecture

Physical Dynamics Module

At timestep $t$, the VLM's intermediate layer-$l$ features $H_t^{(l)}$ (with $l=8$ for a 32-block backbone) are fed to the Physical Dynamics Module. Using 50 learnable queries across 4 Transformer blocks (hidden width 1024, 4 attention heads, FFN expansion ratio 1), the module extracts 50 physical tokens $C_t$ of width 2560 via cross-attention and self-attention. The module adds ~57.3M parameters (1.4% of the backbone).

Action Generation

The action expert receives final-layer VLM features $H_t^{\mathrm{last}}$, physical tokens $C_t$, and proprioceptive state $q_t$:

$$X_t = [H_t^{\mathrm{last}}; C_t]$$

The expert is a Diffusion Transformer (DiT) with conditional flow matching (~150M parameters). It predicts action chunks of $K=8$ robot commands. The 6D EEF action for step $k$:

$$\mathbf{a}_{t,k}^{\mathrm{eef}} = \left[ \mathbf{p}_{t+k}^{*} - \mathbf{p}_{t}, \operatorname{Log}(R_{t+k}^{*} R_{t}^{\mathsf{T}}) \right] \in \mathbb{R}^6$$

where $\operatorname{Log}: SO(3) \to \mathbb{R}^3$ maps relative rotation to axis–angle. At inference, 4 Euler steps integrate predicted velocity from Gaussian noise.

Training

Physical Dynamics Pretraining

Two complementary objectives:

1. Direct Future Prediction: The physical tokens predict future RGB, depth, and optical-flow latent features (encoded by LingBot-Video's frozen Wan VAE). Pooled targets are standardized:

$$\widetilde{y}^m = \frac{y^m - \mu^m}{s^m}$$

with loss:

$$\mathcal{L}_{\mathrm{direct}} = \frac{1}{3} \sum_{m \in \{r,d,f\}} \operatorname{MSE}\left(g_\eta^m(C_t), \widetilde{y}^m\right)$$

2. World-Model-Assisted Prediction: A frozen LingBot-Video-Dense-1.3B receives RGB latent clips with physical tokens as conditioning. Future features are noise-corrupted via linear interpolation:

$$z_{\sigma,\mathrm{current}}^r = z_{\mathrm{current}}^r, \qquad z_{\sigma,\mathrm{future}}^r = (1 - \sigma) z_{\mathrm{future}}^r + \sigma \epsilon$$

with auxiliary loss:

$$\mathcal{L}_{\mathrm{WM}} = \sum_{m \in \{r,d,f\}} \mathrm{MSE}(\widehat{v}_{\mathrm{future}}^m, \epsilon - z_{\mathrm{future}}^m)$$

Total pretraining objective:

$$\mathcal{L}_{\mathrm{pre}} = \mathcal{L}_{\mathrm{WM}} + \lambda(k) \mathcal{L}_{\mathrm{direct}}$$

where $\lambda(k) = 0.5 \min(k/2000, 1)$ ramps up direct prediction weight over 2000 steps.

Joint Policy Training

The full objective combines action, VL, and direct prediction losses:

$$\mathcal{L}_{\mathrm{policy}} = \mathcal{L}_{\mathrm{act}} + 0.1 \mathcal{L}_{\mathrm{VL}} + 0.1 \mathcal{L}_{\mathrm{direct}}$$

The masked velocity-matching action loss over minibatch $\mathcal{B}$:

$$\mathcal{L}_{\mathrm{act}} = \frac{\sum_{i \in \mathcal{B}} \| M_i \odot (\widehat{v}_i - v_i^{*}) \|_F^2}{\sum_{i \in \mathcal{B}} \| M_i \|_1 + \delta}$$

with interpolation $x_\tau = (1-\tau)\xi + \tau A_t$ and target velocity $v^* = A_t - \xi$.

---

Empirical Validation / Results

Benchmark Performance

PDRVLA achieves state-of-the-art results across all four benchmarks:

LIBERO breakdown: 100.0% (Spatial), 99.8% (Object), 99.8% (Goal), 97.6% (Long). PDRVLA achieves perfect scores on Spatial, surpassing all baselines.

LIBERO-Plus: PDRVLA leads in sensor noise, camera viewpoints, light conditions, language instructions, and background textures. Robot-initial-state is the most challenging category (75.2%).

RoboTwin 2.0: 93.9% clean / 93.3% randomized, leading in both settings.

SimplerEnv: Strongest on Eggplant→Basket (97.6%) but weaker on Carrot→Plate (64.6%) compared to InternVLA-A1.5 (85.4%).

Component Analyses

World-Model Supervision Improves Future Prediction

Held-out evaluation on 828 RoboCOIN clips shows direct + world-model outperforms direct-only:

Training curves show faster early convergence in RGB and depth with world-model assistance; optical-flow gains are smaller.

Subtask Conditioning

Self-generated subtask descriptions raise LIBERO-Plus overall success from 89.93% to 90.23%, with largest gains under robot-initial-state (+1.16) and camera-viewpoint (+1.00) perturbations, but small decreases under lighting, language, and texture changes.

Subtask Update Frequency (RoboCasa365)

Subtask change rates: 11.6% (interval 1), 26.5% (interval 8), 56.1% (interval 32). An 8-chunk interval gives comparable aggregate success to per-chunk refreshes at lower computational cost.

ECoT Training and Generation

ECoT training improves success (Long: 92.8→96.0 on LIBERO; 65.18→69.87 on LIBERO-Plus), but inference-time generation provides no consistent additional gain.

Exploratory Object-Pose Supervision

Adding auxiliary object-pose prediction heads increases LIBERO-Plus overall from 89.93% to 90.12% (+0.19), with largest gain under camera-viewpoint changes (+1.38), but decreases under lighting (-0.87) and robot-initial-state (-0.64) perturbations.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Physical tokens as compact dynamics representations: The Physical Dynamics Module shows that a small set of learnable queries attending to intermediate VLM features can encode future appearance, geometry, and motion, challenging the assumption that full video generation is necessary for dynamics-aware action selection.

2. Video world models as auxiliary supervisors: Using a frozen pretrained video world model to guide representation learning during training—rather than at inference—demonstrates a principled way to avoid the latency-accuracy tradeoff inherent in in-the-loop video generation.

3. Intermediate-layer features matter: The choice of layer-$l=8$ intermediate features (motivated by findings that later VLM layers suppress perceptual detail) provides empirical support for layer-wise analysis of visual information in VLA models.

4. Cross-embodiment generalization: The 58-dimensional standardized action vector with per-embodiment masking enables joint training across heterogeneous robots (single-arm, bimanual, UMI, humanoid, dexterous hands) without architectural changes.

Practical Implications

• Deployment efficiency: The policy uses physical tokens without running the video world model or prediction heads, keeping inference lightweight while retaining dynamics awareness.
• Data efficiency through synthesis: Egocentric human-to-robot synthesis (0.90M trajectories), subtask annotation (1.1M VL examples), and ECoT annotation (2.56M keyframes) substantially expand training data from public sources alone.
• Subtask conditioning as a cost-accuracy knob: The 8-chunk refresh interval provides near-optimal accuracy at reduced computational cost, with per-chunk refreshes mostly repeating the same subtask prediction.

---

Conclusion

PDRVLA demonstrates that future prediction can be used to learn compact physical representations for action generation without online video generation. Direct supervision on future RGB, depth, and optical-flow features, augmented by a frozen pretrained video world model, shapes physical tokens that improve policy performance across four simulation benchmarks.

Key takeaways:

• State-of-the-art performance on LIBERO (99.3%), LIBERO-Plus (89.9%), RoboTwin 2.0 (93.6%), and SimplerEnv (81.5%)
• World-model assistance improves future prediction, especially RGB (4.06% error reduction) and depth (2.12%)
• ECoT training improves action success, but inference-time generation adds no consistent benefit
• Subtask conditioning helps most under viewpoint and initial-state perturbations

Future directions include exploring predictive targets beyond RGB, depth, and optical flow—such as object-centric geometry, contact states, and material properties—and investigating how these cues complement one another while retaining a lightweight policy.