# PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations

> PDRVLA achieves state-of-the-art robot manipulation success rates by extracting compact physical tokens from intermediate VLM features, supervised via future prediction without costly online video generation.

- **Source:** [arXiv](https://arxiv.org/abs/2610.37381)
- **Published:** 2026-10-29
- **Permalink:** https://picx.dev/p/KPVrAS
- **Whiteboard:** https://picx.dev/p/KPVrAS/image

## Summary

# PDRVLA: A Vision–Language–Action Model with Physical Dynamics Representations

## Summary (Overview)

- **PDRVLA** is a Vision-Language-Action (VLA) model that introduces a lightweight **Physical Dynamics Module** to extract compact "physical tokens" from intermediate VLM features, conditioning action generation without requiring online video generation during deployment.
- The model learns physical representations through **direct prediction of future RGB, depth, and optical-flow latent features**, with auxiliary supervision from a **frozen pretrained video world model**, avoiding latency issues of in-the-loop video generation.
- The training corpus is **entirely built from publicly available datasets**, enriched through egocentric human-to-robot synthesis, subtask annotation, and embodied chain-of-thought (ECoT) annotation, totaling over 4.35M trajectories and 31.7M VL examples.
- PDRVLA achieves **state-of-the-art success rates** on four simulation benchmarks: **99.3%** on LIBERO, **89.9%** on LIBERO-Plus, **93.6%** on RoboTwin 2.0, and **81.5%** on SimplerEnv, surpassing prior VLA models including Qwen-RobotManip and InternVLA-A1.5.
- Ablation studies demonstrate that world-model-assisted supervision improves future prediction (especially RGB and depth), and characterize the effects of subtask conditioning, future object-pose supervision, and ECoT annotations on policy performance.

---

## Introduction and Theoretical Foundation

Vision–Language–Action (VLA) models connect visual observations and language instructions to robot control, leveraging pretrained vision-language models (VLMs) and heterogeneous robot demonstrations. However, their understanding of physical dynamics remains limited:

> "A recent analysis reports weak encoding of object physical properties across several VLA policies and unreliable adaptation of their motions to changes in object mass" (Xu, 2026).

For manipulation, recognizing the current scene is only part of the problem—grasping, moving, and placing objects change their configurations and spatial relationships. **Anticipating how a scene evolves** during interaction can provide useful context for action selection.

**Two competing approaches to physical dynamics:**

1. **Pretrained video world models** predict how scenes change over time, offering complementary dynamics learning. However, generating future videos *within the control loop* adds significant latency.
2. **Video prediction during training only**—shaping compact representations that condition a policy without online video generation (InternRobotics, 2026; Tang et al., 2026).

**PDRVLA's approach** follows the second direction: a lightweight Physical Dynamics Module extracts a fixed set of physical tokens from intermediate VLM features. These tokens are trained via direct supervision on future RGB, depth, and optical-flow features, with auxiliary supervision from a frozen video world model. At deployment, the policy uses the learned physical tokens **without running the video model or prediction heads**.

**Layer-wise motivation:** Later VLM layers may suppress visual information useful for predicting physical change. Studies have found stronger perceptual representations at *intermediate* layers of large language models (Jain et al., 2025; Kaduri et al., 2025), motivating the use of layer-$l=8$ intermediate features (out of 32 decoder blocks).

---

## Methodology

### Data Corpus

The training corpus is organized into five categories:

| Category | Statistics |
|---|---|
| **VLA Data** | |
| Real-Robot | 1.26M trajectories / 14,893 hours |
| UMI | 1.22M trajectories / 10,259 hours |
| Simulation | 1.87M trajectories / 8,868 hours |
| **VL Data** | |
| General VL | 8.05M VL examples |
| Spatial & Embodied VL | 23.63M VL examples |

**Data Augmentations:**

| Augmentation | Statistics |
|---|---|
| Egocentric Conversion | 0.90M robot trajectories / 2,646 hours |
| Subtask Annotation | 1.79B annotated frames / 1.1M VL examples |
| ECoT | 2.56M annotated keyframes |

**Data sources include:**
- **Real-robot:** RoboSet, DROID, BridgeData V2, BC-Z, RH20T, RoboMIND V1/V2, RoboCOIN, ABC-130K, Humanoid Everyday, Unitree UniFoLM, AgiBot World Colosseo, REASSEMBLE, FuSe, MolmoAct
- **UMI:** UMI, FastUMI, FastUMI-100K, Hy-Embodied-0.5, HiFi-UMI-2K, RealOmni-Open Data
- **Simulation:** ManiSkill2, Meta-World, RoboCasa, InternData-A1, GR00T-X-Embodiment-Sim, InternVLA-M1, RoboTwin 2.0, DexMimicGen
- **VL:** Cambrian-1, COCO-Caption, VQAv2, OneThinker, SpaceThinker, RoboPoint, Robo2VLM, RoboVQA, EO-1, Cosmos-Reason1, ProcCorpus-60M

### Data Processing

**Action Representation.** End-effector (EEF) poses are standardized: positions in meters, orientations as rotation matrices, right-handed coordinates ($+x$ forward, $+y$ left, $+z$ up). The EEF action consists of translation and rotation deltas expressed in reference frame $r$:

$$\Delta a_t^{\mathrm{eef}} = (\Delta \mathbf{p}_{r,t}, \Delta R_{r,t}), \qquad \Delta \mathbf{p}_{r,t} = \mathbf{p}_{r,t}^{*} - \mathbf{p}_{r,t}, \quad \Delta R_{r,t} = R_{re,t}^{*} R_{re,t}^{\mathsf{T}}$$

where $\mathbf{p}_{r,t}, \mathbf{p}_{r,t}^{*} \in \mathbb{R}^3$ are current/target positions, and $R_{re,t}, R_{re,t}^{*} \in SO(3)$ are current/target orientations mapping EEF frame $e$ into reference frame $r$. Gripper openings are normalized to $[0,1]$ (0 = closed, 1 = open).

**Cross-Embodiment Action Vector.** A fixed 58-dimensional vector with separate left/right manipulator slots, each reserving dimensions for: arm-joint positions (7D), EEF pose (9D reserved, 6D used), dexterous-hand joints (12D), and gripper opening (1D). Unsupported dimensions are padded and masked from loss.

### Model Architecture

#### Physical Dynamics Module

At timestep $t$, the VLM's **intermediate layer-$l$ features** $H_t^{(l)}$ (with $l=8$ for a 32-block backbone) are fed to the Physical Dynamics Module. Using **50 learnable queries** across 4 Transformer blocks (hidden width 1024, 4 attention heads, FFN expansion ratio 1), the module extracts 50 physical tokens $C_t$ of width 2560 via cross-attention and self-attention. The module adds **~57.3M parameters** (1.4% of the backbone).

#### Action Generation

The action expert receives final-layer VLM features $H_t^{\mathrm{last}}$, physical tokens $C_t$, and proprioceptive state $q_t$:

$$X_t = [H_t^{\mathrm{last}}; C_t]$$

The expert is a **Diffusion Transformer (DiT)** with **conditional flow matching** (~150M parameters). It predicts action chunks of $K=8$ robot commands. The 6D EEF action for step $k$:

$$\mathbf{a}_{t,k}^{\mathrm{eef}} = \left[ \mathbf{p}_{t+k}^{*} - \mathbf{p}_{t}, \operatorname{Log}(R_{t+k}^{*} R_{t}^{\mathsf{T}}) \right] \in \mathbb{R}^6$$

where $\operatorname{Log}: SO(3) \to \mathbb{R}^3$ maps relative rotation to axis–angle. At inference, 4 Euler steps integrate predicted velocity from Gaussian noise.

### Training

#### Physical Dynamics Pretraining

Two complementary objectives:

**1. Direct Future Prediction:** The physical tokens predict future RGB, depth, and optical-flow latent features (encoded by LingBot-Video's frozen Wan VAE). Pooled targets are standardized:

$$\widetilde{y}^m = \frac{y^m - \mu^m}{s^m}$$

with loss:

$$\mathcal{L}_{\mathrm{direct}} = \frac{1}{3} \sum_{m \in \{r,d,f\}} \operatorname{MSE}\left(g_\eta^m(C_t), \widetilde{y}^m\right)$$

**2. World-Model-Assisted Prediction:** A frozen LingBot-Video-Dense-1.3B receives RGB latent clips with physical tokens as conditioning. Future features are noise-corrupted via linear interpolation:

$$z_{\sigma,\mathrm{current}}^r = z_{\mathrm{current}}^r, \qquad z_{\sigma,\mathrm{future}}^r = (1 - \sigma) z_{\mathrm{future}}^r + \sigma \epsilon$$

with auxiliary loss:

$$\mathcal{L}_{\mathrm{WM}} = \sum_{m \in \{r,d,f\}} \mathrm{MSE}(\widehat{v}_{\mathrm{future}}^m, \epsilon - z_{\mathrm{future}}^m)$$

**Total pretraining objective:**

$$\mathcal{L}_{\mathrm{pre}} = \mathcal{L}_{\mathrm{WM}} + \lambda(k) \mathcal{L}_{\mathrm{direct}}$$

where $\lambda(k) = 0.5 \min(k/2000, 1)$ ramps up direct prediction weight over 2000 steps.

#### Joint Policy Training

The full objective combines action, VL, and direct prediction losses:

$$\mathcal{L}_{\mathrm{policy}} = \mathcal{L}_{\mathrm{act}} + 0.1 \mathcal{L}_{\mathrm{VL}} + 0.1 \mathcal{L}_{\mathrm{direct}}$$

The masked velocity-matching action loss over minibatch $\mathcal{B}$:

$$\mathcal{L}_{\mathrm{act}} = \frac{\sum_{i \in \mathcal{B}} \| M_i \odot (\widehat{v}_i - v_i^{*}) \|_F^2}{\sum_{i \in \mathcal{B}} \| M_i \|_1 + \delta}$$

with interpolation $x_\tau = (1-\tau)\xi + \tau A_t$ and target velocity $v^* = A_t - \xi$.

---

## Empirical Validation / Results

### Benchmark Performance

PDRVLA achieves **state-of-the-art results** across all four benchmarks:

| Method | LIBERO | LIBERO-Plus | RoboTwin 2.0 | SimplerEnv |
|---|---|---|---|---|
| $\pi_0$ | 94.4 | 53.6 | 62.2 | 27.1 |
| $\pi_{0.5}$ | 96.9 | 84.4 | 79.8 | 57.1 |
| GR00T-N1.7 | 97.0 | - | - | 59.3 |
| Cosmos-Policy | 98.5 | 82.2 | - | - |
| LingBot-VLA 2.0 | - | - | 93.2 | - |
| Qwen-RobotManip | 99.1 | 89.0 | 93.0 | - |
| InternVLA-A1.5 | 98.9 | 84.8 | 93.2 | 80.8 |
| **PDRVLA** | **99.3** | **89.9** | **93.6** | **81.5** |

**LIBERO breakdown:** 100.0% (Spatial), 99.8% (Object), 99.8% (Goal), 97.6% (Long). PDRVLA achieves perfect scores on Spatial, surpassing all baselines.

**LIBERO-Plus:** PDRVLA leads in sensor noise, camera viewpoints, light conditions, language instructions, and background textures. Robot-initial-state is the most challenging category (75.2%).

**RoboTwin 2.0:** 93.9% clean / 93.3% randomized, leading in both settings.

**SimplerEnv:** Strongest on Eggplant→Basket (97.6%) but weaker on Carrot→Plate (64.6%) compared to InternVLA-A1.5 (85.4%).

### Component Analyses

#### World-Model Supervision Improves Future Prediction

Held-out evaluation on 828 RoboCOIN clips shows **direct + world-model** outperforms direct-only:

| Target | Direct only | Direct + WM | Reduction (%) | 95% interval (%) |
|---|---|---|---|---|
| RGB | 0.2558 | 0.2454 | **4.06** | [3.04, 4.99] |
| Depth | 0.3154 | 0.3087 | **2.12** | [0.93, 3.31] |
| Flow | 1.3606 | 1.3522 | 0.62 | [-0.07, 1.28] |
| Mean | 0.6439 | 0.6354 | **1.32** | [0.71, 1.89] |

Training curves show **faster early convergence** in RGB and depth with world-model assistance; optical-flow gains are smaller.

#### Subtask Conditioning

Self-generated subtask descriptions raise LIBERO-Plus overall success from **89.93% to 90.23%**, with largest gains under robot-initial-state (+1.16) and camera-viewpoint (+1.00) perturbations, but small decreases under lighting, language, and texture changes.

#### Subtask Update Frequency (RoboCasa365)

| Refresh interval | Average success (%) |
|---|---|
| Baseline (no conditioning) | 26.25 |
| 1 chunk | 32.25 |
| 8 chunks | 32.13 |
| 32 chunks | 20.63 |

Subtask change rates: 11.6% (interval 1), 26.5% (interval 8), 56.1% (interval 32). An 8-chunk interval gives comparable aggregate success to per-chunk refreshes at lower computational cost.

#### ECoT Training and Generation

| Setting | LIBERO Long | LIBERO Avg | LIBERO-Plus Long | LIBERO-Plus Overall |
|---|---|---|---|---|
| No ECoT training | 92.8 | 97.1 | 65.18 | 79.59 |
| ECoT-trained, no generation | **96.0** | 98.2 | **69.87** | 79.25 |
| ECoT-trained, with generation | 95.6 | **98.3** | 68.72 | 79.12 |

**ECoT training improves success** (Long: 92.8→96.0 on LIBERO; 65.18→69.87 on LIBERO-Plus), but **inference-time generation provides no consistent additional gain**.

#### Exploratory Object-Pose Supervision

Adding auxiliary object-pose prediction heads increases LIBERO-Plus overall from 89.93% to **90.12%** (+0.19), with largest gain under camera-viewpoint changes (+1.38), but decreases under lighting (-0.87) and robot-initial-state (-0.64) perturbations.

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Physical tokens as compact dynamics representations:** The Physical Dynamics Module shows that a small set of learnable queries attending to intermediate VLM features can encode future appearance, geometry, and motion, challenging the assumption that full video generation is necessary for dynamics-aware action selection.

2. **Video world models as auxiliary supervisors:** Using a frozen pretrained video world model to guide representation learning during training—rather than at inference—demonstrates a principled way to avoid the latency-accuracy tradeoff inherent in in-the-loop video generation.

3. **Intermediate-layer features matter:** The choice of layer-$l=8$ intermediate features (motivated by findings that later VLM layers suppress perceptual detail) provides empirical support for layer-wise analysis of visual information in VLA models.

4. **Cross-embodiment generalization:** The 58-dimensional standardized action vector with per-embodiment masking enables joint training across heterogeneous robots (single-arm, bimanual, UMI, humanoid, dexterous hands) without architectural changes.

### Practical Implications

- **Deployment efficiency:** The policy uses physical tokens without running the video world model or prediction heads, keeping inference lightweight while retaining dynamics awareness.
- **Data efficiency through synthesis:** Egocentric human-to-robot synthesis (0.90M trajectories), subtask annotation (1.1M VL examples), and ECoT annotation (2.56M keyframes) substantially expand training data from public sources alone.
- **Subtask conditioning as a cost-accuracy knob:** The 8-chunk refresh interval provides near-optimal accuracy at reduced computational cost, with per-chunk refreshes mostly repeating the same subtask prediction.

---

## Conclusion

PDRVLA demonstrates that future prediction can be used to learn compact physical representations for action generation without online video generation. Direct supervision on future RGB, depth, and optical-flow features, augmented by a frozen pretrained video world model, shapes physical tokens that improve policy performance across four simulation benchmarks.

**Key takeaways:**
- State-of-the-art performance on LIBERO (99.3%), LIBERO-Plus (89.9%), RoboTwin 2.0 (93.6%), and SimplerEnv (81.5%)
- World-model assistance improves future prediction, especially RGB (4.06% error reduction) and depth (2.12%)
- ECoT training improves action success, but inference-time generation adds no consistent benefit
- Subtask conditioning helps most under viewpoint and initial-state perturbations

**Future directions** include exploring predictive targets beyond RGB, depth, and optical flow—such as **object-centric geometry, contact states, and material properties**—and investigating how these cues complement one another while retaining a lightweight policy.

---

_Markdown view of https://picx.dev/p/KPVrAS, served by PicX — AI-generated visual whiteboard summaries of research papers._