Summary (Overview)

• Qwen-VLA is a unified vision-language-action (VLA) foundation model that extends Qwen's vision-language modeling stack to continuous action and trajectory generation via a DiT-based flow-matching action decoder, enabling a single model to handle manipulation, navigation, and trajectory prediction across diverse robot embodiments.
• The model introduces embodiment-aware prompt conditioning, where robot-specific textual descriptions specify the current platform, control convention, and prediction horizon, allowing cross-embodiment learning without separate per-embodiment policies.
• A large-scale joint pretraining recipe combines robot manipulation trajectories (74.2%), human egocentric demonstrations (6.0%), navigation data (7.5%), synthetic simulation data (3.7%), and auxiliary vision-language data (8.5%) to provide both low-level motor priors and high-level semantic reasoning.
• A four-stage progressive training recipe (T2A → CPT → SFT → RL) bridges the gap between discrete vision-language tokens and continuous action trajectories, separating action-prior compression, visual grounding, task specialization, and success-driven refinement.
• As a unified generalist, Qwen-VLA-Instruct achieves state-of-the-art results: 97.9% on LIBERO, 73.7% on Simpler-WidowX, 86.1/87.2% on RoboTwin-Easy/Hard, 69.0% OSR on R2R, 59.6% SR on RxR, 76.9% average OOD success in real-world ALOHA experiments, and 26.6% zero-shot success on DOMINO dynamic manipulation.

Introduction and Theoretical Foundation

Embodied intelligence aims to build agents that perceive the physical world, understand natural-language instructions, reason over spatial and temporal context, and execute actions. However, most existing embodied systems remain specialized to narrow task families, robot embodiments, or evaluation settings—manipulation models are trained for tabletop or dexterous control, while navigation models focus on waypoint prediction in indoor environments.

Key insight: Despite surface-level heterogeneity across tasks (different observation formats, control frequencies, prediction horizons, action dimensionalities, and evaluation protocols), all embodied decision-making problems share a common computational structure: an agent must condition on visual observations, language instructions, and embodiment-specific constraints, then predict future actions or trajectories that are physically and semantically aligned with the task.

This motivates a unified formulation for embodied modeling:

$$p_{\theta}(y_{t:t+H-1} \mid o_t, x, e, z)$$

where $o_t$ is the visual context, $x$ is the language instruction, $e$ is the embodiment description, and $z$ is an optional task identifier. The target sequence $y_{t:t+H-1}$ is task-dependent but represented in a unified action-and-trajectory space—covering end-effector poses for manipulation, waypoints for navigation, future trajectories for driving, and human hand/body motion for egocentric data.

Methodology

Model Architecture

Vision-language backbone: Qwen3.5, a natively multimodal model with early vision-language fusion. Visual tokens from a ViT with spatial merging are interleaved directly into the text token stream, enabling unified processing of images, videos, and language. A hybrid attention design combines gated linear attention with grouped-query softmax attention for efficient long-sequence encoding.

Action expert: A single-stream DiT-style flow-matching policy (~1.15B parameters) that concatenates VLM hidden states with a noisy action chunk, processing them through joint self-attention with AdaLN timestep conditioning and multi-section RoPE. It is trained with a flow-matching objective and produces actions via Euler integration steps at inference.

Unified Action Representation

Each training sample contributes a target tensor $\mathbf{Y} \in \mathbb{R}^{H \times K}$, where $H$ is the prediction horizon and $K$ is a fixed channel dimension. Control signals include:

• Manipulation: delta end-effector position $(\Delta x, \Delta y, \Delta z)$, rotations, joint positions, gripper aperture, dexterous-hand angles
• Navigation: $(\Delta x, \Delta y, \Delta \theta)$ per waypoint

A per-channel binary mask $\mathbf{M} \in \{0,1\}^{H \times K}$ records valid channels, preventing padded entries from influencing gradients.

Training Objectives

The flow-matching action loss uses conditional flow matching with two-level averaging:

$$\ell_k = \frac{\sum_{h=1}^{H} M_{h,k} \left\| (v_{\theta}(\mathbf{Y}_{\tau}, \tau \mid o_{1:t}, x, e, z) - (\mathbf{Y}_1 - \mathbf{Y}_0))_{h,k} \right\|_2^2}{\sum_{h=1}^{H} M_{h,k}}$$ $$\mathcal{L}_{\mathrm{act}} = \mathbb{E}_{\tau, \mathbf{Y}_0, \mathbf{Y}_1} \left[ \frac{1}{c} \sum_{k=0}^{c-1} \ell_k \right]$$

The vision-language loss is a standard next-token prediction objective. Total loss: $\mathcal{L} = \lambda_{\mathrm{act}} \mathcal{L}_{\mathrm{act}} + \lambda_{\mathrm{vl}} \mathcal{L}_{\mathrm{vl}}$.

Four-Stage Training Recipe

1. Stage I (T2A): Freeze VLM, train DiT on text-to-action prediction without images, building a language-indexed action prior
2. Stage II (CPT): Unfreeze both modules, train on heterogeneous mixture combining simulation and real-robot trajectories
3. Stage III (SFT): Two parallel branches—multi-task SFT (VQA, grounding, manipulation, navigation) and real-robot teleoperation fine-tuning
4. Stage IV (RL): PPO with GAE, using sparse binary rewards from SimplerEnv; log-probabilities computed via SDE conversion of the probability-flow ODE

Pretraining Data Mixture

The synthetic simulation pipeline generated over 8M trajectories across 6 task templates and 6 robot configurations, with vision-conditioned data (359,848 trajectories) and language-action data (~7.2M trajectories) providing complementary supervision.

Empirical Validation / Results

Simulation Manipulation Results

Table 4: Robot manipulation results across benchmarks (specialists vs. generalist)

Qwen-VLA-Instruct outperforms most specialists despite being a single all-in-one model, demonstrating that joint multi-embodiment training does not sacrifice task-specific performance.

Real-World ALOHA Results

In-domain performance (Table 5): Qwen-VLA-aloha_w/pretrain achieves 83.6% average success vs. 48.5% for the from-scratch variant, confirming pretraining's critical role. It outperforms GR00T N1.6 (28.6%) and π₀.₅ (71.6%).

OOD generalization (Table 6): Qwen-VLA-aloha_w/pretrain achieves 76.9% average OOD success across color, instance, position, background, and instruction generalization—outperforming π₀.₅ by 35.4 percentage points.

Navigation Results

Qwen-VLA-Instruct achieves 69.0% OSR on R2R and 59.6% SR on RxR benchmarks, demonstrating strong vision-language navigation capabilities within the unified model.

Zero-Shot Transfer

The model achieves 26.6% zero-shot success on DOMINO dynamic manipulation, demonstrating that the unified action representation and embodiment-aware prompting enable transfer to unseen embodiments without additional fine-tuning.

Theoretical and Practical Implications

Theoretical contributions:

• Provides empirical evidence that heterogeneous embodied decision-making problems (manipulation, navigation, trajectory prediction) can be unified within a single vision-language-action model, supporting the view that these tasks are "different manifestations of a shared action-and-trajectory prediction problem"
• The compression perspective on action learning (T2A stage) offers a principled framework for bridging the dimensionality gap between compact language instructions and high-dimensional action trajectories
• Demonstrates that embodiment-aware prompt conditioning is sufficient for cross-embodiment transfer without architectural changes

Practical implications:

• A single generalist model can replace multiple specialist policies, reducing deployment complexity and enabling zero-shot transfer to new embodiments via prompt replacement
• Large-scale pretraining (especially with synthetic data) substantially improves OOD robustness across scene layout, background, lighting, object configuration, and embodiment variations
• The four-stage training recipe provides a practical template for co-training cognitive backbones with randomly initialized motor decoders, addressing the asymmetric optimization challenge
• The decoupled client-server RL infrastructure enables scalable on-policy optimization with simulation environments

Conclusion

Qwen-VLA demonstrates that manipulation, navigation, and trajectory-centric embodied tasks can be unified within a single vision-language-action model. The key innovations—embodiment-aware prompt conditioning, unified action-and-trajectory representation, large-scale heterogeneous pretraining, and progressive training—enable a generalist policy that matches or exceeds specialist performance across diverse benchmarks while providing robust OOD generalization.

Future directions identified by the authors include:

• Extending the conditioning context with episodic memory or persistent state for long-horizon planning and failure recovery
• Co-predicting future visual states alongside actions to unify action generation with world modeling
• Further scaling the pretraining mixture and exploring additional embodiments and task families
• Improving real-world deployment robustness and zero-shot transfer capabilities