Summary (Overview)

• Qwen-Drive-1.0 is presented as an initial step toward a vision-language foundation model for autonomous driving, unifying 3D perception, visual question answering (VQA), and motion planning within a single pretrained VLM (Qwen3.5-4B).
• The framework introduces two external modules: a BEV perception head for explicit 3D detection, semantic occupancy, and map segmentation, and a Planning Expert that generates future ego trajectories via flow matching.
• A staged training recipe combines driving-specific supervision (perception, planning, reasoning) with general-purpose vision-language data, preserving broad VLM competence while acquiring driving capabilities.
• Results show competitive performance across 3D perception (e.g., 43.95 mAP on nuScenes), driving VQA, and motion planning (e.g., 90.7 PDMS on NAVSIM, 7.91 RFS on WOD-E2E test split), with minimal degradation on general VQA benchmarks.
• The work demonstrates that explicit 3D perception and trajectory generation can be added to a pretrained VLM without architectural changes, supporting both driving and cockpit applications on a single platform.

Introduction and Theoretical Foundation

The paper addresses a shift in autonomous driving from task-specific modular pipelines toward unified learning-based systems. Vision-language-action (VLA) models leverage pretrained VLMs to connect scene understanding, reasoning, and action generation. However, prior driving VLA methods often rely solely on textual VQA supervision, which has two key limitations:

1. Lack of explicit 3D constraints: Textual supervision does not directly constrain 3D layout, depth, or occupancy, allowing fluent scene descriptions without precise spatial understanding.
2. Catastrophic forgetting: Extensive domain adaptation on driving data can degrade general knowledge from pretraining, which is critical for rare and out-of-distribution (OOD) scenarios.

The paper also motivates cockpit-driving integration, where a single onboard model must serve both driving functions and cockpit applications (e.g., dialogue, open-ended visual queries). This requires preserving general vision-language capability alongside driving competence.

The proposed solution satisfies three design requirements:

• Keep the pretrained VLM architecture unchanged.
• Add an explicit perception probe to expose and evaluate 3D information.
• Retain general-purpose capabilities during driving adaptation.

Methodology

Model Architecture

The unified architecture (Fig. 2) uses:

• A shared vision encoder converting images to visual tokens.
• A VLM encoding tokens with textual prompts and generating responses autoregressively.
• Two external modules leveraging shared features without architectural changes.

Multi-view and multi-frame inputs: View tags (e.g., <FRONT VIEW>) and frame tags (e.g., frame: k) provide spatial and temporal context. Input serialization varies by task:

• Question answering uses frame-major order (all views at one timestep).
• Planning uses view-major order to expose temporal variation per view.

Autoregressive text generation uses standard next-token prediction:

$$\mathcal{L}_{\mathrm{ntp}} = -\sum_{t=1}^{T} \log p(y_t \mid \mathbf{x}, y_{<t}) \tag{1}$$

BEV Perception Head

The head performs single-frame surround-view 3D perception, fusing two feature streams:

• Vision encoder features $F_i^v$: low-level appearance.
• VLM output features $F_i^m$: broader scene context.

A depth-based view transform lifts features into a 3D volume:

$$\mathbf{V}(\mathbf{p}) = \sum_{i \in \Omega(\mathbf{p})} \mathbf{D}_i(u_i, v_i, d_i) \mathbf{F}_i^v(u_i, v_i) \tag{2}$$

A query-based BEV transformer aggregates features onto the BEV plane, which serves three task branches:

• 3D detection (DETR-style decoder).
• Semantic occupancy (3D UNet with height fusion).
• Map segmentation (UNet-style head).

The perception objective combines detection, occupancy, and map losses:

$$\mathcal{L}_{\mathrm{perc}} = \mathcal{L}_{\mathrm{det}} + \mathcal{L}_{\mathrm{occ}} + \mathcal{L}_{\mathrm{map}} \tag{3}$$

Planning Expert

Trajectory prediction is formulated as conditional generation:

$$\boldsymbol{\tau} \sim p(\boldsymbol{\tau} \mid \mathbf{s}, \ell, \boldsymbol{\tau}_{\mathrm{hist}}, \mathbf{n}, \mathbf{e}, \mathbf{r}), \quad \boldsymbol{\tau} = \{(x_k, y_k, \theta_k)\}_{k=1}^{50} \tag{6}$$

The Planning Expert conditions noisy trajectory tokens on cached VLM keys/values and uses flow matching to recover clean trajectories, enabling joint training across multiple datasets.

Staged Training

1. Stage 1: Continued pretraining on general vision-language data.
2. Stage 2: Joint training on driving VQA, 3D perception, and general VQA data.
3. Stage 3: Planning Expert training on unified trajectory annotations.
4. Reinforcement learning: Preference-based optimization with source-specific rewards (PDMS, RFS, ADE) and a shared ADE term.

Empirical Validation / Results

3D Perception

• nuScenes: 43.95 mAP (detection), 60.99 mIoU (occupancy), 71.27 mIoU (map), competitive with specialized detectors.
• OpenScene: 43.45 mAP, demonstrating cross-dataset generalization.
• Unseen camera rigs: Qualitative outputs on WOD-E2E (8-camera) and PAI-AV (6-camera) show plausible predictions without adaptation.

Driving VQA

• DriveVQA-Bench: 63.78 overall score, surpassing all compared methods including larger models.
• CoC-Bench: 41.26 overall, outperforming general-purpose VLMs.

General VQA Preservation

• Knowledge/reasoning benchmarks: 66.41 average vs. 67.40 for Qwen3.5-4B (within 1 point).
• Spatial understanding: 53.96 average, exceeding Qwen3.5-4B (52.99), with best scores on ERQA and ODinW13.

Motion Planning

Open-loop (WOD-E2E test split):

• Qwen-Drive-1.0-SFT w/ reasoning: 7.78 RFS (vs. 7.77 for MindVLA-U1).
• Qwen-Drive-1.0-RL: 7.91 RFS, exceeding reinforced MindVLA-U1 by 0.04.

Open-loop (PAI-AV leakage-free subset):

• Qwen-Drive-1.0-SFT w/ reasoning: 0.42 m (3s Avg. ADE), comparable to Alpamayo-1.5 (0.36 m).

Pseudo-closed-loop (NAVSIM):

• Qwen-Drive-1.0-SFT w/ reasoning: 88.2 PDMS (best among imitation-only methods).
• Qwen-Drive-1.0-RL: 90.7 PDMS (best overall with RL).

Closed-loop (AlpaSim):

• Qwen-Drive-1.0-RL: 11.0% at-fault close encounter rate, 12.0% off-road rate (halved from SFT), 48.0% progress.

Ablations

• Stage 2 mixture: Vision-language training improves Driving QA (+6.55) and CoC (+38.39), while 3D perception supervision preserves all aggregates within 1 point.
• RL rewards: Shared ADE term reduces 5s ADE from 2.24m to 1.27m on WOD-E2E with modest RFS trade-off.
• Data scale: 5s Avg. ADE decreases monotonically from 1.34m to 1.05m as PAI-AV samples increase from 0.17M to 1.38M.

Conclusion

Qwen-Drive-1.0 demonstrates that explicit 3D perception and trajectory generation can be added to a pretrained VLM while retaining broad vision-language competence. The staged training recipe supports domain adaptation without catastrophic forgetting, and the unified framework enables competitive results across perception, reasoning, and planning evaluations. The work supports cockpit-driving integration by allowing a single model to serve both domains.

Limitations and Future Work

1. Causal reasoning instability: The model struggles to identify governing causes when multiple causes act at different time scales (e.g., a red light 20m ahead vs. a child 5m ahead).
2. Rationale-trajectory inconsistency: Generated trajectories do not always adhere to the textual rationale, suggesting gains may partly stem from model-internal information in self-generated traces.
3. Cross-task transfer: Different input formats, temporal contexts, and resolutions across tasks may limit representation transfer.

Future directions include multi-timescale causal modeling, explicit consistency supervision between rationale and trajectory, and better alignment of task configurations for closer joint optimization.