Summary (Overview)

• RAD-2 proposes a unified generator-discriminator framework for closed-loop autonomous driving planning, where a diffusion-based generator produces diverse trajectory candidates and an RL-optimized discriminator reranks them based on long-term driving quality.
• The framework introduces Temporally Consistent Group Relative Policy Optimization (TC-GRPO), which exploits temporal coherence to alleviate the credit assignment problem in high-dimensional continuous action spaces.
• On-policy Generator Optimization (OGO) converts closed-loop feedback into structured longitudinal optimization signals, progressively shifting the generator toward high-reward trajectory manifolds.
• BEV-Warp, a high-throughput simulation environment, performs closed-loop evaluation directly in Bird's-Eye View feature space via spatial warping, bypassing expensive image-level rendering.
• RAD-2 reduces the collision rate by 56% compared to strong diffusion-based planners, and demonstrates improved perceived safety and driving smoothness in real-world deployment.

Introduction and Theoretical Foundation

Background and Motivation

High-level autonomous driving requires motion planners capable of modeling multimodal future uncertainties while remaining robust in closed-loop interactions. The paper identifies key limitations in existing approaches:

• Regression-based planners predict trajectories deterministically, collapsing multimodal behaviors and producing mean-biased outputs.
• Selection-based planners rely on discrete candidate sets, limiting their ability to represent the full range of feasible trajectories.
• Diffusion-based imitation learning (IL) planners model multimodal continuous trajectories but suffer from:
  • Stochastic instabilities — real driving datasets contain noise and uneven distributions, causing the diffusion model to learn certain regions of the trajectory distribution less effectively.
  • Lack of corrective negative feedback — imitation-only training provides no negative feedback to suppress unrealistic behaviors.
  • Causal confusion — IL learns correlations between states and actions instead of underlying causal factors.
  • Open-loop training paradigm — causes a mismatch with the closed-loop nature of real-world driving.

Theoretical Foundation

The framework aligns with the probabilistic inference framework for optimal control. Let $\tau$ denote a future trajectory. The diffusion generator $\mathcal{G}_\theta(\tau | o)$, parameterized by $\theta$, models a broad distribution of feasible actions conditioned on context $o$. The RL-trained discriminator $\mathcal{D}_\phi(\tau | o, \mathcal{C})$, parameterized by $\phi$, provides a reranking distribution over candidate trajectories $\mathcal{C} = \{\tau_1, ..., \tau_M\}$ sampled from the generator. The joint policy distribution is:

$$\Pi_{\theta, \phi}(\tau | o) = \mathbb{E}_{\mathcal{C} \sim \mathcal{G}_\theta(\cdot | o)}[\mathcal{D}_\phi(\tau | o, \mathcal{C})]$$

The global objective is to minimize the KL-divergence between the hybrid policy and the ideal risk-neutral, high-efficiency distribution $\Pi^*$:

$$\min_{\theta, \phi} \mathbb{D}_{\mathrm{KL}}\big(\Pi_{\theta, \phi}(\tau | o) \| \Pi^{*}(\tau | o)\big)$$

Methodology

3.1 Generator-Discriminator Framework

Diffusion-based Generator

The generator models a multimodal distribution over future trajectories conditioned on current observation $o_t$. Scene encoding extracts BEV features $T_b$, static map elements $\mathcal{X}_{\mathrm{map}}$, dynamic agents $\mathcal{X}_{\mathrm{agent}}$, and navigation inputs $\mathcal{X}_{\mathrm{nav}}$:

$$T_m = \mathrm{E}_m(\mathcal{X}_{\text{map}}), \quad T_a = \mathrm{E}_a(\mathcal{X}_{\text{agent}}), \quad T_n = \mathrm{E}_n(\mathcal{X}_{\text{nav}})$$

These are fused into a unified scene embedding:

$$E_{\text{scene}} = \mathcal{F}(T_b, T_m, T_a, T_n)$$

For M independent modes, initial noise trajectories $\tau^{(0,m)} \sim \mathcal{N}(0, I)$ are iteratively denoised for K steps:

$$\tau^{(k,m)} = G\big(\tau^{(k-1,m)}, E_{\text{scene}}, k\big), \quad k = 1, \dots, K$$

The final candidate set is $\widehat{\mathcal{T}} = \{\hat{\tau}_{t:t+H}^m\}_{m=1}^M$, where each $\hat{\tau}_{t:t+H}^m$ is a continuous $(x, y)$ trajectory over the planning horizon $H$.

RL-based Discriminator

The discriminator evaluates candidate trajectories via:

1. Trajectory Encoding — each point embedded via shared MLP, processed by a Transformer encoder with a [CLS] token.
2. Scene Conditioning — static and dynamic encoders share architecture with generator but maintain independent parameters.
3. Trajectory–Scene Interaction — cross-attention $\Psi(Q, KV)$ aggregates multi-source scene context:

$$O_{\mathrm{m}} = \Psi(Q_{\tau}, T_m^{*}), \quad O_{\mathrm{b}} = \Psi(Q_{\tau}, T_b), \quad O_{\mathrm{a}} = \Psi(Q_{\tau}, T_a^{*})$$ $$T_{a \cap m} = \Psi(T_a^{*}, T_m^{*}), \quad O_{\mathrm{a} \cap \mathrm{m}} = \Psi(Q_{\tau}, T_{a \cap m})$$

4. Trajectory Scoring — sigmoid activation produces a scalar score:

$$s(\hat{\tau}_{t:t+H}) = \sigma(E_{\mathrm{fusion}}) \in [0, 1]$$

3.2 Closed-Loop Simulation Environment

BEV-Warp Environment

BEV-Warp constructs a simulation environment by directly manipulating BEV features over time, bypassing image-level rendering. For each simulation step, a warp matrix is derived from relative pose deviation:

$$\mathbf{M}_{t+1} = (\mathcal{P}_{t+1})^{-1}\mathcal{P}_{t+1}^{\mathrm{ref}} \in \mathbb{R}^{3 \times 3}$$

The synthesized feature for the next timestep is obtained via bilinear interpolation:

$$\mathcal{B}_{t+1} = \mathcal{W}\left(\mathcal{B}_{t+1}^{\mathrm{ref}}, \mathbf{M}_{t+1}\right)$$

3DGS Environment

For validation, the policy is also tested in a photorealistic 3D Gaussian Splatting environment where $o_t = \mathcal{R}(G, \mathcal{P}_t)$ renders raw multi-view observations.

Trajectory-Based Controller

An iLQR-based controller tracks the planned trajectory by minimizing a quadratic cost:

$$\mathbf{u}_{t:t+H}^{*} = \arg \min_{\mathbf{u}} \sum_{k=t}^{t+H} \left(\|\mathbf{x}_k - \hat{\mathbf{x}}_k^{\mathrm{ref}}\|_Q^2 + \|\mathbf{u}_k\|_R^2\right)$$

3.3 Joint Policy Optimization

Temporally Consistent Rollout

A trajectory reuse mechanism maintains short-term behavioral consistency. Once an optimal trajectory $\hat{\tau}_t^{*}$ is selected, it is converted into a control sequence and reused over a fixed execution horizon $H_{\mathrm{reuse}} < H$, preventing high-frequency mode-switching that disrupts credit assignment.

Discriminator Optimization via TC-GRPO

Reward Modeling defines two complementary rewards:

(i) Safety-Criticality Reward — Time-To-Collision is defined as:

$$T_t = \inf\left\{k \in [0, T_{\max}] \mid \mathcal{B}_{\text{ego}}(k; t) \cap \mathcal{V}_{\text{env}}(t + k) \neq \emptyset\right\}$$

The sequence-level reward is the worst-case temporal margin:

$$r_{\text{coll}} = \min_{1 \leq t \leq L}\left(T_t - T_{\max}\right)$$

(ii) Navigational Efficiency Reward — based on Ego Progress $\rho$:

$$r_{\mathrm{eff}} = \min(\rho - \rho_{\mathrm{low}}, 0) + \min(\rho_{\mathrm{high}} - \rho, 0) + 1$$

where $\rho_{\mathrm{low}} = 1.05$ and $\rho_{\mathrm{high}} = 1.10$.

Optimization Objective — For a rollout $\mathcal{O}_i$ with reward $r_i$, the standardized advantage is:

$$A_i = \frac{r_i - \operatorname{mean}(\{r_1, \dots, r_G\})}{\operatorname{std}(\{r_1, \dots, r_G\})}$$

The clipped objective at sparse decision points is:

$$\mathcal{L}_{i, t \in \mathcal{K}_i} = \min\left(\rho_{i,t} A_i, \operatorname{clip}(\rho_{i,t}, 1 - \varepsilon, 1 + \varepsilon) A_i\right)$$

An adaptive entropy regularization mechanism with temperature-based control is introduced:

$$\beta = \exp(\lambda) \cdot \mathbb{1}_{[\bar{\mathcal{H}} < \bar{\mathcal{H}}_{\mathrm{target}}]}$$

The full RL objective is:

$$\mathcal{J}_{\mathrm{RL}}(\phi) = \mathbb{E}\left[\frac{1}{\sum_{i=1}^{G}|\mathcal{K}_i|}\sum_{i=1}^{G}\sum_{t=1}^{|\mathcal{K}_i|}\left(\mathcal{L}_{i,t} + \beta \mathcal{H}_{i,t}\right)\right]$$

On-policy Generator Optimization

Closed-loop feedback is converted into structured longitudinal optimization:

• Safety-driven Deceleration — when $T_t < \gamma$, reduce travel distance by ratio $\rho \in (0, 1)$
• Efficiency-driven Acceleration — when ego lags behind reference and no collision risk, increase travel distance by ratio $\rho' > 1$

The generator is fine-tuned via mean squared error loss:

$$\mathcal{L}_{\mathrm{op}}(\theta) = \mathbb{E}_{\tau^{\mathrm{opt}} \sim D_{\mathrm{opt}}}\left[\sum_{k=0}^{H}\left\|\hat{\tau}_{t+k} - \tau_{t+k}^{\mathrm{opt}}\right\|_2^2\right]$$

Empirical Validation / Results

Closed-loop Performance in BEV-Warp Environment

Table 1: Closed-loop performance comparison in the BEV-Warp simulation environment.

Closed-loop Evaluation in 3DGS Environment

Table 2: Evaluation on photorealistic 3DGS benchmark.

Open-loop Trajectory Evaluation

Table 3: Open-loop benchmark of Senna-2.

Ablation Studies

Table 4: Ablation study of the training pipeline.

Key ablation findings:

• Execution horizon ($H_{\text{reuse}} = 8$): Optimal balance between stable credit assignment and reactive flexibility
• Clip filtering: Discarding low-variance clips improves EP@1.0 from 0.662 to 0.728 without compromising safety
• Discriminator initialization: Initializing from planning head achieves CR of 0.337 vs. 0.426 for random initialization
• Group size of 4 in TC-GRPO gives best safety-efficiency balance
• Entropy regularization prevents trajectory score collapse, improving CR from 0.254 to 0.234

Inference-time Scaling

Table 10: Inference-time scaling with candidate count M.

Increasing the candidate pool at inference time consistently improves planning performance without retraining.

Theoretical and Practical Implications

Theoretical Contributions

1. Decoupled optimization: By restricting RL to the low-dimensional discriminator output space rather than the high-dimensional trajectory space, RAD-2 avoids the instability of directly applying sparse scalar rewards to high-dimensional trajectories, improving optimization stability.

2. Temporal consistency as a physical prior: TC-GRPO leverages temporal coherence to "denoise" advantage signals and stabilize policy gradients, addressing the severe credit assignment problem in continuous driving spaces where sparse scalar rewards fail to distinguish which specific variations within a sampled group contribute to superior outcomes.

3. Self-improving closed loop: The generator and discriminator jointly optimize the overall policy through iterative optimization, progressively shifting the distribution toward safer behaviors without additional expert supervision.

Practical Implications

1. Simulation efficiency: BEV-Warp enables high-throughput, feature-level closed-loop training by exploiting spatial equivariance, overcoming limitations of game-engine simulators (simplified agent behavior), reconstruction-based simulators (complexity/cost), and learned world models (long-horizon drift).

2. Real-world deployment: RAD-2 demonstrates improved perceived safety and driving smoothness in complex urban traffic, validating the framework's practical applicability.

3. Inference-time scaling: The architecture inherently supports inference-time scaling — increasing sample count M allows the discriminator to explore a denser action space and identify higher-quality solutions without retraining.

Conclusion

RAD-2 presents a unified generator–discriminator framework for stable reinforcement learning in diffusion-based motion planning. The key innovations include:

• TC-GRPO for improved credit assignment through temporally coherent sampling
• On-policy Generator Optimization for progressive trajectory distribution refinement
• BEV-Warp for scalable closed-loop learning via feature-level simulation

Extensive experiments demonstrate consistent improvements in both safety and efficiency across diverse benchmarks, with collision rate reductions of over 56% compared to strong diffusion-based planners.

Limitations and Future Work

1. Representation Specificity: BEV-Warp's efficiency relies on BEV feature manipulation, limiting applicability to architectures using raw camera pixels or unified latent embeddings without spatial-equivariant grid structures.

2. Transition to Generative World Models: Future work will focus on integrating the optimization pipeline with Generative World Models (WM), optimizing inference efficiency and temporal consistency of latent-based world models to further scale training scenario diversity and bridge the remaining fidelity gap between simulation and real-world driving.