Summary (Overview)

• RLDX-1 is a general-purpose Vision-Language-Action (VLA) model for dexterous manipulation, built on the Multi-Stream Action Transformer (MSAT) architecture that integrates heterogeneous modalities through modality-specific streams with cross-modal joint self-attention.
• The model extends beyond "versatile intelligence" (scene understanding, language-conditioned generalization) to include three additional functional capabilities: motion awareness, long-term memory, and physical sensing (tactile and torque).
• RLDX-1 combines architecture innovation with system-level design choices: a synthetic data pipeline with motion-consistency filtering, a three-stage training procedure (pre-training, mid-training, post-training), and inference optimizations (static graph conversion and custom fused kernels) achieving 1.63× speedup.
• On the ALLEX humanoid, RLDX-1 achieves 86.8% success rate vs. ~40% for π₀.₅ and GR00T N1.6, demonstrating superiority in high-DoF humanoid control under diverse functional demands.
• RLDX-1 consistently outperforms frontier VLAs (π₀.₅, GR00T N1.6) across simulation benchmarks (LIBERO, SIMPLER, RoboCasa Kitchen, GR-1 Tabletop, RoboCasa365) and real-world tasks.

Introduction and Theoretical Foundation

Background and Motivation

Learning generalist robot policies that achieve human-like dexterous manipulation in real-world environments remains a central goal in robotics. Existing VLAs (e.g., π₀, GR00T) primarily focus on versatile intelligence—the ability to understand diverse visual scenes, follow language instructions, generalize across tasks, and remain robust to perturbations. However, the authors argue that versatility alone is insufficient for many real-world tasks that demand:

1. Motion awareness: Operating in dynamic environments (e.g., conveyor-belt manipulation) where static visual observations fail to capture object trajectories or temporal dynamics.
2. Long-term memory: Sequential and long-horizon tasks requiring decisions grounded in prior interactions (e.g., shell games).
3. Physical sensing: Contact-rich tasks involving occlusions or subtle visual changes (e.g., grasping deformable objects, detecting incipient slip) where vision alone is insufficient.

Theoretical Foundation

RLDX-1 builds on:

• Vision-Language Models (VLMs): Uses Qwen3-VL 8B as the backbone for scene understanding and commonsense reasoning.
• Flow Matching: The action model learns a denoising velocity field over action trajectories using the flow-matching objective (Lipman et al., 2023).
• Multi-Modal Diffusion Transformer (MM-DiT): MSAT extends the double-then-single-stream architecture of MM-DiT to action modeling.
• Space-Time Self-Similarity (STSS): For capturing temporal dynamics in video features (Kwon et al., 2021).

Methodology

Neural Architecture

1. Vision-Language Model (VLM)

RLDX-1-VLM: Built on Qwen3-VL 8B, fine-tuned on robot-specific VQA data covering three aspects: (1) spatial relationships between end-effector and objects, (2) intermediate subtasks, (3) low-level actions.

Cognition Tokens: Learnable query tokens appended to the input sequence:

$$\mathbf{x} = [\mathbf{v}_t, \mathbf{l}_t, \mathbf{q}]$$

The output features corresponding to cognition tokens are retained as cognition features $\mathbf{h}_t$ (64 tokens used).

Motion Awareness:

• Vision encoder integrates an STSS module via residual connection: $\tilde{\mathbf{v}}_t^{(i)} = \mathbf{v}_t^{(i)} + S_\theta(\mathbf{S}_t)$, where $\mathbf{S}_t$ is the space-time self-similarity tensor.
• LLM backbone compresses past observation frames into a single context token via average pooling after the 4th layer (using DeepStack design of Qwen3-VL).

Long-Term Memory: Memory module maintains a queue of the most recent $n_{mem}$ cognition features:

$$\mathbf{Q}_t = [\mathbf{h}_{t-n_{mem}H}, \ldots, \mathbf{h}_{t-2H}, \mathbf{h}_{t-H}]$$

processed through a Transformer $\mathcal{M}_\theta$ with causal attention: $\mathbf{m}_t = \mathcal{M}_\theta([\mathbf{Q}_t, \mathbf{h}_t])$. Uses $n_{mem} = 3$.

2. Action Model (MSAT)

Multi-Stream Action Transformer (MSAT): Extends MM-DiT to action modeling with dedicated streams for cognition (C), action (A), and physics (P) modalities. Each stream applies its own normalization and QKV projections, concatenated along token dimension for joint self-attention.

Flow-Matching Objective:

$$\mathcal{L}(\theta; t, \tau, \epsilon) = \left\| \mathbf{u}_\theta(\mathbf{a}_{t:t+H}^{\tau}, \tau, \mathbf{c}_t) - (\mathbf{a}_{t:t+H} - \epsilon) \right\|_2^2 \tag{1}$$

Inference via Euler's method:

$$\mathbf{a}_{t:t+H}^{\tau_{i+1}} = \mathbf{a}_{t:t+H}^{\tau_i} + (\tau_{i+1} - \tau_i) \mathbf{u}_\theta(\mathbf{a}_{t:t+H}^{\tau_i}, \tau_i, \mathbf{c}_t) \tag{2}$$

Physical Sensing: P stream processes physical signals $\mathbf{p}_t$ with auxiliary objective predicting future physical signals $\mathbf{p}_{t+1:t+L}$ via flow matching.

Design Choices: RoPE on action stream, flow-matching timestep τ as in-context token, RMSNorm, SwiGLU activation.

Training Data

Synthetic Data Pipeline: Uses image-to-video (I2V) models with task augmentation (factorized instruction composition, skill-primitive-conditioned variation) and scene augmentation (I2I editing with FLUX.2-dev, V2V transfer with Cosmos-Transfer). Two-stage filtering: video quality filtering (VLM-based) and motion-consistency filtering (replay predicted actions in simulator, compare via learned attentive probe on V-JEPA2).

Training Procedure (Three Stages)

1. Pre-Training: 100K steps, batch size 8192, learning rate 1×10⁻⁴, 64 H200 GPUs, ~195 hours. Freezes VLM backbone except top 4 layers. Uses 4-frame video observations at offsets {−6, −4, −2, 0}.

2. Mid-Training: 25K steps, batch size 1024, learning rate 5×10⁻⁵. ALLEX (in-house + 72K synthetic, 5:5 ratio) and FR3 (DROID + in-house, 8:2 ratio). Adds memory module, STSS, physical sensing. Dropout 0.3 on new modality inputs, 2K-step alignment warmup.

3. Post-Training: Task-specific fine-tuning with adaptive data collection (base + refinement stages) and optional RECAP-style reinforcement learning with a novel text-based VLM critic that predicts values autoregressively using native number tokens.

Inference Optimization

• Graph Capture: Static graph conversion eliminates kernel launch overhead by precomputing rotary position embeddings and attention masks, capturing the forward pass as a single CUDA Graph.
• Kernel Optimization: Hand-designed fused kernels for short-prefill workload, fusing RMSNorm, RoPE, and attention to minimize global memory traffic.

Empirical Validation / Results

Simulation Benchmarks

Table 1(a): Classical Simulation Benchmarks (Success Rates %)

Table 1(b): Challenging Simulation Benchmarks (Success Rates %)

Real-World Experiments

OpenArm Humanoid (Versatile Intelligence): RLDX-1 outperforms baselines on all tasks:

• Basic Pick-and-Place: RLDX-1 achieves higher success than π₀.₅ and GR00T N1.6
• Unseen Object: RLDX-1 54.2% vs. π₀.₅ 37.5%
• Unseen Task: RLDX-1 54.2% vs. π₀.₅ 45.8%

ALLEX Humanoid (Functional Capabilities):

• Conveyor-belt catching: RLDX-1 >87.5% vs. π₀.₅ <29.2%
• Object-in-Box Selection (long-term memory): RLDX-1 91.7% vs. GR00T N1.6 and π₀.₅ ~30%
• Overall ALLEX tasks: RLDX-1 86.8% vs. ~40% for baselines

Inference Speedup

Per-step latency on NVIDIA RTX 5090:

• PyTorch Eager: 71.2 ms
• RLDX-1 optimized: 43.7 ms (1.63× speedup)

Theoretical and Practical Implications

Theoretical Contributions

1. MSAT Architecture: The Multi-Stream Action Transformer provides a principled framework for integrating heterogeneous modalities (visual-linguistic cognition, proprioception, physical signals) while preserving modality-specific representations. The extensible design (adding P stream) demonstrates a scalable approach to incorporating new sensing modalities.

2. Functional Capability Framework: The paper formalizes three functional capabilities beyond versatile intelligence (motion awareness, long-term memory, physical sensing) and provides architectural modules for each, establishing a taxonomy for VLA design.

3. Text-Based VLM Critic: The novel critic design reuses the VLM's native text-prediction interface for value prediction, avoiding distributional mismatch from new prediction heads and enabling reliable value estimation from limited data.

Practical Implications

1. Synthetic Data Pipeline: The motion-consistency filtering approach (replaying predicted actions in simulator and comparing against generated video) addresses the critical challenge of action-label quality in synthetic data, improving success rate by 9.1% on GR-1 Tabletop.

2. Real-Time Deployment: The inference optimization pipeline (static graph + custom kernels) demonstrates that VLA inference can be made practical for real-time control, with the 1.63× speedup being critical for dynamic environments.

3. Humanoid Control: The strong performance on ALLEX humanoid (86.8%) suggests that VLAs can effectively control high-DoF systems when trained with appropriate data and architecture.

Conclusion

RLDX-1 represents a significant step toward reliable VLAs for complex, contact-rich, and dynamic real-world dexterous manipulation. The key takeaways are:

1. Versatile intelligence alone is insufficient for real-world dexterous manipulation; functional capabilities (motion awareness, long-term memory, physical sensing) are essential.

2. MSAT provides a unified framework for integrating these capabilities through modality-specific streams with cross-modal joint self-attention.

3. System-level design matters: The combination of synthetic data generation with motion-consistency filtering, three-stage training, and inference optimization enables practical deployment.

4. Superior empirical performance: RLDX-1 consistently outperforms frontier VLAs (π₀.₅, GR00T N1.6) across simulation and real-world benchmarks, with the largest gains on tasks requiring functional capabilities.

Future Directions

The paper suggests several promising directions:

• Extending RLDX-1 to additional embodiments and sensing modalities
• Scaling up synthetic data generation for rarer manipulation scenarios
• Further improving the RL-based post-training for more challenging dexterous tasks
• Applying the architecture to other domains requiring multimodal integration (e.g., mobile manipulation, human-robot interaction)