Summary (Overview)

• First Large-Scale Tactile Pretraining: N⁰-VTLA is the first vision–tactile–language–action (VTLA) foundation model pretrained on tactile data at scale, using the NeoData multi-platform visuo-tactile corpus.
• Latent Tactile Tokens: Instead of reacting to current touch, the model predicts latent tactile tokens ( z ) that estimate the net tactile change over the upcoming action chunk, conditioning the action expert on anticipatory contact information.
• Three-Stage Training Recipe: A staged curriculum brings the tactile pathway online: Stage 1 grounds the predictor against future-tactile targets (92.3% top-1 retrieval), Stage 2 aligns latents with the action expert, and Stage 3 trains end-to-end.
• ALTER for Offline RL: An advantage-conditioned offline reinforcement learning method converts deployment data (demonstrations, drop events, human corrections) into binary advantage labels via a pairwise progress model, improving task-specific contact-rich skills.
• State-of-the-Art Results: N⁰-VTLA wins all nine NeoReal real-robot tasks (47.2% mean success vs. 29.4% for π₀.₅) and achieves 63.8% mean success on a 20-task simulation suite (vs. 44.0% for the strongest baseline). With ALTER, it reaches 75–95% success on three long-horizon tasks.

Introduction and Theoretical Foundation

Background and Motivation: Vision–language–action (VLA) models have made manipulation policies general by fine-tuning from pretrained vision–language backbones. However, touch has been largely absent from this progress, leaving a persistent weakness in contact-rich manipulation. Previous tactile extensions train on task-scale collections (tens of hours for a handful of skills), not at scale. The paper scopes to vision-based tactile sensing, where contact is read as an image.

Why Existing Approaches Fall Short:

• Concatenation into context: Treating tactile as another camera view in the vision–language prefix is ineffective because tactile frames are sparse, noisy, and informative only during brief contact windows.
• Injection into action pathway: Conditioning the action expert on the current tactile reading mischaracterizes touch—it records contact already made, saying little about contact the next actions must anticipate. The policy stays one step behind its own contact events.

N⁰-VTLA's Third Path: The paper proposes conditioning the action expert on a prediction of touch rather than the current reading. A small predictor reads the vision–language context together with current tactile tokens and emits latent tactile tokens ( z ) that estimate the net tactile change over the coming action chunk. This follows the joint-embedding predictive principle (I-JEPA, LeJEPA, LeWorldModel) for self-supervised representation learning.

Methodology

Model Architecture

Base Architecture (Section 2.1): The base policy pairs a PaliGemma vision–language backbone with a flow-matching action expert. Camera views, the instruction, and the robot state form the model prefix. The expert denoises an action chunk over a horizon of (H = 50) steps in a canonical 32-dimensional container. The flow-matching objective is unmasked over all 32 dimensions.

Latent Tactile Pathway (Section 2.2): Three stages bring the tactile pathway online:

• Stage 1 – Grounding the Predictor (Figure 2): With the base policy frozen, train only the predictor, tactile projection, and reconstruction head. The predictor output ( z ) is pulled toward the future-tactile target ( z^* ):

$$z^* = \frac{1}{n} \sum_{k=1}^n f_{\text{enc}}(\text{tac}^k_{\tau + H} - \text{tac}^k_{\tau}) \in \mathbb{R}^{10 \times d}, \quad H = 50$$

The loss combines a symmetric InfoNCE contrastive loss and an auxiliary L₁ reconstruction of a coarse future-tactile-difference field:

$$\mathcal{L}_1 = \mathcal{L}_{\text{NCE}} + \lambda_{\text{rec}} \mathcal{L}_{\text{rec}}$$

• Stage 2 – Aligning Latents with Action Expert (Figure 3): Hold the tactile perception stack frozen. Mask the vision–language prefix in the expert's attention, so action prediction must draw on ( z ). Train only the latent-to-expert projection and the action expert.

• Stage 3 – End-to-End Joint Training (Figure 4): Unfreeze everything except the frozen tactile encoder backbone. Train the full policy jointly under the action objective. The vision–language mask is removed.

Tactile Encoder: For view ( k ), subtract the episode-start baseline frame (\text{tac}^k_0) from the current frame (\text{tac}^k_\tau) in pixel space, and encode the difference with a frozen DINOv2 encoder followed by a trainable linear projection:

$$g^k = f_{\text{enc}}(\text{tac}^k_\tau - \text{tac}^k_0) \in \mathbb{R}^{10 \times d}$$

Each tactile image yields 10 tokens (1 class token + 9 spatial tokens from a 3×3 adaptive average pool).

Data (Section 3)

• NeoData: Large-scale curated multi-platform visuo-tactile corpus spanning single- and dual-arm manipulators and a UMI-style handheld gripper.
• Canonical Action/State Schema: A fixed 32-dimensional container: first 20 dimensions split into two 10-dimensional arm slots (3D position, 6D rot6d rotation, 1D gripper), remaining 12 unused (zero).
• Tactile Convention: Each episode begins with a 0.5s zero-contact baseline used as the reference frame.
• Data Quality: Verified for completeness, schema consistency, and statistics.

ALTER for Offline RL (Section 4.4)

ALTER stands for Advantage Labeling from Trajectory Events and Relative Progress:

• Duration-Calibrated Stage Progress: For each stage ( k ), compute mean duration ( \bar{d}_k ) across demonstrations. Stage weight and progress:

$$w_k = \frac{\bar{d}_k}{\sum_{j=1}^K \bar{d}_j}, \quad \phi_t = \sum_{j<k} w_j + w_k u_t$$

where ( u_t \in [0,1] ) is the fraction of current stage completed at frame ( t ).

• Pairwise Progress Model ( A_\theta(x_a, x_b) \in [-1, 1] ): A π₀.₅-based architecture encoding paired observations. Trained with:

$$\mathcal{L}_{\text{prog}} = \mathbb{E}_{\mathcal{D}_{\text{stage}}} [A_\theta(x_a, x_b) - (\phi(x_a) - \phi(x_b))]^2 + \lambda_{\text{event}} \mathbb{E}_{\mathcal{D}_{\text{event}}} [\max(0, m - y A_\theta(x_a, x_b))]^2$$

• Advantage Labeling: For each stored episode, compute global progress ( \hat{\phi}_t ) and local change ( \hat{r}_t ). Rank within each stage, assign top 30% as positive, append "Advantage: positive" or "Advantage: negative" to the task prompt.

Empirical Validation / Results

Real-World Results: NeoReal (Section 5.2)

Evaluated on nine contact-rich tasks. N⁰-VTLA beats the strongest baseline (π₀.₅) on every task:

Largest margin on Socket Plugging: N⁰-VTLA 85% vs. π₀.₅ 60%.

Simulation Results (Section 5.3)

UniVTAC Benchmark (8 tasks):

NeoSim Benchmark (12 tasks, 4 single-arm + 8 dual-arm):

Offline Policy Improvement with ALTER (Section 5.4)

Results on three real-robot tasks:

Representation Analyses (Section 5.5)

• Contrastive Retrieval: After Stage 1, predicted ( z ) retrieves its matching future-tactile target ( z^* ) at 92.3% top-1 accuracy (chance: 3.2%).
• Control: Ranking by current tactile encoding ( g ) alone reaches only 57% top-1.
• Perturbation Probe: Swapping tactile input moves ( z ) by ~0.9 in centered-cosine distance; swapping RGB and prompt moves it by ~0.2. Sensitivity ratio: 4.3 after Stage 1, 1.4 after joint training—latent stays tactile-leaning.

Theoretical and Practical Implications

Key Findings (Section 6):

1. Touch turns insertion into a closed-loop process: N⁰-VTLA detects blocked insertion attempts and recovers, whereas π₀.₅ commits to a single visually planned motion.
2. Touch enables fine gripper force control: Continuous aperture adjustment maintains contact without over-gripping (e.g., Bottle Standing: 30% vs. 0% for baselines).
3. Touch changes action at contact-critical moments: Counterfactual probes show the action path with touch diverges at firm contact and grasp, coincides in free space.
4. Stronger tactile pretraining persists under ALTER: The N⁰-VTLA backbone retains a consistent advantage over π₀.₅ under the same offline RL procedure.

Practical Significance:

• The three-stage recipe provides a principled method for integrating novel sensory modalities into pretrained VLA models without destabilization.
• ALTER demonstrates effective offline policy improvement from stored deployment data, converting sparse events (drops, human corrections) into actionable advantages.
• The cross-embodiment canonical action space enables multi-platform pretraining, making the approach applicable to diverse robot hardware.

Conclusion

N⁰-VTLA is a vision–tactile–language–action foundation model that predicts outcomes instead of reacting to them. Key contributions:

1. Large-scale tactile pretraining on NeoData, enabled by a canonical cross-embodiment action space.
2. Latent tactile tokens that condition the action expert on predicted future contact, never entering the vision–language prefix.
3. ALTER for advantage-conditioned offline RL from deployment data.

The model achieves state-of-the-art results across 20 simulation tasks and 9 real-robot tasks, with further improvements from offline policy learning.

Future Work:

• Explore the broader design space of predictive-latent framing beyond the specific predictor architecture.
• Extend ALTER to a wider range of tasks and manipulation settings.