Summary (Overview)

• LaMem-VLA introduces a dual latent memory framework that treats historical robotic experience as context-native latent tokens within the VLA embedding space, rather than as external policy-side context.
• The framework comprises four coordinated modules: a curator that organizes short-term visual and long-term semantic memory vaults, a seeker that retrieves task-relevant evidence, a condenser that compresses retrieved evidence into fixed-length latent tokens, and a weaver that interleaves these tokens with current observation, instruction, and action queries.
• On LIBERO, LaMem-VLA achieves an average success rate of 97.6%, outperforming MemoryVLA (96.5%) and CogACT (93.2%).
• On SimplerEnv-Bridge, it reaches 73.9% average success, surpassing π₀ (69.2%) and CogACT (57.3%).
• Ablations confirm that both short-term and long-term latent memory are complementary, and that latent-native integration (embedding memory into the reasoning sequence) is superior to policy-side external conditioning.

Introduction and Theoretical Foundation

Background and Motivation:
Most existing Vision-Language-Action (VLA) models [2, 1, 15] adopt a Markovian assumption – they predict actions solely from the current observation (o_t) and instruction (I), ignoring temporal dependencies. This creates a temporal short-horizon bias: the model can react to the currently visible state but cannot reason about previous state transitions, completed subtasks, or the current phase of a multi-step task. This limitation becomes critical in long-horizon manipulation.

Previous Approaches and Their Limitations:
Existing memory augmentation for VLA models falls into three categories:

1. Temporal-context expansion (e.g., concatenating historical frames [18, 19], input as video [20, 21, 22]): grows computational cost with context length and has a fixed memory ceiling.
2. Sparse history abstraction (e.g., recurrent latent states [51, 52], action summaries [54]): can lose fine-grained perceptual and semantic details.
3. External memory conditioning (e.g., storing history in a memory bank and retrieving it [23, 24, 25]): memory is stored outside the model’s native token space and consumed as auxiliary policy-side context after multimodal reasoning, preventing fluid interleaving with perception and action formation.

Key Insight:
The paper argues that historical experience should be represented as context-native robotic memory – stored, retrieved, and consumed in the same continuous embedding space where the VLA model already perceives, reasons, and acts. This latent embedding space is a natural medium because modern VLA models already integrate visual observations and language instructions in a continuous token embedding space [33, 2, 1, 15].

Dual Memory Types:
Long-horizon manipulation requires two complementary forms:

• Short-term memory: visually dominant, preserving recent perceptual evidence (object locations, state changes).
• Long-term memory: semantically dominant, preserving task progress, contextual semantics, and action continuity across longer horizons.

Both are ultimately reconstructed as latent memory tokens that can be consumed by the VLA model in the latent space.

Methodology

Problem Formulation

The VLA policy (\Pi_\theta) at timestep (t) takes instruction (I) and current observation (o_t) to predict a chunk of future actions:

$$a_{t:t+H-1} = \Pi_\theta(o_t, I),$$

where (H) is the action horizon, and each (a_t \in \mathbb{R}^7) (3-DoF translation, 3-DoF rotation, 1-DoF gripper).

LaMem-VLA Architecture

The model uses a Prismatic VLM backbone (7B) + a diffusion action expert. At each timestep (t), the vision–language backbone produces visual tokens (X_t) and instruction tokens (I). Learnable action queries (Q_{\text{action}}) are appended.

The four coordinated modules close the loop between latent memory reconstruction and action reasoning:

1. Latent Memory Curator (§3.3)

Organizes historical experience into two vaults:

• Short-term memory vault (M_{\text{short}}): stores visual tokens from the current episode. Each unit (m_s^i = (k_s, v_s)) uses a compression module (C_s) to distill (X_t) into (v_s \in \mathbb{R}^{N_s \times C}) and (k_s = \text{MeanPool}(v_s) \in \mathbb{R}^C).
• Long-term memory vault (M_{\text{long}}): stores action hidden states (H_{\text{action}}) directly.

Updating strategy: when vault size exceeds capacity (L), the most redundant temporally adjacent pair (highest cosine similarity) is consolidated by averaging keys and values (Eq. 4–5).

2. Latent Memory Seeker (§3.4)

Builds a context-aware query (Q_t) from the current multimodal cognition state (concatenated visual and instruction tokens) using a lightweight transformer-based query builder (B):

$$Q_t = B([Q_{\text{con}}^t; Q_{\text{init}}])[-K_q:] \in \mathbb{R}^{K_q \times C}.$$

The mean-pooled query (q_t = \text{MeanPool}(Q_t)) retrieves the top-(K) units from each vault by cosine similarity:

$$\mathcal{I}_s = \text{Top-}K\left( \{\cos(q_t, k_s^i)\}_{i=1}^{|M_{\text{short}}|} \right), \quad Z_{\text{short}} = \text{Concat}_{i \in \mathcal{I}_s}(v_s^i) \in \mathbb{R}^{KN_s \times C}.$$

Same for long-term, producing (Z_{\text{long}} \in \mathbb{R}^{KN_l \times C}).

3. Latent Memory Condenser (§3.4)

Compresses retrieved evidence into fixed-length latent memory tokens using learnable slots (T_s \in \mathbb{R}^{L_s \times C}) and (T_l \in \mathbb{R}^{L_l \times C}), updated with lightweight memory formers (F_v) and (F_c):

$$M_{\text{short}} = F_v([Q_t; Z_{\text{short}}; T_s])[-L_s:], \quad M_{\text{long}} = F_c([Q_t; Z_{\text{long}}; T_l])[-L_l:].$$

The resulting tokens are query-conditioned and in the same embedding space as the VLA reasoning.

4. Latent Memory Weaver (§3.5)

Constructs the memory-augmented input sequence:

$$S_t = [M_{\text{short}} + \mathbf{1}_{L_s} \mathbf{b}_s^\top; \; M_{\text{long}} + \mathbf{1}_{L_l} \mathbf{b}_l^\top; \; X_t; \; I; \; Q_{\text{action}}],$$

where (\mathbf{b}_s, \mathbf{b}l \in \mathbb{R}^C) are learnable source embeddings. The sequence is fed to the VLM backbone, which outputs memory-grounded action tokens (Z{\text{action}}).

Diffusion-based Action Expert

The action expert (≈300M parameters) uses conditional denoising with MSE loss:

$$\mathcal{L}_{\text{action}} = \mathbb{E}_{n,\epsilon} \left[ \|\epsilon - \epsilon_\theta(a_{t:t+H-1}^n, n, Z_{\text{action}})\|_2^2 \right].$$

Inference uses DDIM [63] with 10 denoising steps.

Empirical Validation / Results

Simulated Evaluation on SimplerEnv-Bridge (§4.2)

Setup: Bridge v2 dataset, 50k optimization steps, 24 trials per task.
Results (Table 1):

LaMem-VLA achieves the highest average (73.9%), with a 16.6 point gain over CogACT and 4.7 over π₀.

Simulated Evaluation on LIBERO (§4.3)

Setup: Franka robot, five suites (Spatial, Object, Goal, Long-10, Long-90). 50 demos per task, separate models for first three, joint for long-horizon.
Results (Table 2):

LaMem-VLA outperforms all methods, with the largest gains on the long-horizon suites (Long-10: +2.4 over MemoryVLA; Long-90: +1.4).

Ablation Studies (§4.4)

Dual-scale latent memory (Table 3)

Both memory streams are complementary; removing either degrades performance.

Latent-native vs. policy-side conditioning (Table 4)

Latent-native integration yields the best results.

Number of retrieved units (K) (Table 5)

(K=8) gives best performance; too few or too many hurt.

Number of latent memory tokens (Fig. 3)

Varying (L_s) (short-term tokens) and (L_l) (long-term). Optimal balance: ((L_s, L_l) = (8, 4)) – used as default.

Theoretical and Practical Implications

Theoretical Contribution:
LaMem-VLA introduces a new paradigm for robotic memory in VLA models: historical experience as context-native latent memory. This bridges the gap between memory storage and the internal reasoning process, allowing memory to directly influence action formation rather than being an external post-hoc conditioner. The dual vault design (visual short-term, semantic long-term) is theoretically motivated by the complementary temporal and functional requirements of long-horizon manipulation.

Practical Impact:
The architecture achieves state-of-the-art results on two benchmarks (LIBERO: 97.6%, SimplerEnv: 73.9%) with only a single RGB camera and language input, without proprioception or wrist cameras. The fixed-length latent memory tokens (8 + 4) keep computational overhead low while enabling temporally aware reasoning. The framework is compatible with pretrained VLA backbones and diffusion-based action heads, making it a practical augmentation for existing models.

Limitation: Current validation is in simulation; real-world extension is ongoing.

Conclusion

LaMem-VLA demonstrates that context-native latent memory effectively reduces temporal short-horizon bias in VLA models. By organizing historical experience into short-term visual and long-term semantic vaults, retrieving relevant evidence via a context-aware query, condensing it into compact latent tokens, and weaving these tokens directly into the VLA reasoning sequence, the model achieves strong performance on long-horizon and temporally dependent manipulation tasks.

Future directions include real-world deployment, scaling to longer trajectories, and exploring more sophisticated memory condensation techniques. The paper provides open-source release of the project page and code.