# Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

> LaMem-VLA achieves state-of-the-art 97.6% on LIBERO and 73.9% on Sim

- **Source:** [arXiv](https://arxiv.org/abs/2607.07608)
- **Published:** 2026-07-10
- **Permalink:** https://picx.dev/p/fPG4Zi
- **Whiteboard:** https://picx.dev/p/fPG4Zi/image

## Summary

## Summary (Overview)

- LaMem-VLA introduces a **dual latent memory framework** that treats historical robotic experience as **context-native latent tokens** within the VLA embedding space, rather than as external policy-side context.
- The framework comprises four coordinated modules: a **curator** that organizes short-term visual and long-term semantic memory vaults, a **seeker** that retrieves task-relevant evidence, a **condenser** that compresses retrieved evidence into fixed-length latent tokens, and a **weaver** that interleaves these tokens with current observation, instruction, and action queries.
- On **LIBERO**, LaMem-VLA achieves an average success rate of **97.6%**, outperforming MemoryVLA (96.5%) and CogACT (93.2%).
- On **SimplerEnv-Bridge**, it reaches **73.9%** average success, surpassing π₀ (69.2%) and CogACT (57.3%).
- Ablations confirm that both short-term and long-term latent memory are complementary, and that **latent-native integration** (embedding memory into the reasoning sequence) is superior to policy-side external conditioning.

## Introduction and Theoretical Foundation

**Background and Motivation:**  
Most existing Vision-Language-Action (VLA) models [2, 1, 15] adopt a **Markovian assumption** – they predict actions solely from the current observation \(o_t\) and instruction \(I\), ignoring temporal dependencies. This creates a **temporal short-horizon bias**: the model can react to the currently visible state but cannot reason about previous state transitions, completed subtasks, or the current phase of a multi-step task. This limitation becomes critical in long-horizon manipulation.

**Previous Approaches and Their Limitations:**  
Existing memory augmentation for VLA models falls into three categories:  

1. **Temporal-context expansion** (e.g., concatenating historical frames [18, 19], input as video [20, 21, 22]): grows computational cost with context length and has a fixed memory ceiling.  
2. **Sparse history abstraction** (e.g., recurrent latent states [51, 52], action summaries [54]): can lose fine-grained perceptual and semantic details.  
3. **External memory conditioning** (e.g., storing history in a memory bank and retrieving it [23, 24, 25]): memory is stored **outside** the model’s native token space and consumed as auxiliary policy-side context after multimodal reasoning, preventing fluid interleaving with perception and action formation.

**Key Insight:**  
The paper argues that historical experience should be represented as **context-native robotic memory** – stored, retrieved, and consumed in the same continuous embedding space where the VLA model already perceives, reasons, and acts. This latent embedding space is a natural medium because modern VLA models already integrate visual observations and language instructions in a continuous token embedding space [33, 2, 1, 15].

**Dual Memory Types:**  
Long-horizon manipulation requires two complementary forms:  

- **Short-term memory**: visually dominant, preserving recent perceptual evidence (object locations, state changes).  
- **Long-term memory**: semantically dominant, preserving task progress, contextual semantics, and action continuity across longer horizons.

Both are ultimately reconstructed as **latent memory tokens** that can be consumed by the VLA model in the latent space.

## Methodology

### Problem Formulation

The VLA policy \(\Pi_\theta\) at timestep \(t\) takes instruction \(I\) and current observation \(o_t\) to predict a chunk of future actions:

$$
a_{t:t+H-1} = \Pi_\theta(o_t, I),
$$

where \(H\) is the action horizon, and each \(a_t \in \mathbb{R}^7\) (3-DoF translation, 3-DoF rotation, 1-DoF gripper).

### LaMem-VLA Architecture

The model uses a **Prismatic VLM backbone** (7B) + a diffusion action expert. At each timestep \(t\), the vision–language backbone produces visual tokens \(X_t\) and instruction tokens \(I\). Learnable action queries \(Q_{\text{action}}\) are appended.

The four coordinated modules close the loop between latent memory reconstruction and action reasoning:

#### 1. Latent Memory Curator (§3.3)

Organizes historical experience into two vaults:

- **Short-term memory vault** \(M_{\text{short}}\): stores visual tokens from the current episode. Each unit \(m_s^i = (k_s, v_s)\) uses a compression module \(C_s\) to distill \(X_t\) into \(v_s \in \mathbb{R}^{N_s \times C}\) and \(k_s = \text{MeanPool}(v_s) \in \mathbb{R}^C\).
- **Long-term memory vault** \(M_{\text{long}}\): stores action hidden states \(H_{\text{action}}\) directly.

**Updating strategy**: when vault size exceeds capacity \(L\), the most redundant temporally adjacent pair (highest cosine similarity) is consolidated by averaging keys and values (Eq. 4–5).

#### 2. Latent Memory Seeker (§3.4)

Builds a context-aware query \(Q_t\) from the current multimodal cognition state (concatenated visual and instruction tokens) using a lightweight transformer-based query builder \(B\):

$$
Q_t = B([Q_{\text{con}}^t; Q_{\text{init}}])[-K_q:] \in \mathbb{R}^{K_q \times C}.
$$

The mean-pooled query \(q_t = \text{MeanPool}(Q_t)\) retrieves the top-\(K\) units from each vault by cosine similarity:

$$
\mathcal{I}_s = \text{Top-}K\left( \{\cos(q_t, k_s^i)\}_{i=1}^{|M_{\text{short}}|} \right), \quad Z_{\text{short}} = \text{Concat}_{i \in \mathcal{I}_s}(v_s^i) \in \mathbb{R}^{KN_s \times C}.
$$

Same for long-term, producing \(Z_{\text{long}} \in \mathbb{R}^{KN_l \times C}\).

#### 3. Latent Memory Condenser (§3.4)

Compresses retrieved evidence into fixed-length latent memory tokens using learnable slots \(T_s \in \mathbb{R}^{L_s \times C}\) and \(T_l \in \mathbb{R}^{L_l \times C}\), updated with lightweight memory formers \(F_v\) and \(F_c\):

$$
M_{\text{short}} = F_v([Q_t; Z_{\text{short}}; T_s])[-L_s:], \quad M_{\text{long}} = F_c([Q_t; Z_{\text{long}}; T_l])[-L_l:].
$$

The resulting tokens are **query-conditioned** and in the same embedding space as the VLA reasoning.

#### 4. Latent Memory Weaver (§3.5)

Constructs the memory-augmented input sequence:

$$
S_t = [M_{\text{short}} + \mathbf{1}_{L_s} \mathbf{b}_s^\top; \; M_{\text{long}} + \mathbf{1}_{L_l} \mathbf{b}_l^\top; \; X_t; \; I; \; Q_{\text{action}}],
$$

where \(\mathbf{b}_s, \mathbf{b}_l \in \mathbb{R}^C\) are learnable source embeddings. The sequence is fed to the VLM backbone, which outputs memory-grounded action tokens \(Z_{\text{action}}\).

#### Diffusion-based Action Expert

The action expert (≈300M parameters) uses conditional denoising with MSE loss:

$$
\mathcal{L}_{\text{action}} = \mathbb{E}_{n,\epsilon} \left[ \|\epsilon - \epsilon_\theta(a_{t:t+H-1}^n, n, Z_{\text{action}})\|_2^2 \right].
$$

Inference uses DDIM [63] with 10 denoising steps.

## Empirical Validation / Results

### Simulated Evaluation on SimplerEnv-Bridge (§4.2)

**Setup**: Bridge v2 dataset, 50k optimization steps, 24 trials per task.  
**Results** (Table 1):

| Method | Spoon on Towel | Carrot on Plate | Stack Cube | Eggplant in Basket | Avg. Success |
|--------|----------------|-----------------|------------|-------------------|--------------|
| RT-1-X | 0.0 | 4.2 | 0.0 | 0.0 | 1.1 |
| OpenVLA | 4.2 | 0.0 | 0.0 | 12.5 | 4.2 |
| CogACT | 58.3 | 45.8 | 29.2 | 95.8 | 57.3 |
| π₀ | 83.8 | 52.5 | 52.5 | 87.9 | 69.2 |
| MemoryVLA | 75.0 | 75.0 | 37.5 | 100.0 | 71.9 |
| **LaMem-VLA** | **83.3** | **75.0** | **41.7** | **95.8** | **73.9** |

LaMem-VLA achieves the highest average (73.9%), with a **16.6 point gain** over CogACT and **4.7 over π₀**.

### Simulated Evaluation on LIBERO (§4.3)

**Setup**: Franka robot, five suites (Spatial, Object, Goal, Long-10, Long-90). 50 demos per task, separate models for first three, joint for long-horizon.  
**Results** (Table 2):

| Method | Spatial | Object | Goal | Long-10 | Long-90 | Avg. |
|--------|---------|--------|------|---------|---------|------|
| CogACT | 97.2 | 98.0 | 90.2 | 88.8 | 92.1 | 93.2 |
| π₀* | 96.8 | 98.8 | 95.8 | 85.2 | – | 94.2 |
| MemoryVLA | 98.4 | 98.4 | 96.4 | 93.4 | 95.6 | 96.5 |
| **LaMem-VLA** | **98.8** | **99.0** | **97.2** | **95.8** | **97.0** | **97.6** |

LaMem-VLA outperforms all methods, with the largest gains on the long-horizon suites (Long-10: +2.4 over MemoryVLA; Long-90: +1.4).

### Ablation Studies (§4.4)

#### Dual-scale latent memory (Table 3)

| Setting | SimplerEnv | LIBERO-90 |
|---------|------------|-----------|
| w/o Dual Memory | 57.3 | 92.1 |
| w/o Short-term | 65.6 | 95.4 |
| w/o Long-term | 64.6 | 94.8 |
| **LaMem-VLA** | **73.9** | **97.0** |

Both memory streams are complementary; removing either degrades performance.

#### Latent-native vs. policy-side conditioning (Table 4)

| Method | SimplerEnv | LIBERO-90 |
|--------|------------|-----------|
| Baseline (no memory) | 57.3 | 92.1 |
| Policy-side memory | 71.9 | 94.8 |
| Raw retrieval cond. | 69.8 | 95.1 |
| **LaMem-VLA** | **73.9** | **97.0** |

Latent-native integration yields the best results.

#### Number of retrieved units \(K\) (Table 5)

| \(K\) | SimplerEnv | LIBERO-90 |
|------|------------|-----------|
| 2    | 66.7       | 94.4      |
| 4    | 70.8       | 95.9      |
| 8    | **73.9**   | **97.0**  |
| 12   | 71.8       | 96.2      |

\(K=8\) gives best performance; too few or too many hurt.

#### Number of latent memory tokens (Fig. 3)

Varying \(L_s\) (short-term tokens) and \(L_l\) (long-term). Optimal balance: \((L_s, L_l) = (8, 4)\) – used as default.

## Theoretical and Practical Implications

**Theoretical Contribution:**  
LaMem-VLA introduces a new paradigm for robotic memory in VLA models: **historical experience as context-native latent memory**. This bridges the gap between memory storage and the internal reasoning process, allowing memory to directly influence action formation rather than being an external post-hoc conditioner. The dual vault design (visual short-term, semantic long-term) is theoretically motivated by the complementary temporal and functional requirements of long-horizon manipulation.

**Practical Impact:**  
The architecture achieves **state-of-the-art results on two benchmarks** (LIBERO: 97.6%, SimplerEnv: 73.9%) with only a single RGB camera and language input, without proprioception or wrist cameras. The fixed-length latent memory tokens (8 + 4) keep computational overhead low while enabling temporally aware reasoning. The framework is compatible with pretrained VLA backbones and diffusion-based action heads, making it a practical augmentation for existing models.

**Limitation:** Current validation is in simulation; real-world extension is ongoing.

## Conclusion

LaMem-VLA demonstrates that **context-native latent memory** effectively reduces temporal short-horizon bias in VLA models. By organizing historical experience into short-term visual and long-term semantic vaults, retrieving relevant evidence via a context-aware query, condensing it into compact latent tokens, and weaving these tokens directly into the VLA reasoning sequence, the model achieves strong performance on long-horizon and temporally dependent manipulation tasks.

**Future directions** include real-world deployment, scaling to longer trajectories, and exploring more sophisticated memory condensation techniques. The paper provides open-source release of the project page and code.

---

_Markdown view of https://picx.dev/p/fPG4Zi, served by PicX — AI-generated visual whiteboard summaries of research papers._