# VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction

> VOICEMEM introduces a dual-brain streaming memory framework that retrieves both factual and emotional context in 134ms, outperforming prior systems by up to 24 points.

- **Source:** [arXiv](https://arxiv.org/abs/2608.26005)
- **Published:** 2026-08-28
- **Permalink:** https://picx.dev/p/Tcpu93
- **Whiteboard:** https://picx.dev/p/Tcpu93/image

## Summary

# VOICEMEM: Streaming Dual-Brain Memory for Real-Time Interaction

## Summary (Overview)

- **VOICEMEM** is a streaming dual-brain memory framework designed for real-time conversational systems (speech language models, SLMs), addressing both informational and emotional memory needs without adding perceptible latency.
- The architecture features a **left brain** for factual memory (schema–entity indexing with cluster emergence) and a **right brain** for persona/affective memory (independent and cross-entity persona nodes with short- and long-horizon attribution).
- A **four-stage streaming retrieval** process (listening, speech tail, anticipation, searching) hides retrieval latency within the standard VAD (voice activity detection) silence window (~500 ms), achieving retrieval in just **134 ms**.
- The paper introduces **CHATMEM-400K**, a large-scale training corpus, and **CHATMEM-BENCH**, a benchmark spanning four dimensions (Information, Persona, Affective Attribution, Paralinguistics & Environment) across 14 fine-grained categories.
- Experimental results show VOICEMEM outperforms existing memory systems by **+24.12 points over Mem0** and **+15.90 over full-context** on factual memory, and achieves **+1.89 over the strongest baseline** on persona memory, all at a top-5 retrieval budget.

## Introduction and Theoretical Foundation

### Background and Motivation

The paper addresses a critical gap in conversational AI: while modern speech language models (SLMs) and duplex models (e.g., Qwen2.5-Omni, Step-Audio2) have made real-time interaction increasingly natural, they lack **persistent memory** — the ability to accumulate and retrieve user-specific knowledge across sessions. The authors identify three key obstacles:

1. **O1 — Unified architecture for informational and emotional intelligence**: Existing memory systems focus almost exclusively on factual information, with limited capacity for emotional association and long-term affective accumulation.
2. **O2 — High information density under zero latency**: Conventional memory pipelines require 2–3 seconds for retrieval, far exceeding the ~500 ms budget of real-time conversation. Additionally, retrieving top-100 candidates overwhelms SLM context capacity, while top-5 risks omitting relevant memories.
3. **O3 — Infrastructure and evolvability**: Memory methods and speech models evolve rapidly; systems must be decoupled from specific backends to remain adaptable.

### Theoretical Foundation

The paper builds on two lines of prior work:

**Embedding-Based Memory Retrieval**: Standard RAG retrieves semantically similar records:

$$\mathcal{R}_t^{\mathrm{sem}} = \mathrm{TopK}_{m_i \in \mathcal{M}_t} \, \mathrm{sim}\left(f_\theta(q_t), f_\theta(m_i)\right) \tag{1}$$

with generation and update:

$$o_t = \operatorname{LLM}\left(q_t, \mathcal{R}_t^{\text{sem}}\right), \quad \mathcal{M}_{t+1} = \operatorname{Update}(\mathcal{M}_t, u_t, o_t) \tag{2}$$

**Emotion-Aware Retrieval**: Systems like Emotional RAG add affective compatibility:

$$\mathcal{R}_t^{\mathrm{aff}} = \mathrm{TopK}_{m_i \in \mathcal{M}_{\star t}} \left[ \lambda \, \mathrm{sim}\left(f_{\star\theta}(q_t), f_{\star\theta}(m_i)\right) + (1 - \lambda) \, \kappa(\mathbf{a}_t, \mathbf{a}_i) \right] \tag{3}$$

The paper extends this into a **streaming dual-brain memory** framework:

$$\mathcal{B}_t = \left(\mathcal{G}_t^L, \mathcal{G}_t^R, \mathcal{E}_t^{LR}\right) \tag{4}$$

where the left brain $\mathcal{G}_t^L$ organizes factual cells and the right brain $\mathcal{G}_t^R$ maintains affective-attribution cells, connected by cross-brain associations $\mathcal{E}_t^{LR}$.

## Methodology

### 3.1 Left Brain: Efficient Memory Access

The left brain addresses the challenge of dense retrieval under tight budgets through a **Cluster–Entity–MemItem indexing** scheme:

$$\mathcal{G}^L = (\mathcal{S}, \mathcal{V}, \mathcal{E}), \qquad v = (d_v, \mathcal{N}_v^{\mathrm{micro}}, \mathcal{I}_v), \qquad s = (d_s, \mathcal{N}_s^{\mathrm{macro}}, \mathcal{V}_s)$$

Each entity $v \in \mathcal{V}$ belongs to exactly one schema $s \in \mathcal{S}$. The edge set $\mathcal{E} = \mathcal{E}_{\mathrm{micro}} \cup \mathcal{E}_{\mathrm{macro}}$ supports lightweight semantic expansion.

**Retrieval** uses a streaming matcher that identifies relevant schemas and entities from partial transcripts, then expands through one-hop connections:

$$(\mathcal{V}_t, \mathcal{S}_t) = \mathrm{Match}(x_{\leq t}, \mathcal{V}, \mathcal{S}), \qquad \mathcal{Z}_t = \mathcal{V}_t \cup \mathcal{V}_{\mathcal{S}_t} \cup \mathcal{N}_1^{\mathrm{strong}}(\cdot) \cup \mathcal{N}_1^{\mathrm{weak}}(\cdot) \tag{1}$$

The candidate pool is then:

$$\mathcal{C}_t^L = \bigcup_{z \in \mathcal{Z}_t} \mathcal{I}_z, \qquad \mathcal{R}_t^L = \mathrm{MemSearch}(q_t, \mathcal{C}_t; K) \tag{2}$$

**Cluster Emergence Mechanism**: As clusters grow, information density decreases. The system measures query coherence:

$$\rho(H) = \frac{1}{|\mathcal{Q}|} \sum_{q \in \mathcal{Q}} \frac{|A_q \cap H|}{|A_q \cup H|} \tag{3}$$

If the largest qualifying subgraph exceeds threshold $\alpha$, an LLM judge evaluates relevance, importance, and completeness before promoting it to a new cluster (Algorithm 1).

### 3.2 Right Brain: Knowing the Person

The right brain maintains two types of persona nodes:

$$\mathcal{G}^R = \left(\mathcal{V}^I, \mathcal{V}^C\right), \qquad v^I = \left(d_v^I, \mathcal{I}_v^I\right) \in \mathcal{V}^I, \qquad v_e^C = \left(d_{v,e}^C, \mathcal{I}_{v,e}^C, \rho_{v,e}\right) \in \mathcal{V}^C, \quad e \in \mathcal{V} \tag{4}$$

- **Independent nodes** $v^I$: encode user-intrinsic properties (enduring dispositions, behavioral regularities)
- **Cross-entity nodes** $v_e^C$: capture context-dependent affect linked to left-brain entities

**Retrieval** jointly activates both node types:

$$\mathcal{Z}_t^R = \mathcal{V}_t^I \cup \mathcal{V}_t^C \cup \left\{v_e^C \in \mathcal{V}^C : e \in \mathcal{Z}_t\right\}, \qquad \mathcal{R}_t^R = \mathrm{MemSearch}(q_t, \mathcal{C}_t^R; K) \tag{5}$$

**Short-Horizon Attribution** processes each turn:

$$e_t = \phi(x_t), \qquad \mathcal{G}_t^R = \mathrm{Modify}\left(\mathcal{G}_{t-1}^R; x_t, e_t\right), \qquad t = 1, \ldots, T \tag{6}$$

**Long-Horizon Attribution** consolidates after each session:

$$\mathcal{V}^I \leftarrow \text{Consolidate}\left(\mathcal{V}^I; (x_1, e_1), (x_2, e_2), \ldots, (x_T, e_T)\right) \tag{7}$$

### 3.3 Streaming Dual-Brain Retrieval

Four stages hide retrieval latency within the VAD silence window:

1. **Listening & Speech Tail (0–200 ms)**: Streaming ASR, entity/schema matching, and speaker identification
2. **Anticipation (200–400 ms)**: Query embedding and dual-brain graph expansion
3. **Searching (400–500 ms)**: Backend search and merging of both brains' results

The full pipeline costs only **134 ms** for dense dual-brain retrieval, well within the 500 ms VAD threshold.

### 3.4 Multimodal Extension

VOICEMEM extends to audio memory beyond text: speaker voiceprints, acoustic embeddings, or raw waveforms attached as multimodal nodes to corresponding entities.

### 4.1 Black-box OPD Training

The training pipeline converts standard SLMs into memory-augmented models through **online black-box on-policy distillation**:

- **Stage I**: Memory world construction (Persona → Background → Events → Messages → Memory)
- **Stage II**: SLM-verified online distillation with iterative generation, contrastive distillation, and verification
- **Stage III**: Human curation for harder samples with richer emotional dynamics
- **Stage IV**: Validation on CHATMEM-BENCH

### 4.2 Decoupled Architecture

The system decouples upper-level routing from the underlying memory engine (abstracted as `MemSearch`), with MEM0 as the current backend due to its generality and performance.

## Empirical Validation / Results

### 5.2 Main Results — Factual Memory (Table 1)

VOICEMEM achieves **76.39 average** across 11 information sub-categories on LoCoMo, LongMemEval, and Memora:

| Method | LoCoMo Avg | LongMemEval Avg | Memora Avg | Overall Avg |
|--------|-----------|-----------------|------------|-------------|
| **VoiceMem** | **90.40** | **82.73** | **49.33** | **76.39** |
| EverMemOS | 83.13 | 80.89 | 22.38 | 65.75 |
| MemOS | 69.94 | 77.32 | 45.05 | 65.83 |
| Mem0 | 61.68 | 54.83 | 36.33 | 52.27 |
| Full-Context | 68.97 | 57.29 | 53.44 | 60.49 |

Key findings:
- **+24.12 over Mem0** (its own storage backend)
- **+15.90 over full-context** (which sees the entire history)
- Largest margin on temporal reasoning (+54.9), smallest on update tracking (+7.4)

### 5.2 Persona Memory (Table 2)

VOICEMEM reaches **74.16** (with GPT-4o-mini) and **76.56** (with fine-tuned model), surpassing MemOS by **+1.89**. Notably on ES-MemEval, full-context scores only 12.10 on conflict detection and 21.70 on user modeling, versus 69.10 and 74.00 for VOICEMEM.

### 5.3 CHATMEM-BENCH (Table 3)

On the long-horizon audio benchmark (316 questions, 53 hours of dialogue):

- **11 of 14 categories** led by VOICEMEM
- **Paralinguistics & Environment**: text systems score 3.23–26.92, VOICEMEM reaches 45.16–53.84
- **Affective Attribution**: text baselines remain competitive; audio increment appears on attachment-guided decisions and affect-aware phrasing

### 5.4 Ablation and Analysis

**Time–Cost–Accuracy (Figure 5)**: VOICEMEM at K=5 achieves 91.2 with 430 memory tokens and 134 ms retrieval. Strongest baseline (EverMemOS) reaches 83.13 with 1,899 tokens — +8.1 points at 4.4× fewer tokens.

**Retrieval Budget Sweep (Figure 6)**: VOICEMEM is highest at every K. Margin widest at K=1 (+11.8 over EverMemOS, +26.5 over Mem0). Removing schema routing costs 4.53 points at K=3 but closes to 0.05 by K=5.

**Component Ablation (Figure 7)**: All five mechanisms contribute on all four datasets:

| Ablation | LoCoMo | ES-MemEval | ChatMem-Bench | Memora |
|----------|--------|------------|---------------|--------|
| w/o upper-layer index | −9.9 | −5.3 | −6.7 | −4.4 |
| w/o right brain | −6.3 | −4.3 | −5.4 | −4.4 |
| w/o emergent cluster | −5.5 | −2.0 | −3.4 | −0.2 |
| w/o dual horizon | −5.4 | −2.0 | −3.2 | −1.4 |
| w/o joint retrieval | −2.6 | −3.1 | −2.7 | −0.4 |

**Structure Analysis (Figure 8)**: Emergent clusters span 49.8% of the store (254 of 510 items), re-partitioning across preset boundaries rather than refining within them.

**Backend Transfer (Table 4)**: The index improves all three stores by 15.8–29.5 points without threshold retuning:

| Backend | bare | + ours | Δ |
|---------|------|--------|---|
| Mem0 | 61.68 | 91.20 | +29.52 |
| LangMem | 56.18 | 71.94 | +15.76 |
| Zep | 62.93 | 85.85 | +22.92 |

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Dual-brain memory architecture**: The separation of factual (left brain) and affective/persona (right brain) memory with cross-brain associations provides a principled framework for modeling both information and emotion in conversational systems.

2. **Query-driven cluster emergence**: The coherence-based emergence mechanism ($\rho(H)$ with threshold $\alpha$) demonstrates that semantic structure can emerge from retrieval patterns rather than requiring predefined taxonomy.

3. **Dense candidate pool over sophisticated ranking**: The key insight that at small K, performance depends on the semantic density of the candidate space rather than ranking sophistication challenges conventional RAG assumptions.

### Practical Implications

1. **Real-time feasibility**: Retrieval in 134 ms fits within standard 500 ms VAD budgets, making memory-augmented voice assistants practical for deployment.

2. **Cost efficiency**: 430 tokens at K=5 versus 1,899 tokens for baselines represents a 4.4× reduction in context usage, with accuracy gains.

3. **Backend agnosticism**: The decoupled architecture allows rapid adoption of new memory engines without redesign.

4. **Multimodal memory**: Extension to audio (voiceprints, acoustic embeddings, environmental sounds) enables capabilities unavailable to text-only systems.

## Conclusion

VOICEMEM presents a streaming dual-brain memory framework that equips real-time conversational systems with both informational and emotional memory without breaking latency budgets. The left brain organizes factual knowledge through schema–entity indexing with query-driven cluster emergence, while the right brain models persona through independent and cross-entity nodes maintained by short- and long-horizon affective attribution. A four-stage streaming query hides retrieval within standard VAD silence.

Key results demonstrate:
- **91.2 on LoCoMo** with 430 memory tokens and 134 ms retrieval
- **+24.12 over Mem0** and **+15.90 over full-context** on factual memory
- **+1.89 over strongest baseline** on persona memory
- **15.8–29.5 point improvements** across three different backends

By turning turn-level speech understanding into continuous, personalized user understanding at essentially no latency cost, VOICEMEM lays a practical memory foundation for the next generation of empathetic, real-time voice assistants. Future directions include deeper integration of multimodal memory, exploration of parameterized and latent memory mechanisms, and expansion of the CHATMEM-BENCH benchmark.

---

_Markdown view of https://picx.dev/p/Tcpu93, served by PicX — AI-generated visual whiteboard summaries of research papers._