VOICEMEM: Streaming Dual-Brain Memory for Real-Time Interaction

Summary (Overview)

• VOICEMEM is a streaming dual-brain memory framework designed for real-time conversational systems (speech language models, SLMs), addressing both informational and emotional memory needs without adding perceptible latency.
• The architecture features a left brain for factual memory (schema–entity indexing with cluster emergence) and a right brain for persona/affective memory (independent and cross-entity persona nodes with short- and long-horizon attribution).
• A four-stage streaming retrieval process (listening, speech tail, anticipation, searching) hides retrieval latency within the standard VAD (voice activity detection) silence window (~500 ms), achieving retrieval in just 134 ms.
• The paper introduces CHATMEM-400K, a large-scale training corpus, and CHATMEM-BENCH, a benchmark spanning four dimensions (Information, Persona, Affective Attribution, Paralinguistics & Environment) across 14 fine-grained categories.
• Experimental results show VOICEMEM outperforms existing memory systems by +24.12 points over Mem0 and +15.90 over full-context on factual memory, and achieves +1.89 over the strongest baseline on persona memory, all at a top-5 retrieval budget.

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a critical gap in conversational AI: while modern speech language models (SLMs) and duplex models (e.g., Qwen2.5-Omni, Step-Audio2) have made real-time interaction increasingly natural, they lack persistent memory — the ability to accumulate and retrieve user-specific knowledge across sessions. The authors identify three key obstacles:

1. O1 — Unified architecture for informational and emotional intelligence: Existing memory systems focus almost exclusively on factual information, with limited capacity for emotional association and long-term affective accumulation.
2. O2 — High information density under zero latency: Conventional memory pipelines require 2–3 seconds for retrieval, far exceeding the ~500 ms budget of real-time conversation. Additionally, retrieving top-100 candidates overwhelms SLM context capacity, while top-5 risks omitting relevant memories.
3. O3 — Infrastructure and evolvability: Memory methods and speech models evolve rapidly; systems must be decoupled from specific backends to remain adaptable.

Theoretical Foundation

The paper builds on two lines of prior work:

Embedding-Based Memory Retrieval: Standard RAG retrieves semantically similar records:

$$\mathcal{R}_t^{\mathrm{sem}} = \mathrm{TopK}_{m_i \in \mathcal{M}_t} \, \mathrm{sim}\left(f_\theta(q_t), f_\theta(m_i)\right) \tag{1}$$

with generation and update:

$$o_t = \operatorname{LLM}\left(q_t, \mathcal{R}_t^{\text{sem}}\right), \quad \mathcal{M}_{t+1} = \operatorname{Update}(\mathcal{M}_t, u_t, o_t) \tag{2}$$

Emotion-Aware Retrieval: Systems like Emotional RAG add affective compatibility:

$$\mathcal{R}_t^{\mathrm{aff}} = \mathrm{TopK}_{m_i \in \mathcal{M}_{\star t}} \left[ \lambda \, \mathrm{sim}\left(f_{\star\theta}(q_t), f_{\star\theta}(m_i)\right) + (1 - \lambda) \, \kappa(\mathbf{a}_t, \mathbf{a}_i) \right] \tag{3}$$

The paper extends this into a streaming dual-brain memory framework:

$$\mathcal{B}_t = \left(\mathcal{G}_t^L, \mathcal{G}_t^R, \mathcal{E}_t^{LR}\right) \tag{4}$$

where the left brain $\mathcal{G}_t^L$ organizes factual cells and the right brain $\mathcal{G}_t^R$ maintains affective-attribution cells, connected by cross-brain associations $\mathcal{E}_t^{LR}$.

Methodology

3.1 Left Brain: Efficient Memory Access

The left brain addresses the challenge of dense retrieval under tight budgets through a Cluster–Entity–MemItem indexing scheme:

$$\mathcal{G}^L = (\mathcal{S}, \mathcal{V}, \mathcal{E}), \qquad v = (d_v, \mathcal{N}_v^{\mathrm{micro}}, \mathcal{I}_v), \qquad s = (d_s, \mathcal{N}_s^{\mathrm{macro}}, \mathcal{V}_s)$$

Each entity $v \in \mathcal{V}$ belongs to exactly one schema $s \in \mathcal{S}$. The edge set $\mathcal{E} = \mathcal{E}_{\mathrm{micro}} \cup \mathcal{E}_{\mathrm{macro}}$ supports lightweight semantic expansion.

Retrieval uses a streaming matcher that identifies relevant schemas and entities from partial transcripts, then expands through one-hop connections:

$$(\mathcal{V}_t, \mathcal{S}_t) = \mathrm{Match}(x_{\leq t}, \mathcal{V}, \mathcal{S}), \qquad \mathcal{Z}_t = \mathcal{V}_t \cup \mathcal{V}_{\mathcal{S}_t} \cup \mathcal{N}_1^{\mathrm{strong}}(\cdot) \cup \mathcal{N}_1^{\mathrm{weak}}(\cdot) \tag{1}$$

The candidate pool is then:

$$\mathcal{C}_t^L = \bigcup_{z \in \mathcal{Z}_t} \mathcal{I}_z, \qquad \mathcal{R}_t^L = \mathrm{MemSearch}(q_t, \mathcal{C}_t; K) \tag{2}$$

Cluster Emergence Mechanism: As clusters grow, information density decreases. The system measures query coherence:

$$\rho(H) = \frac{1}{|\mathcal{Q}|} \sum_{q \in \mathcal{Q}} \frac{|A_q \cap H|}{|A_q \cup H|} \tag{3}$$

If the largest qualifying subgraph exceeds threshold $\alpha$, an LLM judge evaluates relevance, importance, and completeness before promoting it to a new cluster (Algorithm 1).

3.2 Right Brain: Knowing the Person

The right brain maintains two types of persona nodes:

$$\mathcal{G}^R = \left(\mathcal{V}^I, \mathcal{V}^C\right), \qquad v^I = \left(d_v^I, \mathcal{I}_v^I\right) \in \mathcal{V}^I, \qquad v_e^C = \left(d_{v,e}^C, \mathcal{I}_{v,e}^C, \rho_{v,e}\right) \in \mathcal{V}^C, \quad e \in \mathcal{V} \tag{4}$$

• Independent nodes $v^I$: encode user-intrinsic properties (enduring dispositions, behavioral regularities)
• Cross-entity nodes $v_e^C$: capture context-dependent affect linked to left-brain entities

Retrieval jointly activates both node types:

$$\mathcal{Z}_t^R = \mathcal{V}_t^I \cup \mathcal{V}_t^C \cup \left\{v_e^C \in \mathcal{V}^C : e \in \mathcal{Z}_t\right\}, \qquad \mathcal{R}_t^R = \mathrm{MemSearch}(q_t, \mathcal{C}_t^R; K) \tag{5}$$

Short-Horizon Attribution processes each turn:

$$e_t = \phi(x_t), \qquad \mathcal{G}_t^R = \mathrm{Modify}\left(\mathcal{G}_{t-1}^R; x_t, e_t\right), \qquad t = 1, \ldots, T \tag{6}$$

Long-Horizon Attribution consolidates after each session:

$$\mathcal{V}^I \leftarrow \text{Consolidate}\left(\mathcal{V}^I; (x_1, e_1), (x_2, e_2), \ldots, (x_T, e_T)\right) \tag{7}$$

3.3 Streaming Dual-Brain Retrieval

Four stages hide retrieval latency within the VAD silence window:

1. Listening & Speech Tail (0–200 ms): Streaming ASR, entity/schema matching, and speaker identification
2. Anticipation (200–400 ms): Query embedding and dual-brain graph expansion
3. Searching (400–500 ms): Backend search and merging of both brains' results

The full pipeline costs only 134 ms for dense dual-brain retrieval, well within the 500 ms VAD threshold.

3.4 Multimodal Extension

VOICEMEM extends to audio memory beyond text: speaker voiceprints, acoustic embeddings, or raw waveforms attached as multimodal nodes to corresponding entities.

4.1 Black-box OPD Training

The training pipeline converts standard SLMs into memory-augmented models through online black-box on-policy distillation:

• Stage I: Memory world construction (Persona → Background → Events → Messages → Memory)
• Stage II: SLM-verified online distillation with iterative generation, contrastive distillation, and verification
• Stage III: Human curation for harder samples with richer emotional dynamics
• Stage IV: Validation on CHATMEM-BENCH

4.2 Decoupled Architecture

The system decouples upper-level routing from the underlying memory engine (abstracted as MemSearch), with MEM0 as the current backend due to its generality and performance.

Empirical Validation / Results

5.2 Main Results — Factual Memory (Table 1)

VOICEMEM achieves 76.39 average across 11 information sub-categories on LoCoMo, LongMemEval, and Memora:

Key findings:

• +24.12 over Mem0 (its own storage backend)
• +15.90 over full-context (which sees the entire history)
• Largest margin on temporal reasoning (+54.9), smallest on update tracking (+7.4)

5.2 Persona Memory (Table 2)

VOICEMEM reaches 74.16 (with GPT-4o-mini) and 76.56 (with fine-tuned model), surpassing MemOS by +1.89. Notably on ES-MemEval, full-context scores only 12.10 on conflict detection and 21.70 on user modeling, versus 69.10 and 74.00 for VOICEMEM.

5.3 CHATMEM-BENCH (Table 3)

On the long-horizon audio benchmark (316 questions, 53 hours of dialogue):

• 11 of 14 categories led by VOICEMEM
• Paralinguistics & Environment: text systems score 3.23–26.92, VOICEMEM reaches 45.16–53.84
• Affective Attribution: text baselines remain competitive; audio increment appears on attachment-guided decisions and affect-aware phrasing

5.4 Ablation and Analysis

Time–Cost–Accuracy (Figure 5): VOICEMEM at K=5 achieves 91.2 with 430 memory tokens and 134 ms retrieval. Strongest baseline (EverMemOS) reaches 83.13 with 1,899 tokens — +8.1 points at 4.4× fewer tokens.

Retrieval Budget Sweep (Figure 6): VOICEMEM is highest at every K. Margin widest at K=1 (+11.8 over EverMemOS, +26.5 over Mem0). Removing schema routing costs 4.53 points at K=3 but closes to 0.05 by K=5.

Component Ablation (Figure 7): All five mechanisms contribute on all four datasets:

Structure Analysis (Figure 8): Emergent clusters span 49.8% of the store (254 of 510 items), re-partitioning across preset boundaries rather than refining within them.

Backend Transfer (Table 4): The index improves all three stores by 15.8–29.5 points without threshold retuning:

Theoretical and Practical Implications

Theoretical Contributions

1. Dual-brain memory architecture: The separation of factual (left brain) and affective/persona (right brain) memory with cross-brain associations provides a principled framework for modeling both information and emotion in conversational systems.

2. Query-driven cluster emergence: The coherence-based emergence mechanism ($\rho(H)$ with threshold $\alpha$) demonstrates that semantic structure can emerge from retrieval patterns rather than requiring predefined taxonomy.

3. Dense candidate pool over sophisticated ranking: The key insight that at small K, performance depends on the semantic density of the candidate space rather than ranking sophistication challenges conventional RAG assumptions.

Practical Implications

1. Real-time feasibility: Retrieval in 134 ms fits within standard 500 ms VAD budgets, making memory-augmented voice assistants practical for deployment.

2. Cost efficiency: 430 tokens at K=5 versus 1,899 tokens for baselines represents a 4.4× reduction in context usage, with accuracy gains.

3. Backend agnosticism: The decoupled architecture allows rapid adoption of new memory engines without redesign.

4. Multimodal memory: Extension to audio (voiceprints, acoustic embeddings, environmental sounds) enables capabilities unavailable to text-only systems.

Conclusion

VOICEMEM presents a streaming dual-brain memory framework that equips real-time conversational systems with both informational and emotional memory without breaking latency budgets. The left brain organizes factual knowledge through schema–entity indexing with query-driven cluster emergence, while the right brain models persona through independent and cross-entity nodes maintained by short- and long-horizon affective attribution. A four-stage streaming query hides retrieval within standard VAD silence.

Key results demonstrate:

• 91.2 on LoCoMo with 430 memory tokens and 134 ms retrieval
• +24.12 over Mem0 and +15.90 over full-context on factual memory
• +1.89 over strongest baseline on persona memory
• 15.8–29.5 point improvements across three different backends

By turning turn-level speech understanding into continuous, personalized user understanding at essentially no latency cost, VOICEMEM lays a practical memory foundation for the next generation of empathetic, real-time voice assistants. Future directions include deeper integration of multimodal memory, exploration of parameterized and latent memory mechanisms, and expansion of the CHATMEM-BENCH benchmark.