Summary (Overview)

• LightMem-Ego is a lightweight streaming multimodal memory system for everyday-life assistance, designed to be deployed on smartphones and AI glasses.
• It continuously captures egocentric visual and audio streams, aligns them on a shared timeline, and organizes them into a three-level memory hierarchy: current memory (ongoing context), short-term memory (recent micro-events), and long-term memory (consolidated episodic and semantic knowledge).
• The system supports four key scenarios: object finding, conversation recall, life summarization, and routine discovery, routing user queries to the appropriate memory level and generating answers grounded in multimodal evidence.
• Quantitative evaluation shows overall retrieval Recall@3 of 74.1% and MRR of 0.627, with end-to-end QA latency of 5.86s (P50) on phone for short-term memory queries.
• Compared to existing systems (ChatGPT Memory, Gemini Live, Ray-Ban Meta AI, EgoButler, etc.), LightMem-Ego is the only one to explicitly support all five capabilities: real-time visual-audio stream, current/short-term multimodal memory, long-term multimodal episodic memory, long-term semantic memory, and timestamped evidence retrieval.

Introduction and Theoretical Foundation

The paper addresses the fundamental challenge of turning mobile and wearable devices (smartphones, AI glasses) into everyday memory companions. While recent multimodal large language models (MLLMs) make human-computer interaction more natural, and wearable devices provide continuous hands-free perception, three key challenges remain:

1. Continuous egocentric streams without explicit event boundaries – raw observations must be segmented into coherent events.
2. Incremental memory organization – experiences must be organized into current, short-term, episodic, and semantic memory efficiently for long-term deployment.
3. Dynamic temporal retrieval – user queries span multiple horizons (present, recent past, long-term routines), requiring routing across different memory levels rather than a single context window.

The theoretical foundation draws from cognitive science concepts of episodic and semantic memory (Packer et al., 2023; Chhikara et al., 2025; Fang et al., 2025; Xu et al., 2026), and builds on prior work in conversational memory (MemGPT, Mem0), wearable AI assistants (Vinci, Egocentric Co-Pilot, EgoButler), and multimodal personal memory systems (Memories.ai, long-context egocentric VQA).

Methodology

LightMem-Ego consists of four main components:

1. Multimodal Life Capture – Everyday experience is represented as a temporally ordered multimodal stream:

$$X = \{ x_t \}_{t=1}^T, \quad x_t = (v_t, a_t, m_t),$$

where $v_t$, $a_t$, and $m_t$ denote visual observations, audio observations, and auxiliary metadata respectively. All modalities are aligned on a shared session timeline using relative timestamp $\tau = t - t_0$.

2. Event Segmentation – The stream is incrementally partitioned into short event segments $E = \{ e_i \}_{i=1}^N$, each corresponding to a contiguous interval. Segmentation uses temporal continuity and cross-frame change signals, without full semantic parsing. Each segment stores temporal span, representative frames, provisional visual description, and aligned/pending audio context.

3. Hierarchical Memory – Experiences are organized into three levels:

$$M = \{ M_{cur}, M_{st}, M_{lt} \},$$

where $M_{cur}$ is current working memory (ongoing scene), $M_{st}$ stores recent event segments with visual evidence and transcripts, and $M_{lt}$ contains episodic memory $M_{epi}$ (event-centered past experiences) and semantic memory $M_{sem}$ (routines, preferences, relationships). Consolidation happens asynchronously.

4. Edge-Oriented Efficiency – Client-side processing is lightweight (sampling, compression, timestamping, upload). Backend handles online updates for $M_{cur}$ and $M_{st}$, and asynchronous operations for ASR backfilling, event refinement, indexing, and semantic extraction. Query-time routing selects the cheapest sufficient source: $M_{cur}$ for current-scene, $M_{st}$ for recent, $M_{lt}$ for retrospective queries.

5. Experience Retrieval and QA – Given query $q$, a router produces evidence set $R(q) \subseteq M_{cur} \cup M_{st} \cup M_{lt}$, guided by temporal scope and semantic intent. Retrieved evidence (observations, event records, episodic entries, summaries, timestamps, frames, transcripts) is fused into compact evidence view $E_q$, and answer is generated as $\hat{y} = f(q, E_q)$.

Empirical Validation / Results

The system is evaluated on three daily memory scenarios with manually annotated gold evidence. Key results are in Tables 1–3.

Table 1: Memory Retrieval Accuracy

Table 2: Experience QA Accuracy

Table 3: Latency Breakdown by Memory Scope and Client

Table 4: Capability Comparison – LightMem-Ego is the only system among 10 compared (ChatGPT Memory, Mem0, Memories.ai, Gemini Live, Ray-Ban Meta AI, Vinci, VisualClaw, VisionClaw, Egocentric Co-Pilot, EgoButler) that explicitly supports all five capabilities: real-time visual-audio stream, current/short-term multimodal memory, long-term multimodal episodic memory, long-term semantic memory, and timestamped evidence retrieval.

Theoretical and Practical Implications

• Theoretical: The paper demonstrates that a hierarchical memory architecture (current → short-term → long-term) with explicit event segmentation and dynamic routing is effective for everyday-life memory assistance. It connects cognitive science concepts of episodic and semantic memory with practical streaming multimodal systems.
• Practical: The system shows that smartphones and AI glasses can serve as everyday memory companions, with near-interactive response times for recent queries (5.86s phone P50) and acceptable latency for long-term recall (14.87s phone P50). The hierarchical design enables both immediate assistance and retrospective reasoning within a unified backend.
• Comparison: LightMem-Ego fills a gap in the landscape of AI assistants – most existing systems focus on either text-based conversation memory, real-time perception, or agentic task execution, but not on long-horizon multimodal experience memory with explicit event organization and routine discovery.

Conclusion

The paper introduces LightMem-Ego, a streaming multimodal memory system for everyday-life assistance. It connects mobile/wearable capture with hierarchical memory construction and memory-grounded QA, supporting object finding, conversation recall, life summarization, and routine discovery. The evaluation suggests that explicit long-horizon multimodal memory is a promising direction for personal AI assistants.

Limitations acknowledged: reliance on upstream API calls (ASR, VLM, LLM) making accuracy and latency sensitive to external services; error propagation from transcripts/descriptions; non-trivial overhead for continuous segmentation, embedding, indexing; and a preliminary memory update mechanism lacking principled revision, merging, forgetting, or promotion policies.

Future work will focus on improving efficiency, robustness, and adaptive memory lifecycle management, as well as integrating privacy protection (on-device preprocessing, selective capture, sensitive-content filtering, encrypted storage, user-controlled editing/deletion, privacy-aware consolidation).