# LightMem-Ego: Your AI Memory for Everyday Life

> LightMem-Ego is a lightweight hierarchical multimodal memory system enabling real-time everyday assistance on smartphones and AI glasses.

- **Source:** [arXiv](https://arxiv.org/abs/2607.11487)
- **Published:** 2026-07-15
- **Permalink:** https://picx.dev/p/QCAwQB
- **Whiteboard:** https://picx.dev/p/QCAwQB/image

## Summary

## Summary (Overview)

- **LightMem-Ego** is a lightweight streaming multimodal memory system for everyday-life assistance, designed to be deployed on smartphones and AI glasses.
- It continuously captures egocentric visual and audio streams, aligns them on a shared timeline, and organizes them into a three-level memory hierarchy: current memory (ongoing context), short-term memory (recent micro-events), and long-term memory (consolidated episodic and semantic knowledge).
- The system supports four key scenarios: **object finding**, **conversation recall**, **life summarization**, and **routine discovery**, routing user queries to the appropriate memory level and generating answers grounded in multimodal evidence.
- Quantitative evaluation shows overall retrieval Recall@3 of 74.1% and MRR of 0.627, with end-to-end QA latency of 5.86s (P50) on phone for short-term memory queries.
- Compared to existing systems (ChatGPT Memory, Gemini Live, Ray-Ban Meta AI, EgoButler, etc.), LightMem-Ego is the only one to explicitly support all five capabilities: real-time visual-audio stream, current/short-term multimodal memory, long-term multimodal episodic memory, long-term semantic memory, and timestamped evidence retrieval.

## Introduction and Theoretical Foundation

The paper addresses the fundamental challenge of turning mobile and wearable devices (smartphones, AI glasses) into **everyday memory companions**. While recent multimodal large language models (MLLMs) make human-computer interaction more natural, and wearable devices provide continuous hands-free perception, three key challenges remain:

1. **Continuous egocentric streams** without explicit event boundaries – raw observations must be segmented into coherent events.
2. **Incremental memory organization** – experiences must be organized into current, short-term, episodic, and semantic memory efficiently for long-term deployment.
3. **Dynamic temporal retrieval** – user queries span multiple horizons (present, recent past, long-term routines), requiring routing across different memory levels rather than a single context window.

The theoretical foundation draws from cognitive science concepts of **episodic and semantic memory** (Packer et al., 2023; Chhikara et al., 2025; Fang et al., 2025; Xu et al., 2026), and builds on prior work in conversational memory (MemGPT, Mem0), wearable AI assistants (Vinci, Egocentric Co-Pilot, EgoButler), and multimodal personal memory systems (Memories.ai, long-context egocentric VQA).

## Methodology

LightMem-Ego consists of four main components:

**1. Multimodal Life Capture** – Everyday experience is represented as a temporally ordered multimodal stream:
$$X = \{ x_t \}_{t=1}^T, \quad x_t = (v_t, a_t, m_t),$$
where $v_t$, $a_t$, and $m_t$ denote visual observations, audio observations, and auxiliary metadata respectively. All modalities are aligned on a shared session timeline using relative timestamp $\tau = t - t_0$.

**2. Event Segmentation** – The stream is incrementally partitioned into short event segments $E = \{ e_i \}_{i=1}^N$, each corresponding to a contiguous interval. Segmentation uses temporal continuity and cross-frame change signals, without full semantic parsing. Each segment stores temporal span, representative frames, provisional visual description, and aligned/pending audio context.

**3. Hierarchical Memory** – Experiences are organized into three levels:
$$M = \{ M_{cur}, M_{st}, M_{lt} \},$$
where $M_{cur}$ is current working memory (ongoing scene), $M_{st}$ stores recent event segments with visual evidence and transcripts, and $M_{lt}$ contains episodic memory $M_{epi}$ (event-centered past experiences) and semantic memory $M_{sem}$ (routines, preferences, relationships). Consolidation happens asynchronously.

**4. Edge-Oriented Efficiency** – Client-side processing is lightweight (sampling, compression, timestamping, upload). Backend handles online updates for $M_{cur}$ and $M_{st}$, and asynchronous operations for ASR backfilling, event refinement, indexing, and semantic extraction. Query-time routing selects the cheapest sufficient source: $M_{cur}$ for current-scene, $M_{st}$ for recent, $M_{lt}$ for retrospective queries.

**5. Experience Retrieval and QA** – Given query $q$, a router produces evidence set $R(q) \subseteq M_{cur} \cup M_{st} \cup M_{lt}$, guided by temporal scope and semantic intent. Retrieved evidence (observations, event records, episodic entries, summaries, timestamps, frames, transcripts) is fused into compact evidence view $E_q$, and answer is generated as $\hat{y} = f(q, E_q)$.

## Empirical Validation / Results

The system is evaluated on three daily memory scenarios with manually annotated gold evidence. Key results are in Tables 1–3.

**Table 1: Memory Retrieval Accuracy**

| Scenario | R@1 | R@3 | R@5 | MRR |
|----------|:---:|:---:|:---:|:---:|
| Object Finding | 22.2 | 66.7 | 77.8 | 0.454 |
| Conversation Recall | 44.4 | 55.6 | 55.6 | 0.481 |
| Life Summarization | 88.9 | 100.0 | 100.0 | 0.944 |
| **Overall** | **51.9** | **74.1** | **77.8** | **0.627** |

**Table 2: Experience QA Accuracy**

| Scenario | LLM-Judge | Human Acc. |
|----------|:---------:|:----------:|
| Object Finding | 44.4 | 55.6 |
| Conversation Recall | 33.3 | 33.3 |
| Life Summarization | 77.8 | 77.8 |
| **Overall** | **51.9** | **55.6** |

**Table 3: Latency Breakdown by Memory Scope and Client**

| Stage | Phone P50 | Phone P90 | Glasses-style P50 | Glasses-style P90 |
|:------|:---------:|:---------:|:-----------------:|:-----------------:|
| **Short-term memory QA** | | | | |
| Retrieval | 13 ms | 15 ms | 14 ms | 29 ms |
| Answer gen. | 5.77 s | 10.38 s | 6.10 s | 9.79 s |
| End-to-end QA | 5.86 s | 10.95 s | 7.01 s | 9.96 s |
| **Long-term memory QA** | | | | |
| Retrieval | 4.09 s | 15.39 s | 10.39 s | 28.93 s |
| Answer gen. | 9.00 s | 22.40 s | 9.25 s | 22.62 s |
| End-to-end QA | 14.87 s | 35.15 s | 19.96 s | 42.70 s |

**Table 4: Capability Comparison** – LightMem-Ego is the only system among 10 compared (ChatGPT Memory, Mem0, Memories.ai, Gemini Live, Ray-Ban Meta AI, Vinci, VisualClaw, VisionClaw, Egocentric Co-Pilot, EgoButler) that explicitly supports all five capabilities: real-time visual-audio stream, current/short-term multimodal memory, long-term multimodal episodic memory, long-term semantic memory, and timestamped evidence retrieval.

## Theoretical and Practical Implications

- **Theoretical**: The paper demonstrates that a hierarchical memory architecture (current → short-term → long-term) with explicit event segmentation and dynamic routing is effective for everyday-life memory assistance. It connects cognitive science concepts of episodic and semantic memory with practical streaming multimodal systems.
- **Practical**: The system shows that smartphones and AI glasses can serve as everyday memory companions, with near-interactive response times for recent queries (5.86s phone P50) and acceptable latency for long-term recall (14.87s phone P50). The hierarchical design enables both immediate assistance and retrospective reasoning within a unified backend.
- **Comparison**: LightMem-Ego fills a gap in the landscape of AI assistants – most existing systems focus on either text-based conversation memory, real-time perception, or agentic task execution, but not on long-horizon multimodal experience memory with explicit event organization and routine discovery.

## Conclusion

The paper introduces LightMem-Ego, a streaming multimodal memory system for everyday-life assistance. It connects mobile/wearable capture with hierarchical memory construction and memory-grounded QA, supporting object finding, conversation recall, life summarization, and routine discovery. The evaluation suggests that explicit long-horizon multimodal memory is a promising direction for personal AI assistants.

**Limitations** acknowledged: reliance on upstream API calls (ASR, VLM, LLM) making accuracy and latency sensitive to external services; error propagation from transcripts/descriptions; non-trivial overhead for continuous segmentation, embedding, indexing; and a preliminary memory update mechanism lacking principled revision, merging, forgetting, or promotion policies.

**Future work** will focus on improving efficiency, robustness, and adaptive memory lifecycle management, as well as integrating privacy protection (on-device preprocessing, selective capture, sensitive-content filtering, encrypted storage, user-controlled editing/deletion, privacy-aware consolidation).

---

_Markdown view of https://picx.dev/p/QCAwQB, served by PicX — AI-generated visual whiteboard summaries of research papers._