PEARL: Personalized Streaming Video Understanding Model

Summary (Overview)

• New Task Definition: The paper proposes and formally defines Personalized Streaming Video Understanding (PSVU) — a novel task requiring models to recognize user-defined concepts in real-time within continuous video streams, supporting multi-turn interactions with dynamically defined concepts.

• New Benchmark (PEARL-Bench): The first comprehensive benchmark for PSVU, comprising 132 unique videos and 2,173 fine-grained timestamped annotations, covering both Frame-level (person/object recognition) and Video-level (personalized action recognition) personalization modes.

• Novel Framework (PEARL): A training-free, plug-and-play framework featuring a Dual-grained Memory System (decoupling concept-centric knowledge from stream-centric observations) and a Concept-aware Retrieval Algorithm for precise historical evidence retrieval.

• State-of-the-Art Results: PEARL achieves consistent improvements across 3 distinct VLM architectures (LLaVA-OV-7B, Qwen2-VL-7B, Qwen3-VL-8B), with average gains of 13.79% at frame-level and 12.80% at video-level, outperforming 8 offline and online baselines.

• Efficiency: PEARL maintains real-time responsiveness with minimal latency overhead (retrieval and rewriting modules add only ~5-155ms), demonstrating practical viability for streaming AI assistants.

---

Introduction and Theoretical Foundation

Background and Motivation

Human cognition of new concepts is inherently streaming: we continuously recognize new objects/identities and update memories over time. However, current multimodal personalization methods are limited to:

• Static images: Yo'LLaVA, MC-LLaVA (image-text tasks only)
• Offline videos: PVChat (single-turn, offline, pre-recorded)

This disconnect between continuous visual input and instant real-world feedback limits the development of real-time, interactive personalized AI assistants for applications like:

• Wearable AI for social interactions
• Personalized robotics
• Customized fitness coaching (e.g., monitoring specific weightlifting actions for instant form correction)

Key Limitations of Existing Methods

Theoretical Foundation

The PSVU task is formally defined as follows. Given a streaming video $V = [X_1, X_2, \ldots]$ where each $X_i$ denotes a semantic scene clip, and a dynamically evolving set of user-defined concepts $C = \{C_1, C_2, \ldots\}$, for a query $Q$ at time $t_q$:

$$A = M(C_{sub}, V_{context}, Q) \tag{1}$$

where $C_{sub} \subseteq C$ is the query-relevant concept subset and $V_{context}$ is the necessary visual context. Solving this requires overcoming two challenges:

1. Prohibitive cost of maintaining unbounded stream history alongside evolving concepts
2. Accurate retrieval of personalized $C_{sub}$ and $V_{context}$ in real-time

---

Methodology

1. PEARL-Bench Curation Pipeline

The benchmark construction follows a four-stage pipeline:

Stage 1: Video Collection and Filtering

• Frame-level split: Videos from anime, movies, reality shows (high dynamics, repeated definable concepts, ≥480p resolution)
• Video-level split: Digital human synthesis using Mixamo assets — 8 characters × 20 actions × 20 backgrounds for diversity

Stage 2: Concept-Definition QA Annotation

• Registers new concepts at specific timestamps
• Uses 10,000 common names from U.S. SSA database to randomly replace original concept names (preventing prior-knowledge bias)

Stage 3: Real-Time & Past-Time QA Annotation

• Real-Time QA: Current-scene questions with VLM-generated distractors
• Past-Time QA: Requires historical evidence clip retrieval (current clip insufficient)

Stage 4: Quality Control

• Automated ablation-based filtering (removes questions answerable without concepts/evidence)
• Human verification by 10 researchers (6 annotators + 4 reviewers)

2. PEARL Framework Architecture

Dual-grained Memory System

Streaming Memory: Maintains entries of $(X_i, e_i)$ where $e_i = f_{emb}(X_i)$ is a multimodal embedding. Scene boundaries are detected via PySceneDetect (threshold 27.0, clip duration 1.0–8.0s).

Concept Memory: Stores structured entries with three components:

1. Concept name
2. Visual evidence (last frame for frame-level; full clip for video-level)
3. Textual description (generated via standardized prompts focusing on permanent/stable features)

Concept-aware Retrieval Algorithm

Given a user query $Q$ at timestamp $t_q$:

1. Concept Retrieval: Identify concept names in $Q$, retrieve corresponding Concept Memory entries → $C_{sub}$
2. Query Rewriting: Replace concept names with descriptions to form $\tilde{Q}$ (making embeddings matchable)
3. Streaming Memory Retrieval: Compute $e_Q = f_{emb}(\tilde{Q})$, then cosine similarity with all stored clip embeddings $\{e_i\}_{i \leq t_q}$
4. Top-K Selection: Select top-$K$ clips (K=4 default), expand each with $N$ adjacent clips (N=1 frame-level, N=0 video-level)
5. Final Response: Feed retrieved concepts, historical clips, current clip, and original query into VLM

---

Empirical Validation / Results

Main Results (Table 3)

Key findings:

• PEARL adds +8.55% to +23.47% over offline baselines across architectures
• Qwen3-VL-8B+PEARL surpasses Gemini3-pro-preview by >4% (frame-level) and ~24% (video-level)
• LLaVA-OV-7B+PEARL outperforms ReKV (same backbone) by 7.21% (Real-Time) and 5.18% (Past-Time), isolating framework design as the source of gains

Ablation Study (Table 4)

Three key takeaways:

1. Concept Memory is indispensable (+35% Real-Time gain)
2. Streaming Memory is essential for Past-Time QA (+20% gain)
3. Query Rewriting improves both metrics (+4.28% average)

Efficiency Analysis (Table 5)

LLaVA-OV-7B+PEARL is faster than all online baselines while achieving higher accuracy. Latency breakdown shows PEARL's core modules add only ~5-155ms; LLM inference is the dominant bottleneck.

Model Scale Effects (Table 7)

Finding: Offline models show no scaling benefits (paradigm mismatch), but with PEARL, larger models significantly outperform smaller ones.

---

Theoretical and Practical Implications

Theoretical Contributions

1. New Task Paradigm: PSVU bridges the gap between continuous visual input and instant real-world feedback, formalizing what previous personalization methods (static image or offline video) could not address.

2. Memory Architecture Design: The Dual-grained Memory System demonstrates that explicitly decoupling concept-centric knowledge from stream-centric observations is more effective than compressing history into fixed-size states (as online models do).

3. Retrieval Strategy: The Concept-aware Retrieval Algorithm shows that query rewriting (replacing personalized names with descriptive semantics) is crucial for making embedding-based retrieval effective for unseen user-defined concepts.

Practical Implications

• Real-time AI Assistants: PEARL enables practical deployment of personalized assistants in dynamic environments (fitness coaching, social wearables, robotics) without retraining.
• Plug-and-play Compatibility: Works with any off-the-shelf VLM, requiring only a multimodal embedding model and scene detector.
• Efficiency: Retrieval modules add negligible latency (~5-155ms), making real-time interaction feasible.
• Benchmark Utility: PEARL-Bench provides a rigorous evaluation standard with human upper bounds (97%+) and text-only lower bounds (14%), confirming the task requires genuine visual grounding.

---

Conclusion

Main Takeaways

1. PSVU is a novel, challenging task requiring models to handle streaming video, dynamically defined concepts, and multi-turn interactions — capabilities absent in all prior personalization methods.

2. PEARL-Bench is the first comprehensive benchmark for this setting, with 132 videos, 2,173 timestamped annotations, and strict quality control (automated filtering + human verification).

3. PEARL framework achieves state-of-the-art performance across 8 baselines and 3 architectures via training-free, plug-and-play design, with average gains of 13.79% (frame-level) and 12.80% (video-level).

4. The paradigm mismatch of offline models is highlighted: increasing model scale without a PSVU-specific framework yields no gains, while PEARL unlocks scaling benefits.

Future Directions

• Extending PEARL to handle larger-scale concepts and longer-term memory
• Exploring reinforcement learning approaches for personalized streaming understanding
• Investigating multi-modal memory compression for unbounded stream histories
• Applying PSVU to real-world deployment scenarios (wearable AI, robotics, fitness coaching)

---

Code and benchmark are publicly available at: https://github.com/Yuanhong-Zheng/PEARL