# PEARL: Personalized Streaming Video Understanding Model

> PEARL introduces a training-free framework for personalized streaming video understanding, achieving 13.79% frame-level and 12.80% video-level accuracy gains across three VLM architectures.

- **Source:** [arXiv](https://arxiv.org/abs/2603.20422)
- **Published:** 2026-03-26
- **Permalink:** https://picx.dev/p/LNL2gG

## Summary

# PEARL: Personalized Streaming Video Understanding Model

## Summary (Overview)

- **New Task Definition**: The paper proposes and formally defines **Personalized Streaming Video Understanding (PSVU)** — a novel task requiring models to recognize user-defined concepts in real-time within continuous video streams, supporting multi-turn interactions with dynamically defined concepts.

- **New Benchmark (PEARL-Bench)**: The first comprehensive benchmark for PSVU, comprising **132 unique videos** and **2,173 fine-grained timestamped annotations**, covering both Frame-level (person/object recognition) and Video-level (personalized action recognition) personalization modes.

- **Novel Framework (PEARL)**: A **training-free, plug-and-play** framework featuring a Dual-grained Memory System (decoupling concept-centric knowledge from stream-centric observations) and a Concept-aware Retrieval Algorithm for precise historical evidence retrieval.

- **State-of-the-Art Results**: PEARL achieves consistent improvements across **3 distinct VLM architectures** (LLaVA-OV-7B, Qwen2-VL-7B, Qwen3-VL-8B), with average gains of **13.79%** at frame-level and **12.80%** at video-level, outperforming 8 offline and online baselines.

- **Efficiency**: PEARL maintains real-time responsiveness with minimal latency overhead (retrieval and rewriting modules add only ~5-155ms), demonstrating practical viability for streaming AI assistants.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Human cognition of new concepts is inherently **streaming**: we continuously recognize new objects/identities and update memories over time. However, current multimodal personalization methods are limited to:

- **Static images**: Yo'LLaVA, MC-LLaVA (image-text tasks only)
- **Offline videos**: PVChat (single-turn, offline, pre-recorded)

This disconnect between continuous visual input and instant real-world feedback limits the development of **real-time, interactive personalized AI assistants** for applications like:
- Wearable AI for social interactions
- Personalized robotics
- Customized fitness coaching (e.g., monitoring specific weightlifting actions for instant form correction)

### Key Limitations of Existing Methods

| Method | Modality | Streaming | Multi-turn | Video-level Concepts |
|--------|----------|-----------|------------|---------------------|
| MyVLM, Yo'LLaVA, MC-LLaVA | Image | ✗ | ✗ | ✗ |
| UnifyBench, MMPB | Image | ✗ | ✗ | ✗ |
| PVChat, This-is-My | Short Video | ✗ | ✗ | ✗ |
| **PEARL-Bench** | **Long Video** | **✓** | **✓** | **✓** |

### Theoretical Foundation

The PSVU task is formally defined as follows. Given a streaming video $V = [X_1, X_2, \ldots]$ where each $X_i$ denotes a semantic scene clip, and a dynamically evolving set of user-defined concepts $C = \{C_1, C_2, \ldots\}$, for a query $Q$ at time $t_q$:

$$A = M(C_{sub}, V_{context}, Q) \tag{1}$$

where $C_{sub} \subseteq C$ is the query-relevant concept subset and $V_{context}$ is the necessary visual context. Solving this requires overcoming two challenges:
1. **Prohibitive cost** of maintaining unbounded stream history alongside evolving concepts
2. **Accurate retrieval** of personalized $C_{sub}$ and $V_{context}$ in real-time

---

## Methodology

### 1. PEARL-Bench Curation Pipeline

The benchmark construction follows a **four-stage pipeline**:

**Stage 1: Video Collection and Filtering**
- Frame-level split: Videos from anime, movies, reality shows (high dynamics, repeated definable concepts, ≥480p resolution)
- Video-level split: Digital human synthesis using **Mixamo assets** — 8 characters × 20 actions × 20 backgrounds for diversity

**Stage 2: Concept-Definition QA Annotation**
- Registers new concepts at specific timestamps
- Uses **10,000 common names** from U.S. SSA database to randomly replace original concept names (preventing prior-knowledge bias)

**Stage 3: Real-Time & Past-Time QA Annotation**
- Real-Time QA: Current-scene questions with VLM-generated distractors
- Past-Time QA: Requires historical evidence clip retrieval (current clip insufficient)

**Stage 4: Quality Control**
- Automated ablation-based filtering (removes questions answerable without concepts/evidence)
- Human verification by 10 researchers (6 annotators + 4 reviewers)

### 2. PEARL Framework Architecture

#### Dual-grained Memory System

**Streaming Memory**: Maintains entries of $(X_i, e_i)$ where $e_i = f_{emb}(X_i)$ is a multimodal embedding. Scene boundaries are detected via PySceneDetect (threshold 27.0, clip duration 1.0–8.0s).

**Concept Memory**: Stores structured entries with three components:
1. Concept name
2. Visual evidence (last frame for frame-level; full clip for video-level)
3. Textual description (generated via standardized prompts focusing on **permanent/stable features**)

#### Concept-aware Retrieval Algorithm

Given a user query $Q$ at timestamp $t_q$:

1. **Concept Retrieval**: Identify concept names in $Q$, retrieve corresponding Concept Memory entries → $C_{sub}$
2. **Query Rewriting**: Replace concept names with descriptions to form $\tilde{Q}$ (making embeddings matchable)
3. **Streaming Memory Retrieval**: Compute $e_Q = f_{emb}(\tilde{Q})$, then cosine similarity with all stored clip embeddings $\{e_i\}_{i \leq t_q}$
4. **Top-K Selection**: Select top-$K$ clips (K=4 default), expand each with $N$ adjacent clips (N=1 frame-level, N=0 video-level)
5. **Final Response**: Feed retrieved concepts, historical clips, current clip, and original query into VLM

---

## Empirical Validation / Results

### Main Results (Table 3)

| Method | #Frames | Frame-level RT | Frame-level PT | Frame-level Avg | Video-level RT |
|--------|---------|----------------|----------------|-----------------|----------------|
| Human Score | - | 97.61 | 96.45 | 97.03 | 97.49 |
| Text-only (Qwen3-VL-8B) | - | 11.06 | 17.45 | 14.26 | 7.04 |
| Gemini3-pro-preview | 64 | 47.40 | 48.98 | 48.19 | 24.51 |
| LLaVA-OV-7B | 64 | 24.95 | 34.01 | 29.48 | 10.86 |
| Qwen2-VL-7B | 64 | 23.21 | 35.79 | 29.50 | 17.89 |
| InternVL3.5-8B | 64 | 30.35 | 35.06 | 32.71 | 5.57 |
| Qwen3-VL-8B | 64 | 27.33 | 30.20 | 28.77 | 25.51 |
| ReKV (LLaVA-OV-7B) | 0.5fps | 26.20 | 37.46 | 31.83 | 24.11 |
| StreamForest-7B | 1fps | 29.18 | 40.86 | 35.02 | 10.85 |
| TimeChat-Online-7B | 1fps | 31.89 | 35.28 | 33.59 | 22.29 |
| **LLaVA-OV-7B+PEARL** | 1fps | **33.41** ↑8.46 | **42.64** ↑8.63 | **38.03** ↑8.55 | **19.94** ↑9.08 |
| **Qwen2-VL-7B+PEARL** | 1fps | **33.30** ↑10.09 | **44.42** ↑8.63 | **38.86** ↑9.36 | **24.34** ↑6.45 |
| **Qwen3-VL-8B+PEARL** | 1fps | **54.99** ↑27.66 | **49.49** ↑19.29 | **52.24** ↑23.47 | **48.39** ↑22.88 |

**Key findings**:
- PEARL adds **+8.55% to +23.47%** over offline baselines across architectures
- **Qwen3-VL-8B+PEARL surpasses Gemini3-pro-preview by >4%** (frame-level) and ~24% (video-level)
- **LLaVA-OV-7B+PEARL** outperforms ReKV (same backbone) by 7.21% (Real-Time) and 5.18% (Past-Time), isolating framework design as the source of gains

### Ablation Study (Table 4)

| Text | Current Clip | Concept Memory | Streaming Memory | Rewrite | Real-Time | Past-Time | Avg |
|------|-------------|----------------|------------------|---------|-----------|-----------|-----|
| ✓ | ✗ | ✗ | ✗ | ✗ | 11.06 | 17.45 | 14.26 |
| ✓ | ✓ | ✗ | ✗ | ✗ | 15.84 | 20.30 | 18.07 |
| ✓ | ✓ | ✓ | ✗ | ✗ | 51.41 | 25.43 | 38.42 |
| ✓ | ✓ | ✓ | ✓ | ✗ | 50.22 | 45.69 | 47.96 |
| ✓ | ✓ | ✓ | ✓ | ✓ | 54.99 | 49.49 | 52.24 |

**Three key takeaways**:
1. **Concept Memory is indispensable** (+35% Real-Time gain)
2. **Streaming Memory is essential for Past-Time QA** (+20% gain)
3. **Query Rewriting improves both metrics** (+4.28% average)

### Efficiency Analysis (Table 5)

| Method | #Frames | F-Avg | Latency (ms) |
|--------|---------|-------|--------------|
| LLaVA-OV-7B | 64 | 29.48 | 670 |
| Qwen3-VL-8B | 64 | 28.77 | 1,594 |
| ReKV (LLaVA-OV-7B) | 0.5fps | 31.83 | 1,818 |
| StreamForest-7B | 1fps | 35.02 | 1,164 |
| TimeChat-Online-7B | 1fps | 33.59 | 4,769 |
| **LLaVA-OV-7B+PEARL** | 1fps | **38.03** | **775** |
| **Qwen3-VL-8B+PEARL** | 1fps | **52.24** | **2,111** |

LLaVA-OV-7B+PEARL is **faster than all online baselines** while achieving higher accuracy. Latency breakdown shows PEARL's core modules add only ~5-155ms; LLM inference is the dominant bottleneck.

### Model Scale Effects (Table 7)

| Model | Real-Time | Past-Time | Avg |
|--------|-----------|-----------|-----|
| Qwen2-VL-2B | 31.24 | 22.84 | 27.04 |
| Qwen2-VL-2B+PEARL | 29.93 ↓1.31 | 32.49 ↑9.65 | 31.21 ↑4.17 |
| Qwen3-VL-4B | 24.08 | 30.96 | 27.52 |
| Qwen3-VL-4B+PEARL | 40.78 ↑16.70 | 50.25 ↑19.29 | 45.52 ↑18.00 |

**Finding**: Offline models show no scaling benefits (paradigm mismatch), but with PEARL, larger models significantly outperform smaller ones.

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **New Task Paradigm**: PSVU bridges the gap between continuous visual input and instant real-world feedback, formalizing what previous personalization methods (static image or offline video) could not address.

2. **Memory Architecture Design**: The Dual-grained Memory System demonstrates that **explicitly decoupling concept-centric knowledge from stream-centric observations** is more effective than compressing history into fixed-size states (as online models do).

3. **Retrieval Strategy**: The Concept-aware Retrieval Algorithm shows that **query rewriting** (replacing personalized names with descriptive semantics) is crucial for making embedding-based retrieval effective for unseen user-defined concepts.

### Practical Implications

- **Real-time AI Assistants**: PEARL enables practical deployment of personalized assistants in dynamic environments (fitness coaching, social wearables, robotics) without retraining.
- **Plug-and-play Compatibility**: Works with any off-the-shelf VLM, requiring only a multimodal embedding model and scene detector.
- **Efficiency**: Retrieval modules add negligible latency (~5-155ms), making real-time interaction feasible.
- **Benchmark Utility**: PEARL-Bench provides a rigorous evaluation standard with human upper bounds (97%+) and text-only lower bounds (14%), confirming the task requires genuine visual grounding.

---

## Conclusion

### Main Takeaways

1. **PSVU is a novel, challenging task** requiring models to handle streaming video, dynamically defined concepts, and multi-turn interactions — capabilities absent in all prior personalization methods.

2. **PEARL-Bench** is the first comprehensive benchmark for this setting, with 132 videos, 2,173 timestamped annotations, and strict quality control (automated filtering + human verification).

3. **PEARL framework** achieves state-of-the-art performance across 8 baselines and 3 architectures via training-free, plug-and-play design, with average gains of 13.79% (frame-level) and 12.80% (video-level).

4. **The paradigm mismatch** of offline models is highlighted: increasing model scale without a PSVU-specific framework yields no gains, while PEARL unlocks scaling benefits.

### Future Directions

- Extending PEARL to handle **larger-scale concepts** and longer-term memory
- Exploring **reinforcement learning** approaches for personalized streaming understanding
- Investigating **multi-modal memory compression** for unbounded stream histories
- Applying PSVU to **real-world deployment** scenarios (wearable AI, robotics, fitness coaching)

---

*Code and benchmark are publicly available at: https://github.com/Yuanhong-Zheng/PEARL*

---

_Markdown view of https://picx.dev/p/LNL2gG, served by PicX — AI-generated visual whiteboard summaries of research papers._