# StreamEval: Evaluating Interactive Streaming Video Generation Models

> StreamEval benchmark of 37 models shows request execution lags far behind visual quality, with best edit success only 15 out of 100.

- **Source:** [arXiv](https://arxiv.org/abs/2610.28402)
- **Published:** 2026-10-23
- **Permalink:** https://picx.dev/p/wnTpHj
- **Whiteboard:** https://picx.dev/p/wnTpHj/image

## Summary

## Summary (Overview)

- **StreamEval** is a unified benchmark for interactive streaming video generation, spanning five tasks: digital-human dialogue, avatar performance, world generation, instruction editing, and reference editing.
- The benchmark comprises **300 scripted test cases** (15–300 seconds each) with incrementally arriving multimodal requests and explicit requirements for state changes and content preservation.
- Performance is assessed along **four evaluation dimensions** (task execution, state consistency, visual fidelity, audiovisual coordination) instantiated through **ten task-specific metrics** on a 0–100 scale.
- The study evaluates **37 distinct models** across 300 test cases, revealing that high visual quality and identity retention do not necessarily coincide with reliable request execution or sustained state consistency.
- **Request execution remains the central challenge**: the highest Following scores are 36.90 (avatar performance), 25.93 (world generation), and 15.00 Edit Success (reference editing) on a 0–100 scale.

## Introduction and Theoretical Foundation

Real-time interactive video generation requires systems that accept new user inputs while a session is in progress, supporting conversational avatars, controllable virtual worlds, and live video editing. Beyond generating plausible frames, systems must maintain alignment with incoming inputs as user requests and context evolve over time.

**Core requirements for sustained interaction:**
- Execute new requests while preserving relevant prior content and state
- Respect timing and scope requirements (when a change occurs, what it modifies, what remains intact)
- Maintain conversational context and identity (avatars), unaffected scene content (worlds), and source tracking (editors)

The authors position StreamEval relative to existing benchmarks:

> "VBench and EvalCrafter assess visual quality and temporal behavior, while T2V-CompBench examines compositional alignment. WBench and WBench-Editing evaluate multi-turn control, successive edits, and content preservation within generated worlds. StreamAV-Bench assesses interactive updates and state retention in streaming audio-video generation, and LongV2VBench examines editing quality over sustained source streams."

StreamEval uniquely unifies avatar interaction, world generation, and real-time video editing into a single framework with **explicit, task-specific execution and preservation criteria**.

## Methodology

### Benchmark Construction

1. **Test case structure**: Each case bundles initialization assets (portrait, scene, or source video) with a sequence of timed user requests and explicit criteria for change and preservation.

2. **Authoring pipeline**: 
   - Qwen3.8-Max drafts session goals, timed controls, protected attributes, and expected outcomes
   - An authored oracle specifies what to establish, retain, cancel, or restore at each checkpoint
   - Media inputs are screened and revised against requirements

3. **Data statistics**:
   - 60 test cases per task, balanced across 15, 30, 60, 120, and 300 seconds (12 cases per duration)
   - 1,347 planned online events across 300 cases
   - 240 independent source families, with 60 text-dialogue derivatives
   - 525 minutes of active streaming generation per full pass

### Evaluation Protocol

**Four dimensions, ten metrics:**

| Dimension | Metrics |
|------------|---------|
| Task Execution | Dialogue, Following, Edit Success |
| State Consistency | Stability, Continuity, Preservation |
| Visual Fidelity | Quality, Identity, Reference |
| Audiovisual Coordination | A–V Sync |

**Score aggregation** (equation from paper):

$$
S_{m} = \frac{1}{|\mathcal{D}|} \sum_{d \in \mathcal{D}} \frac{1}{|\mathcal{C}_{d,m}|} \sum_{c \in \mathcal{C}_{d,m}} \frac{1}{3} \sum_{r=1}^{3} s_{c,r,m} \tag{1}
$$

**Quality-Overall** (equal-weight mean of applicable metrics):

$$
\mathrm{QOverall}_{t,k} = \frac{1}{|\mathcal{M}_{t,k}|} \sum_{m \in \mathcal{M}_{t,k}} S_{t,k,m} \tag{2}
$$

**Sweet-spot bonus** for avatar customization:

$$
\mathrm{Overall}_{t,k} = \mathrm{QOverall}_{t,k} + B_{t,k}, \qquad B_{t,k} = \left\{ \begin{array}{ll} 5, & \text{avatar task with custom-avatar support}, \\ 0, & \text{otherwise}. \end{array} \right. \tag{3}
$$

**Key metric definitions:**
- **Quality**: 5-point defect scale ($r \in \{5,4,3,2,1\}$), scored as $25(r-1)$
- **Stability**: Duration covered by clips with no severe temporal defect, ≤1 minor temporal defect, and Quality ≥3
- **A–V Sync**: Percentage of speech events with lip-opening/audio-envelope correlation ≥0.3
- **Identity**: Lower rating from two judges (Qwen3.8-Max-0902 and Qwen3-VL-Plus) with a 5-point feature-change scale

### Quality Verification

- Automated checks for media decodability, duration bounds, and event agreement
- Qwen3-ASR transcribes speech (WER ≤ 0.25 acceptance threshold)
- Judge calibration targets: ≤5% false positive, ≤10% false negative, ≤10% unresolved decisions

## Empirical Validation / Results

### Digital-Human Dialogue (Table 2 highlights)

| Model | Quality | A–V Sync | Stability | Dialogue | Identity | Q-Overall | Overall |
|--------|---------|----------|-----------|----------|----------|-----------|---------|
| **Vidu S2 Avatar** | 96.78 | 84.87 | 97.83 | **75.29** | **90.12** | 88.98 | **93.98** |
| Tavus CVI Full | 75.04 | **94.44** | 99.95 | 73.91 | 89.89 | 86.65 | 86.65 |
| AVTR-1 | 82.97 | 69.38 | 99.87 | 54.97 | 88.54 | 79.14 | 84.14 |
| MuseTalk 1.5 | 95.34 | 38.09 | 97.11 | 54.34 | 88.44 | 74.66 | 79.66 |

**Key finding**: MuseTalk 1.5 attains 95.34 Quality and 97.11 Stability yet only 38.09 A–V Sync and 54.34 Dialogue, demonstrating that visual quality does not imply conversational competence.

### Avatar Performance (Table 3)

| Model | Quality | A–V Sync | Stability | Following | Identity | Overall |
|--------|---------|----------|-----------|-----------|----------|---------|
| **Vidu S2 Avatar** | 63.68 | **99.60** | **97.25** | **36.90** | **86.12** | **81.71** |
| LiveTalk 1.3B | **71.27** | 86.44 | 96.06 | 16.59 | 83.89 | 75.85 |
| AptAvatar 14B | 55.55 | 98.89 | 78.06 | 13.04 | 81.33 | 70.37 |

Following scores span only 6.37–36.90 even for systems exceeding 80 in Identity, confirming **action execution under timing constraints as the primary challenge**.

### World Generation (Table 4 highlights)

| Model | Quality | Stability | Following | Continuity | Overall |
|--------|---------|-----------|-----------|------------|---------|
| **MemFlow** | 40.93 | 40.38 | 22.59 | 29.54 | **33.36** |
| Rolling Forcing | 45.19 | 44.47 | 17.78 | 24.61 | 33.01 |
| Krea Realtime | 35.61 | 42.79 | **25.93** | 22.12 | 31.61 |
| HappyOyster 1.0 | **58.34** | **56.67** | 5.56 | 2.78 | 30.84 |

HappyOyster leads in visual quality (58.34) but has very low control performance (5.56 Following), showing model strengths diverge sharply across dimensions.

### Instruction Editing (Table 5)

| Model | Quality | Stability | Following | Edit Success | Preservation | Overall |
|--------|---------|-----------|-----------|--------------|--------------|---------|
| **Vidu S2 Editing** | **22.99** | **13.94** | **72.10** | **57.20** | 60.93 | **45.43** |
| JoyAI-Video-Edit | 11.63 | 9.57 | 63.89 | 43.52 | **83.33** | 42.39 |
| SANA-Streaming | 15.15 | 9.22 | 11.48 | 4.63 | 44.26 | 16.95 |

Preservation (83.33 for JoyAI) versus Following (72.10 for Vidu S2 Editing) reveals **trade-offs between source retention and edit execution**.

### Reference Editing (Table 6)

| Model | Quality | Stability | Reference | Edit Success | Preservation | Overall |
|--------|---------|-----------|-----------|--------------|--------------|---------|
| **Vidu S2 Editing** | **37.72** | **46.74** | **31.20** | **15.00** | **64.44** | **39.02** |
| JoyAI-Video-Edit | 6.54 | 3.46 | 26.11 | 8.89 | 26.67 | 14.33 |
| XMax X2.0 | 8.20 | 7.51 | 20.56 | 1.11 | 10.00 | 9.47 |

Even the best-performing model achieves only 15.00 Edit Success, with a recorded session showing systems retain the original grey cape instead of adopting the requested gold satin appearance when references change mid-stream.

## Theoretical and Practical Implications

1. **Decoupling of capabilities**: The benchmark demonstrates that visual quality, identity retention, and request execution are **largely independent capabilities**. Models excelling at one dimension often perform poorly on others.

2. **Timing and preservation constraints**: Reliable request execution requires models to handle *when* changes occur, *what* should change, and *what* must remain intact—a multidimensional challenge current systems do not yet solve.

3. **Unified evaluation framework**: StreamEval provides a common protocol across previously disparate domains, enabling cross-task comparison and identifying which fundamental capabilities transfer across application settings.

4. **Benchmark coverage gaps**: The evaluation reveals that even state-of-the-art systems struggle with:
   - Sustained instruction following under timing constraints
   - Reference switching without losing source fidelity
   - Full-edit completion (including restoration to source)

## Conclusion

StreamEval establishes a unified benchmark for interactive streaming video generation with 300 scripted test cases and ten metrics across four dimensions. The evaluation of 37 models reveals that:

> "Online action following and reference-based editing remain challenging. In reference editing, the highest reported Edit Success score is 15.00 on a 0–100 scale."

**Future directions identified by the authors:**
- Improving request execution alongside visual fidelity and state continuity
- Collecting human pairwise judgments to examine agreement with the task-specific evaluation rubric
- Extending coverage to broader identities, environments, global resets, and selective restoration with concurrent active edits

**Availability**: Leaderboard at https://streameval.ai; code and benchmark data at https://github.com/StreamEval/StreamEval and https://huggingface.co/datasets/StreamEval/StreamEval.

---

_Markdown view of https://picx.dev/p/wnTpHj, served by PicX — AI-generated visual whiteboard summaries of research papers._