Summary (Overview)

• StreamEval is a unified benchmark for interactive streaming video generation, spanning five tasks: digital-human dialogue, avatar performance, world generation, instruction editing, and reference editing.
• The benchmark comprises 300 scripted test cases (15–300 seconds each) with incrementally arriving multimodal requests and explicit requirements for state changes and content preservation.
• Performance is assessed along four evaluation dimensions (task execution, state consistency, visual fidelity, audiovisual coordination) instantiated through ten task-specific metrics on a 0–100 scale.
• The study evaluates 37 distinct models across 300 test cases, revealing that high visual quality and identity retention do not necessarily coincide with reliable request execution or sustained state consistency.
• Request execution remains the central challenge: the highest Following scores are 36.90 (avatar performance), 25.93 (world generation), and 15.00 Edit Success (reference editing) on a 0–100 scale.

Introduction and Theoretical Foundation

Real-time interactive video generation requires systems that accept new user inputs while a session is in progress, supporting conversational avatars, controllable virtual worlds, and live video editing. Beyond generating plausible frames, systems must maintain alignment with incoming inputs as user requests and context evolve over time.

Core requirements for sustained interaction:

• Execute new requests while preserving relevant prior content and state
• Respect timing and scope requirements (when a change occurs, what it modifies, what remains intact)
• Maintain conversational context and identity (avatars), unaffected scene content (worlds), and source tracking (editors)

The authors position StreamEval relative to existing benchmarks:

"VBench and EvalCrafter assess visual quality and temporal behavior, while T2V-CompBench examines compositional alignment. WBench and WBench-Editing evaluate multi-turn control, successive edits, and content preservation within generated worlds. StreamAV-Bench assesses interactive updates and state retention in streaming audio-video generation, and LongV2VBench examines editing quality over sustained source streams."

StreamEval uniquely unifies avatar interaction, world generation, and real-time video editing into a single framework with explicit, task-specific execution and preservation criteria.

Methodology

Benchmark Construction

1. Test case structure: Each case bundles initialization assets (portrait, scene, or source video) with a sequence of timed user requests and explicit criteria for change and preservation.

2. Authoring pipeline:

   • Qwen3.8-Max drafts session goals, timed controls, protected attributes, and expected outcomes
   • An authored oracle specifies what to establish, retain, cancel, or restore at each checkpoint
   • Media inputs are screened and revised against requirements

3. Data statistics:

   • 60 test cases per task, balanced across 15, 30, 60, 120, and 300 seconds (12 cases per duration)
   • 1,347 planned online events across 300 cases
   • 240 independent source families, with 60 text-dialogue derivatives
   • 525 minutes of active streaming generation per full pass

Evaluation Protocol

Four dimensions, ten metrics:

Score aggregation (equation from paper):

$$S_{m} = \frac{1}{|\mathcal{D}|} \sum_{d \in \mathcal{D}} \frac{1}{|\mathcal{C}_{d,m}|} \sum_{c \in \mathcal{C}_{d,m}} \frac{1}{3} \sum_{r=1}^{3} s_{c,r,m} \tag{1}$$

Quality-Overall (equal-weight mean of applicable metrics):

$$\mathrm{QOverall}_{t,k} = \frac{1}{|\mathcal{M}_{t,k}|} \sum_{m \in \mathcal{M}_{t,k}} S_{t,k,m} \tag{2}$$

Sweet-spot bonus for avatar customization:

$$\mathrm{Overall}_{t,k} = \mathrm{QOverall}_{t,k} + B_{t,k}, \qquad B_{t,k} = \left\{ \begin{array}{ll} 5, & \text{avatar task with custom-avatar support}, \\ 0, & \text{otherwise}. \end{array} \right. \tag{3}$$

Key metric definitions:

• Quality: 5-point defect scale ($r \in \{5,4,3,2,1\}$), scored as $25(r-1)$
• Stability: Duration covered by clips with no severe temporal defect, ≤1 minor temporal defect, and Quality ≥3
• A–V Sync: Percentage of speech events with lip-opening/audio-envelope correlation ≥0.3
• Identity: Lower rating from two judges (Qwen3.8-Max-0902 and Qwen3-VL-Plus) with a 5-point feature-change scale

Quality Verification

• Automated checks for media decodability, duration bounds, and event agreement
• Qwen3-ASR transcribes speech (WER ≤ 0.25 acceptance threshold)
• Judge calibration targets: ≤5% false positive, ≤10% false negative, ≤10% unresolved decisions

Empirical Validation / Results

Digital-Human Dialogue (Table 2 highlights)

Key finding: MuseTalk 1.5 attains 95.34 Quality and 97.11 Stability yet only 38.09 A–V Sync and 54.34 Dialogue, demonstrating that visual quality does not imply conversational competence.

Avatar Performance (Table 3)

Following scores span only 6.37–36.90 even for systems exceeding 80 in Identity, confirming action execution under timing constraints as the primary challenge.

World Generation (Table 4 highlights)

HappyOyster leads in visual quality (58.34) but has very low control performance (5.56 Following), showing model strengths diverge sharply across dimensions.

Instruction Editing (Table 5)

Preservation (83.33 for JoyAI) versus Following (72.10 for Vidu S2 Editing) reveals trade-offs between source retention and edit execution.

Reference Editing (Table 6)

Even the best-performing model achieves only 15.00 Edit Success, with a recorded session showing systems retain the original grey cape instead of adopting the requested gold satin appearance when references change mid-stream.

Theoretical and Practical Implications

1. Decoupling of capabilities: The benchmark demonstrates that visual quality, identity retention, and request execution are largely independent capabilities. Models excelling at one dimension often perform poorly on others.

2. Timing and preservation constraints: Reliable request execution requires models to handle when changes occur, what should change, and what must remain intact—a multidimensional challenge current systems do not yet solve.

3. Unified evaluation framework: StreamEval provides a common protocol across previously disparate domains, enabling cross-task comparison and identifying which fundamental capabilities transfer across application settings.

4. Benchmark coverage gaps: The evaluation reveals that even state-of-the-art systems struggle with:

   • Sustained instruction following under timing constraints
   • Reference switching without losing source fidelity
   • Full-edit completion (including restoration to source)

Conclusion

StreamEval establishes a unified benchmark for interactive streaming video generation with 300 scripted test cases and ten metrics across four dimensions. The evaluation of 37 models reveals that:

"Online action following and reference-based editing remain challenging. In reference editing, the highest reported Edit Success score is 15.00 on a 0–100 scale."

Future directions identified by the authors:

• Improving request execution alongside visual fidelity and state continuity
• Collecting human pairwise judgments to examine agreement with the task-specific evaluation rubric
• Extending coverage to broader identities, environments, global resets, and selective restoration with concurrent active edits

Availability: Leaderboard at https://streameval.ai; code and benchmark data at https://github.com/StreamEval/StreamEval and https://huggingface.co/datasets/StreamEval/StreamEval.