# SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation > Semantic task completion video generation remains unsolved: the best model achieves only 37.8% outcome achievement, revealing critical gaps in reference-grounded video generation. - **Source:** [arXiv](https://arxiv.org/abs/2608.17426) - **Published:** 2026-08-21 - **Permalink:** https://picx.dev/p/qkfB3H - **Whiteboard:** https://picx.dev/p/qkfB3H/image ## Summary ## Summary This paper introduces **Semantic Task Completion Video Generation**, a new task for video generation models requiring them to achieve a specified outcome while maintaining semantic grounding to a reference image. The authors construct **SemComp-Data**, a large-scale evaluation dataset, and **SemComp-Bench**, a VLM-based evaluation protocol, to systematically assess model capabilities. ### Summary (Overview) - Proposes a novel task: generating videos that realize an instructed outcome (e.g., "fold a banknote into a turtle") while preserving task-relevant semantic attributes from a reference image, without requiring intermediate process steps or pixel-level appearance matching. - Builds **SemComp-Data** (1,273 instances) via a scalable four-stage pipeline from Koala-36M videos, with each instance containing a reference frame, paired brief/detailed instructions, and an outcome-centric clip extracted from the same source video. - Introduces **SemComp-Bench**, an evaluation protocol using binary VLM questions across two dimensions: **Outcome Achievement (OA)** and **Generation Reliability (GR)**, with interpretable criterion-level scores. - Evaluates seven state-of-the-art models (e.g., Seedance 2.0, Wan2.2, HunyuanVideo) and finds the best OA Score is only 37.8%, revealing significant limitations in semantic task completion. ### Introduction and Theoretical Foundation - Existing benchmarks focus on visual fidelity, temporal coherence, and prompt adherence but overlook whether videos achieve a task outcome grounded in a reference context. - The task requires preserving task-relevant entities/attributes (e.g., person identity, object appearance) while allowing irrelevant attributes (e.g., background, spatial layout) to change. - SemComp-Data ensures task authenticity by extracting reference and outcome frames from the same real-world video, avoiding synthetic or hallucinated pairings. - SemComp-Bench provides fine-grained diagnosis via criteria such as outcome realization, semantic grounding, entity consistency, and visual continuity. ### Methodology - **Data Curation Pipeline** (Fig. 3): 1. **Candidate Filtering**: Title-based keyword filtering removes narration-dependent videos; video abstracts (mosaics) are used to categorize videos into six domains (e.g., Arts and Precision, Food and Cooking) via a VLM; uncertain videos are discarded. 2. **State Mining**: A VLM performs frame-level timestamp localization for reference and outcome states based on manual state definitions; QA-based quality checking discards ambiguous pairs. 3. **Video Extension**: Shot detection and same-scene merging (Panda-70M method) extract an outcome-centric clip (≥3s, avg 4.03s) anchored at the verified outcome timestamp. 4. **Instruction Structuring**: Generates brief instructions (≤30 words, template-based) and detailed instructions (including alignment type, preserve/discard attributes, and outcome characteristics) via VLM. - **Evaluation Protocol**: - **OA Score** (Eq. 1): Joint pass rate across four criteria—outcome realization, semantic grounding, grounded entity consistency, global visual continuity. - **GR Score** (Eq. 2): Mean of five criteria—physical plausibility, visual clarity, artifact-free rendering, within-scene spatiotemporal coherence, text/interface integrity. - All evaluations use 27 uniformly sampled frames per video, scored via three independent VLM calls (Doubao-Seed-1.8). ### Empirical Validation / Results - **Outcome Achievement** (Table 1): HunyuanVideo-1.5-720P-I2V achieves the highest OA Score (37.8%), followed by Wan2.2-I2V-A14B (28.3%). Seedance 2.0 excels in outcome realization but fails on entity consistency and visual continuity. Wan2.2-TI2V-5B shows complementary strengths (high consistency, low grounding). - **Generation Reliability** (Table 2): Seedance 2.0 leads with a GR Score of 91.8%; Wan2.2-I2V-A14B is best among open-source models (89.0%). Within-scene spatiotemporal coherence is the weakest criterion across all models (0.328–0.739). - **Conditioning Effects** (Table 3): I2V variants outperform T2V variants across all model families under detailed instructions, especially in grounding and consistency. Detailed instructions yield higher OA Scores than brief instructions in T2V settings, but brief instructions improve entity consistency and visual continuity, revealing a specificity–generation difficulty trade-off. - **Visualizations** (Fig. 5): Models often preserve reference appearance and depict task processes but fail to reach the intended outcome; common failures include physically implausible transformations and abrupt visual transitions. ### Theoretical and Practical Implications - The task highlights a critical gap in current video generation: achieving task outcomes with reference grounding remains largely unsolved, even for state-of-the-art models. - SemComp-Bench offers a structured, interpretable evaluation framework that can guide future model development by identifying specific failure modes (e.g., poor semantic grounding, low within-scene coherence). - SemComp-Data may serve as a training resource for task-specific fine-tuning, though this potential requires further empirical validation. ### Conclusion - Semantic Task Completion Video Generation is a challenging and underexplored capability for video models. - SemComp-Data and SemComp-Bench provide a scalable, authentic, and fine-grained evaluation foundation. - Future work should explore improving outcome realization and reference grounding, potentially via task-specific training on SemComp-Data. --- _Markdown view of https://picx.dev/p/qkfB3H, served by PicX — AI-generated visual whiteboard summaries of research papers._