VGI-BENCH: Probing Visual Intelligence in Video Generation Models

Summary (Overview)

• Introduces VGI-BENCH, a benchmark for evaluating visual intelligence in video generation models through 27 tasks and 810 instances organized in a two-level taxonomy (domains + skill tags).
• Addresses three key gaps in prior benchmarks: (1) distribution-mismatched inputs (line-art vs. photorealistic), (2) lack of process-sensitive evaluation (final-state-only checking), and (3) uncontrolled task difficulty (tasks too easy or too hard to be diagnostic).
• Evaluates 9 video generation models (commercial + open-source) and 11 image generation models, showing the strongest model (Seedance 2.0) achieves only 51.0 overall score.
• Provides multi-faceted diagnostic analyses covering failure modes, input sensitivity, synthetic fine-tuning transfer, and denoising dynamics.
• Demonstrates that current video models show emerging reasoning abilities but remain far from general visual intelligence, with limited self-correction during generation.

Introduction and Theoretical Foundation

Background

Video generation models were initially developed as content creation systems focused on visual quality, motion realism, and condition alignment (Yang et al., 2025c; Kong et al., 2024; Wan et al., 2025). As temporal coherence and physical plausibility improved, they became viewed as visual world simulators—generated videos serve as explicit predictions of scene evolution (Brooks et al., 2024; Qin et al., 2024). More recently, studies suggest they may act as zero-shot visual reasoners, expressing solutions through generated frame sequences (Wiedemer et al., 2025; Tong et al., 2025).

Key Gaps in Prior Benchmarks

The paper identifies three critical limitations in existing evaluation approaches:

1. Distribution-mismatched visual appearances: Many benchmarks (e.g., TiVi-Bench, V-ReasonBench, MMGR) use line-art or abstract inputs that deviate from natural-image priors of video models, conflating reasoning limitations with visual-domain mismatch.

2. Limited demand for visual rollout reasoning: Many visual QA tasks can be answered directly from the input without requiring the model to simulate scene evolution, failing to test the model's ability to generate valid intermediate trajectories.

3. Uncontrolled task difficulty: Existing benchmarks include tasks far beyond feasible regimes (long-horizon tasks exceeding practical video duration, knowledge-heavy tasks requiring non-visual expertise), making failures less diagnostic.

Methodology

Benchmark Design

Two-Level Taxonomy:

• Domains (mutually exclusive, based on visual characteristics):

  • Visual Organization: arranging, grouping, or selecting objects based on visual attributes
  • Physical Manipulation: object-level actions (moving, placing, stacking, tool use)
  • Structured Puzzles: rule-governed visual puzzles with explicit constraints
  • Spatiotemporal Dynamics: reasoning about state evolution over time

• Skill Tags (non-exclusive, capability-level view):

  • Spatial, Temporal, Planning, Attribute Grounding, Physics, Topology, Affordance

Task Collection:

• Unified I/O format: text prompt + input image (first frame) → generated video
• Process-sensitive design: success depends on intermediate state evolution, not final-state correctness
• Three difficulty levels per task (Easy/Mid/Hard), ~10 instances per level
• Input images sourced from web/datasets or generated (GPT-Image-2, Nano Banana Pro) with human-in-the-loop review
• Reference solutions (image or text description) guide criteria construction

Quality Control:

• Pre-generation filter: Test easiest-level instances on state-of-the-art models; accept task only if solved by ≥1 model and failed by ≥1 model
• Manual review: Verify goal fidelity, duration fit (5-10s), and prompt clarity

Evaluation Criteria

Completeness (Comp.): Measures global progress toward task goal via tiered standard (<complete>, <partial>, <failed>), judged on uniformly sampled frames at 2fps.

Rubric Score (Rub.): Measures local process validity via fine-grained checklist, using coarse-to-fine adaptive frame sampling (4fps → 8fps for flagged intervals) and sliding focus windows (10-frame with edge overlap). Each item scored by inverse decay penalty: $1/(x+1)$ where x = violation count.

Final Score: Multiplicative combination: $\text{Final} = \text{Comp.} \times \text{Rub.}$—penalizes either type of failure, treating both as jointly necessary conditions.

Data Augmentation

Adapts tasks to single-image output format for evaluating image generation models (e.g., MAZE → draw valid path; RECOVER 2D NET → render completed 3D structure).

Empirical Validation / Results

Video Generation Models (Table 2)

Key findings:

• Commercial models outperform open-source models consistently
• Structured Puzzles is the most challenging domain (multi-step rules, state tracking)
• Topology and Temporal are the weakest skill dimensions (connectivity preservation, multi-step state tracking)

Image Generation Models (Table 4)

On adapted subset (Success Rate %): Nano-Banana-Pro leads (55.0), followed by Seedream-5.0-Pro (52.1). Consistent easy→hard difficulty gradient confirms benchmark calibration.

Evaluator Reliability (Table 3)

Full evaluator achieves strongest agreement with human judgments; both adaptive frame sampling and focus window are necessary components.

Failure Modes (Figure 3)

1. Physical collapse: Unrealistic deformation, object penetration, sudden disappearance; physical causality sacrificed for goal-like visual outcomes
2. Rule violation: Physically plausible but rule-breaking videos (prohibited actions, skipped steps, direct scene alteration)
3. Object/state inconsistency: Objects disappear, transform, or reset to earlier states despite smooth motion

Input Condition Sensitivity (Figure 5)

• Oracle prompting (explicit solution descriptions): Improves performance for some models (especially closed-source), but gains limited—models still struggle with instruction following and physical simulation under strong constraints
• Visual style (line-art vs. realistic): Model rankings differ noticeably; open-source models show stronger sensitivity to visual appearance, confirming that abstract inputs conflate reasoning limitations with domain mismatch

Synthetic Fine-Tuning Transfer (Tables 5-6)

Performance gains decrease with structural overlap with training data, but transfer is non-uniform: some non-overlap tasks still benefit (e.g., UN-TIE_KNOT rubric score rises 0.02→0.34). Improvements concentrate on planning and spatial skills; physical interaction and temporal dependency remain hard to improve or may degrade.

Denoising Dynamics (Table 7)

Self-correction stays below 1% everywhere and stops in later steps; wrong→wrong′ is an order of magnitude more frequent (23.1% at 4→10, 24.8% at 10→20). Once readable, states mostly stay put (stability rises to 90.6% in second half). Later steps lock in and refine early hypotheses rather than correcting errors.

Theoretical and Practical Implications

Theoretical Significance

• Provides empirical evidence that video generation models encode structured representations beyond low-level appearance and motion (spatial-temporal relations, rule constraints, action-outcome dependencies)
• Demonstrates the emerging reasoning paradigm where solutions are expressed through generated frame sequences
• Reveals that current models operate as "refiners rather than correctors" during denoising—they refine early hypotheses but lack true self-correction capabilities
• Shows that synthetic fine-tuning transfer is bounded by structural coverage of training distribution, informing theoretical understanding of generalization in generative reasoning

Practical Implications

• Benchmark design: Photorealistic inputs, process-sensitive evaluation, and calibrated difficulty are essential for diagnostic evaluation
• Model development: Identifies specific weaknesses (physical collapse, rule violation, state inconsistency) that should guide future training objectives
• Evaluation methodology: Multiplicative Comp. × Rub. scoring prevents gaming either dimension; adaptive frame sampling captures transient violations efficiently
• Fine-tuning strategies: Synthetic data offers scalable supervision but requires careful attention to structural coverage; improvements may not transfer uniformly across skills
• Prompt engineering: Oracle prompting provides limited gains, suggesting instruction-following bottlenecks beyond prompt quality

Conclusion

VGI-BENCH provides a diagnostic testbed for evaluating whether video models can solve tasks through valid visual rollouts. Key takeaways:

1. Current state: Models exhibit emerging reasoning ability but remain far from general visual intelligence (best score 51.0)
2. Key bottlenecks: Multi-step rule following, physical coherence under goal pressure, temporal state tracking, and self-correction during generation
3. Sensitivity: Performance varies substantially with input conditions (prompts, visual styles), especially for open-source models
4. Training insights: Synthetic fine-tuning transfers but is bounded by training distribution coverage

Future directions: Longer-horizon procedural reasoning, text-to-video and multi-image conditioning settings, multilingual evaluation, and extension of task suite as model capabilities improve.

Limitations acknowledged: Tasks limited to 5-10s duration, image-to-video setting with fixed 16:9 aspect ratio, English-only prompts, representative (not exhaustive) task coverage.