Summary (Overview)

• KeyFrame-Compass is the first comprehensive benchmark dedicated to multi-keyframe-conditioned video generation, comprising 386 curated test cases across three application domains (daily capture, product visualization, cinematic narrative), two video structures (one-take and multi-shot), two prompt granularities (minimal and segment-specific), two conditioning formats (multi-image list and storyboard grid), and four keyframe densities (3, 6, 9, 12).
• The benchmark introduces a two-axis evaluation framework: (1) keyframe response metrics (six complementary measures: Hit Rate, Keyframe Similarity, Order Consistency, Position Accuracy, Persistence Around Keyframe, Response Uniqueness) and (2) general quality metrics (Video Quality, Spatiotemporal Coherence, Instruction Adherence, Audio-Visual Coordination) assessed via an evidence-grounded MLLM judge augmented with specialized perception models.
• Experiments on nine representative video generation systems (proprietary: Gemini-Omni-Flash, Kling-3.0-Omni, Seedance 2.0, Wan2.7-I2V; open-source: LTX-2.3, daVinci-MagiHuman-1080p-I2V, HunyuanVideo1.5-I2V, SkyReels-V2-I2V, Wan2.2-I2V-A14B) reveal fundamental limitations: a persistent trade-off between keyframe fidelity and natural video quality, degradation under denser constraints, and failure of open-source models to interpret storyboard-grid inputs.
• Seedance 2.0 achieves the highest overall score (0.807) on the joint audio-video leaderboard, balancing keyframe fidelity and general quality, while LTX-2.3 excels in keyframe faithfulness (0.855) but suffers in transition synthesis.
• Human alignment validates the automatic evaluation framework: Spearman’s $\rho_s$ between human and automatic rankings ranges from 0.70 to 1.00 across evaluation dimensions.

Introduction and Theoretical Foundation

Background and Motivation

Video generation increasingly adopts keyframe-based workflows, where creators first specify key moments as reference images and expect the model to generate a coherent video that faithfully reproduces the prescribed visual plan. Recent advances in image generation (e.g., GPT-Image-2, Nano Banana Pro) make creating such visual plans easy, shifting the bottleneck from designing keyframes to executing them as complete videos. Keyframe-conditioned video generation takes an ordered set of reference images together with a text prompt and must:

• preserve the entire sequence of keyframes,
• place each keyframe at the correct temporal location,
• and maintain coherent motion between them.

Gap in Existing Benchmarks

Existing benchmarks fall short:

• General video generation benchmarks (VBench, EvalCrafter, TC-Bench, VBench-2.0) assess perceptual quality, temporal consistency, and prompt following, but do not measure whether a complete keyframe sequence is reproduced with correct appearance, order, and timing.
• Image-conditioned benchmarks (AIGCBench, UI2V-Bench, VideoCanvasBench, ViStoryBench) focus on single-image preservation, spatiotemporal completion, or story visualization, but not ordered multi-keyframe execution.
• Multi-keyframe generation exhibits distinct failure modes (omission, inaccurate reproduction, misplacement, wrong order, implausible transitions) that require diagnostic evaluation along separate dimensions.

Theoretical Contribution

KeyFrame-Compass decomposes keyframe execution into six measurable aspects and provides a unified framework to diagnose controllability, temporal alignment, and overall generation quality. It systematically varies five controlled factors to enable stratified analysis.

Methodology

Benchmark Design

Each test case defines a keyframe-conditioned video generation task: the model receives an ordered sequence of keyframe images and a text prompt, and must generate a video that realizes the specified visual states in the intended temporal order.

Keyframe positioning is defined by the video structure:

• Multi-shot: each keyframe is assigned to a specific shot with a role: first, last, or representative.
• One-take: each keyframe is assigned a target timestamp along a continuous trajectory.

Controlled factors (5 dimensions):

1. Input format: multi-image list vs. storyboard grid.
2. Prompt control level: minimal (only order, synopsis, structure, duration) vs. segment-specific (temporal placement, subject states, camera language, narrative content).
3. Video structure: one-take vs. multi-shot.
4. Keyframe count: 3, 6, 9, or 12.
5. Application domain: daily capture, product visualization, cinematic narrative.

Data Construction

Data is sourced from:

• Narrative datasets: ViStoryBench, VIST, ROCStories.
• Real-video transcription: selected real videos with MLLM-generated narrative captions.

Each story is converted into a structured scene specification containing synopsis, video structure, subject descriptions, shot definitions, temporal layout, keyframe positions, and cinematic annotations. Keyframe images are generated using GPT-Image-2 and Nano Banana Pro, then screened via multimodal consistency checks and human review. Video prompts are rewritten by Gemini 3.1 Pro.

Evaluation Metrics

The evaluation framework has two components:

Keyframe Response Metrics

All metrics depend on a shared matching pipeline (Figure 4 in paper):

1. Use Gemini 3.1 Pro to infer actual shot structure and assign each input keyframe to a generated segment.
2. Determine expected temporal windows (one-to-one, multi-to-one, unassigned).
3. Within each window, compare keyframe against every generated frame using semantic (DINOv3 cosine similarity, threshold $\tau_d$) and pixel-level (PSNR $\geq \tau_p$ or SSIM $\geq \tau_s$) criteria.
4. Select the candidate with highest DINOv3 similarity; mark as unmatched if none satisfies thresholds.

Six metrics:

• Hit Rate (HR) – fraction of input keyframes with a valid match:

  $$\text{HR} = \frac{N_{\text{hit}}}{N_{\text{key}}}$$

• Keyframe Similarity (KFS) – reproduction fidelity of matched frames:

  $$q_i = w_{\text{pix}} s_i^{\text{pix}} + w_{\text{dino}} c_i^{\text{dino}}, \quad i \in \mathcal{M}, \quad \text{KFS} = \frac{1}{N_{\text{kf}}} \sum_{i \in \mathcal{M}} q_i$$

  where $w_{\text{pix}}=0.4$, $w_{\text{dino}}=0.6$.

• Keyframe Position Accuracy (KPA) – temporal placement score:

  $$p_i = \begin{cases} \max\left(0, 1 - \frac{|t_i^{\text{match}} - t_i^{\text{target}}|}{\epsilon_i}\right), & i \in P \text{ (point-position)} \\ \mathbf{1}[t_i^{\text{match}} \in [t_i^{\text{start}}, t_i^{\text{end}}]], & i \in R \text{ (representative)} \end{cases}$$ $$\text{KPA} = \frac{1}{N_{\text{match}}} \sum_{i=1}^{N_{\text{match}}} p_i$$
  • Keyframe Order Consistency (KOC) – Kendall’s $\tau$ between input order and matched timestamps (global, without window constraints), normalized to $[0,1]$:
  $$\text{KOC} = \frac{\tau + 1}{2}$$

• Persistence Around Keyframe (PAK) – detects flash/freeze failures; takes the minimum of a flash score (based on response duration) and a freeze score (based on temporal variation) per keyframe.

• Response Uniqueness (RU) – measures whether each keyframe appears in a single coherent temporal region. Clusters matched frames by temporal proximity; score is fraction of matches in the dominant cluster if multiple clusters, else 1.

General Quality Metrics

Evaluated with a checklist-based protocol using GPT-5.5 for checklist generation and Gemini 3.1 Pro for scoring, augmented by specialized perception models.

Four dimensions:

1. Video Quality: Static Visual Quality (DOVER, MUSIQ) and Dynamic Visual Quality (DOVER).
2. Spatiotemporal Coherence: Attribute Consistency (using SAM 3.1 tracking, DINOv3, ElasticFace, etc.), Spatial Orientation Consistency (MonST3R camera motion descriptors), Physical Rationality (five commonsense dimensions).
3. Instruction Adherence: Video Modality Adherence (camera execution, shot structure, narrative pacing, subject-scene alignment) and Audio Modality Adherence (CLAP similarity fusion with Gemini score): $$S_{\text{audio}} = \lambda_{\text{audio}} S_{\text{Gemini}}^{\text{audio}} + (1-\lambda_{\text{audio}}) \tilde{C}_{\text{CLAP}}^{\text{audio}}$$
4. Audio-Visual Coordination: ImageBind Similarity and JavisScore (window-level synchrony averaging least-aligned frames).

All scores normalized to $[0,1]$.

Empirical Validation / Results

Experimental Setup

• Models evaluated: 4 proprietary (Gemini-Omni-Flash, Kling-3.0-Omni, Seedance 2.0, Wan2.7-I2V) + 5 open-source (LTX-2.3, daVinci-MagiHuman-1080p-I2V, HunyuanVideo1.5-I2V, SkyReels-V2-I2V, Wan2.2-I2V-A14B).
• Short videos (≤10s) generated in single pass; long videos (10–45s) via agent-mode workflows (only Kling-3.0-Omni, Seedance 2.0).
• Input format adapted per model (multi-image or storyboard grid).

Main Results – Short-Video Leaderboard (Table 3)

Key insights:

• Seedance 2.0 ranks first overall (0.807) with balanced performance.
• LTX-2.3 leads in keyframe fidelity and temporal organization but lags in video quality, revealing the fidelity-quality trade-off.
• Gemini-Omni-Flash excels in video quality and AV coordination but has low keyframe fidelity (0.483).

Per-Metric Results (Table 4 – short video, segment-specific prompt – abridged)

Instruction Adherence by Keyframe Density (Table 5)

Instruction adherence declines as keyframe count increases, driven by VMA; AMA remains stable.

Key Failure Modes

1. LTX-2.3: Faithful keyframe reproduction but abrupt, slideshow-like transitions (Figure 6a) and implausible morphing (Figure 6b).
2. Gemini-Omni-Flash: Uses keyframes as semantic references rather than visual anchors; re-stages shots with new appearances.
3. Open-source models (except LTX-2.3): Fail to decompose storyboard grids (Figure 6c), outputting grid as static image; contamination from unrelated visual artifacts (Figure 6d); daVinci reads prompt text aloud.

Human Alignment (Table 6)

Spearman’s rank correlation between human and automatic rankings (n=5 models):

High alignment, especially for Instruction Adherence ($\rho_s = 1.00$, $p=0.017$).

Theoretical and Practical Implications

• Diagnostic separation of capabilities: The two-axis framework reveals that keyframe fidelity and general video quality are non-interchangeable. Ranking by fidelity alone favors LTX-2.3; ranking by quality alone favors Gemini-Omni-Flash; joint evaluation favors Seedance 2.0. This demonstrates the need for multi-dimensional evaluation.
• Fidelity vs. quality trade-off: Models that adhere strictly to keyframes often produce unnatural transitions (LTX-2.3), while models with high quality may drift from input appearances (Gemini-Omni-Flash). Practical applications must choose the right balance: applications requiring strict visual adherence (e.g., product visualization) may prefer high-fidelity models, whereas those prioritizing narrative flow may favor high-quality models.
• Open-source gap in multi-keyframe comprehension: The failure of open-source models (except LTX-2.3) to interpret storyboard grids indicates a training data gap rather than capacity limitation. Future work should include multi-image conditioning or grid decomposition training.
• Degradation under dense constraints: Increasing keyframe count reduces instruction adherence, especially in visual and temporal aspects. This suggests that current models cannot effectively handle dense visual plans; generating longer videos with many keyframes remains challenging.
• Utility for benchmarking: KeyFrame-Compass provides a standardized benchmark with controlled factors, enabling fair comparison and diagnosing specific weaknesses (e.g., transition synthesis, temporal localization, ordering).

Conclusion

KeyFrame-Compass establishes the first comprehensive benchmark for keyframe-conditioned video generation. It covers diverse settings (386 samples, 5 controlled factors) and evaluates both keyframe execution (6 metrics) and overall quality (4 dimensions) via an automated pipeline combining MLLM judges and specialized perception models. Experiments on 9 models reveal:

• A consistent trade-off between keyframe fidelity and natural video synthesis.
• Degradation of instruction adherence under denser keyframe