Summary

• KeyFrame-Compass is the first comprehensive benchmark dedicated to multi-keyframe-conditioned video generation, containing 386 curated test cases spanning three application domains (daily capture, product visualization, cinematic narrative), two video structures (one-take vs. multi-shot), two prompt granularities (minimal vs. segment-specific), two conditioning formats (multi-image list vs. storyboard grid), and four keyframe densities (3, 6, 9, 12).
• The benchmark introduces a decomposed evaluation framework with six keyframe-response metrics (presence, fidelity, temporal ordering, localization, persistence, uniqueness) and four general quality groups (video quality, spatiotemporal coherence, instruction adherence, audio-visual coordination), enabling diagnosis of distinct failure modes rather than collapsing them into a single score.
• Experiments on nine representative video generation systems reveal a persistent trade-off between keyframe fidelity and natural video synthesis—no evaluated system performs best on both axes. Seedance 2.0 ranks first overall (0.807), while LTX-2.3 leads in keyframe fidelity (0.855) but suffers in general quality.
• Key findings include: performance degrades as keyframe constraints become denser; most open-source models fail to interpret storyboard-grid inputs as temporally ordered keyframe sequences; and semantic instruction compliance does not imply visual grounding to input keyframes.

Introduction and Theoretical Foundation

Video generation increasingly relies on keyframe-based workflows, where creators specify key moments as reference images and expect the generation model to transform them into a coherent video. Keyframe-conditioned video generation takes an ordered set of reference images together with a text prompt as input and generates a video that faithfully realizes the prescribed visual plan. Unlike conventional image-conditioned generation, it must preserve an entire sequence of keyframes, place each one at the correct time, and maintain coherent motion between them.

Existing benchmarks are not designed for this setting:

• General video generation benchmarks (VBench, EvalCrafter, TC-Bench, VBench-2.0) mainly assess perceptual quality, temporal consistency, and prompt following.
• Image-conditioned benchmarks (AIGCBench, UI2V-Bench, VideoCanvasBench, ViStoryBench) focus on single-image preservation, spatiotemporal completion, or story visualization.

None measures whether a complete keyframe sequence is reproduced with the correct appearance, order, and timing. Multi-keyframe-conditioned generation exhibits several distinct failure modes: keyframes may be omitted, inaccurately reproduced, misplaced in time, presented in the wrong order, or connected by implausible transitions. These errors arise from different underlying abilities—visual preservation, temporal localization, sequence ordering, and motion synthesis—requiring separate evaluation per dimension.

Methodology

Benchmark Design

Each sample defines a keyframe-conditioned video generation task where a model receives a sequence of keyframe images and a text prompt, and must reproduce each keyframe at its intended location while synthesizing motion between anchors. Samples are annotated along five dimensions:

• Prompt control levels: Minimal prompts (keyframe order, brief synopsis, structure, duration) vs. segment-specific prompts (temporal placement, subject states, camera language, narrative content per segment).
• Video structure: One-take (all keyframes in a single continuous shot) vs. multi-shot (keyframes distributed across shots).
• Keyframe count: 3, 6, 9, or 12 keyframes.
• Input formats: Multi-image list vs. storyboard grid (single image).
• Application domain: Daily capture, product visualization, cinematic narrative.

Data Construction

Narrative sources from ViStoryBench, VIST, and ROCStories, plus real videos with generated captions, are converted into structured story summaries. These become scene specifications containing narrative synopsis, video structure, subject descriptions, shot definitions, temporal layout, and cinematic annotations. Keyframe images are generated with GPT-Image-2 and Nano Banana Pro, then screened through multimodal consistency checks and human review. Prompts are rewritten by Gemini 3.1 Pro into video-oriented formats.

Keyframe Response Metrics

All metrics rely on a shared matching pipeline that uses Gemini 3.1 Pro to infer the actual shot structure of generated videos, maps input keyframes to generated segments, and determines expected temporal windows. Within windows, candidate frames are filtered using DINOv3 semantic similarity and PSNR/SSIM pixel fidelity.

Keyframe Fidelity metrics:

• Hit Rate (HR): Fraction of input keyframes with a valid match within the expected window:

$$\mathrm{HR} = \frac{N_{\mathrm{hit}}}{N_{\mathrm{key}}}$$

• Keyframe Similarity (KFS): Combines pixel-level similarity (PSNR, SSIM) with DINOv3 similarity:

$$q_i = w_{\mathrm{pix}} s_i^{\mathrm{pix}} + w_{\mathrm{dino}} c_i^{\mathrm{dino}}, \quad \mathrm{KFS} = \frac{1}{N_{\mathrm{kf}}} \sum_{i \in \mathcal{M}} q_i$$

with $w_{\mathrm{pix}} = 0.4$, $w_{\mathrm{dino}} = 0.6$.

Keyframe Temporal Organization metrics:

• Keyframe Position Accuracy (KPA): Linear decay for point-position keyframes, binary presence rule for representative keyframes.
• Keyframe Order Consistency (KOC): Kendall's τ between input keyframe order and matched timestamps, normalized: $\mathrm{KOC} = \frac{\tau + 1}{2}$.
• Persistence Around Keyframe (PAK): Detects "flash" (brief appearance) and "freeze" (near-static) failure modes, taking the minimum of both scores.
• Response Uniqueness (RU): Measures whether each keyframe appears in a single coherent temporal region rather than repeatedly at disconnected locations.

General Quality Metrics

Four dimensions evaluated via a checklist-based protocol where GPT-5.5 generates observable items and Gemini 3.1 Pro scores against them:

1. Video Quality: Static visual quality (clarity, artifacts, color, style) and dynamic visual quality (motion discontinuities, stutter, flickering), with DOVER and MUSIQ as auxiliary evidence.
2. Spatiotemporal Coherence: Attribute consistency (using SAM 3.1 tracking, DINOv3 embeddings, ElasticFace for faces, InceptionNeXt for bodies), spatial orientation consistency (using MonST3R camera trajectories), and physical rationality (interaction plausibility, anatomical consistency, structural stability, gravity/material behavior, scene-level common sense).
3. Instruction Adherence: Video modality adherence (camera execution, shot structure, narrative pacing, subject-scene alignment) and audio modality adherence (combining Gemini scores with CLAP audio-text similarity).
4. Audio-Visual Coordination: ImageBind similarity and JavisScore for window-level audio-visual synchrony.

Empirical Validation / Results

Main Results

On the common set of 115 samples for the joint audio-video leaderboard:

Rank reversals demonstrate that input preservation, temporal organization, and general generation quality are non-interchangeable capabilities: LTX-2.3 leads in keyframe fidelity but ranks fourth overall; Gemini-Omni-Flash leads in video quality despite low keyframe fidelity.

Key Insights

1. Strong keyframe execution highlights transition synthesis as the remaining bottleneck: LTX-2.3 reproduces keyframes with high fidelity but produces abrupt, slideshow-like transitions or implausible morphing between anchors.

2. Semantic compliance does not imply visual grounding: Gemini-Omni-Flash satisfies shot-level instructions but re-stages shots, reconstructing scene layout and character appearance from scratch. Seedance 2.0 shows the opposite preference, using keyframes as strict appearance anchors.

3. Instruction adherence declines with constraint density: VMA decreases from 0.849 (3 keyframes) to 0.818 (6 keyframes) to 0.756 (9&12 keyframes), driven primarily by visual/temporal control rather than audio adherence.

4. Open-source models lack storyboard-grid comprehension: Models receiving keyframes as a storyboard grid (daVinci, HunyuanVideo, SkyReels, Wan2.2-I2V) fail before generation begins—Hit Rates stay below 0.28, often reproducing the grid as a whole in near-static videos.

Human Alignment

Human evaluation on 60 segment-level cases with 600 pairwise comparisons shows Spearman's ρ between human and automatic rankings ranging from 0.70 to 1.00 across dimensions, with Instruction Adherence achieving perfect alignment (ρ = 1.00, p = 0.017).

Theoretical and Practical Implications

For evaluation methodology: KeyFrame-Compass demonstrates that keyframe-conditioned generation requires decomposed diagnostic evaluation rather than aggregate scoring. The six keyframe-response metrics isolate distinct failure modes (omission, infidelity, misordering, mistiming, flash/freeze, duplication) that arise from different underlying model capabilities.

For model development: The observed trade-off between keyframe fidelity and natural video synthesis suggests fundamental architectural tensions. Models that anchor strictly to input keyframes (LTX-2.3, Seedance 2.0) risk abrupt transitions; models that use keyframes as loose semantic guidance (Gemini-Omni-Flash) drift from visual content. This points to a need for explicit transition synthesis modules that can bridge visual anchors with plausible motion.

For open-source progress: The failure of open-source models to decompose storyboard grids indicates a training data gap rather than model capacity limitations, suggesting that multi-keyframe conditioning should be incorporated into training pipelines.

Conclusion

KeyFrame-Compass provides the first standardized benchmark for diagnosing keyframe-conditioned video generation, combining keyframe response metrics with comprehensive quality assessment. Experimental results show that current models remain limited in: (1) following fine-grained instructions under dense keyframe constraints, (2) generating coherent transitions between keyframes, (3) decomposing storyboard-grid inputs (particularly in open-source models), and (4) producing one-take videos with genuine continuity.

Future directions include improving transition synthesis between visual anchors, developing better multi-keyframe conditioning interfaces for open-source models, and extending evaluation to additional application domains and generation settings.