# KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation

> KeyFrame-Compass reveals a persistent trade-off between keyframe fidelity and natural video quality in multi-keyframe-conditioned generation.

- **Source:** [arXiv](https://arxiv.org/abs/2607.14202)
- **Published:** 2026-07-18
- **Permalink:** https://picx.dev/p/HQbP63
- **Whiteboard:** https://picx.dev/p/HQbP63/image

## Summary

## Summary (Overview)

- **KeyFrame-Compass** is the first comprehensive benchmark dedicated to **multi-keyframe-conditioned video generation**, comprising **386 curated test cases** across three application domains (daily capture, product visualization, cinematic narrative), two video structures (one-take and multi-shot), two prompt granularities (minimal and segment-specific), two conditioning formats (multi-image list and storyboard grid), and four keyframe densities (3, 6, 9, 12).
- The benchmark introduces a **two-axis evaluation framework**: (1) **keyframe response metrics** (six complementary measures: Hit Rate, Keyframe Similarity, Order Consistency, Position Accuracy, Persistence Around Keyframe, Response Uniqueness) and (2) **general quality metrics** (Video Quality, Spatiotemporal Coherence, Instruction Adherence, Audio-Visual Coordination) assessed via an evidence-grounded MLLM judge augmented with specialized perception models.
- Experiments on **nine representative video generation systems** (proprietary: Gemini-Omni-Flash, Kling-3.0-Omni, Seedance 2.0, Wan2.7-I2V; open-source: LTX-2.3, daVinci-MagiHuman-1080p-I2V, HunyuanVideo1.5-I2V, SkyReels-V2-I2V, Wan2.2-I2V-A14B) reveal fundamental limitations: a **persistent trade-off between keyframe fidelity and natural video quality**, degradation under denser constraints, and failure of open-source models to interpret storyboard-grid inputs.
- **Seedance 2.0** achieves the highest overall score (0.807) on the joint audio-video leaderboard, balancing keyframe fidelity and general quality, while **LTX-2.3** excels in keyframe faithfulness (0.855) but suffers in transition synthesis.
- Human alignment validates the automatic evaluation framework: Spearman’s $\rho_s$ between human and automatic rankings ranges from **0.70 to 1.00** across evaluation dimensions.

## Introduction and Theoretical Foundation

### Background and Motivation

Video generation increasingly adopts **keyframe-based workflows**, where creators first specify key moments as reference images and expect the model to generate a coherent video that faithfully reproduces the prescribed visual plan. Recent advances in image generation (e.g., GPT-Image-2, Nano Banana Pro) make creating such visual plans easy, shifting the bottleneck from designing keyframes to executing them as complete videos. Keyframe-conditioned video generation takes an **ordered set of reference images** together with a text prompt and must:
- preserve the entire sequence of keyframes,
- place each keyframe at the correct temporal location,
- and maintain coherent motion between them.

### Gap in Existing Benchmarks

Existing benchmarks fall short:
- **General video generation benchmarks** (VBench, EvalCrafter, TC-Bench, VBench-2.0) assess perceptual quality, temporal consistency, and prompt following, but do **not** measure whether a complete keyframe sequence is reproduced with correct appearance, order, and timing.
- **Image-conditioned benchmarks** (AIGCBench, UI2V-Bench, VideoCanvasBench, ViStoryBench) focus on single-image preservation, spatiotemporal completion, or story visualization, but **not** ordered multi-keyframe execution.
- Multi-keyframe generation exhibits distinct failure modes (omission, inaccurate reproduction, misplacement, wrong order, implausible transitions) that require **diagnostic evaluation along separate dimensions**.

### Theoretical Contribution

KeyFrame-Compass decomposes keyframe execution into six measurable aspects and provides a unified framework to diagnose controllability, temporal alignment, and overall generation quality. It systematically varies five controlled factors to enable stratified analysis.

## Methodology

### Benchmark Design

Each test case defines a keyframe-conditioned video generation task: the model receives an ordered sequence of keyframe images and a text prompt, and must generate a video that realizes the specified visual states in the intended temporal order.

**Keyframe positioning** is defined by the video structure:
- **Multi-shot**: each keyframe is assigned to a specific shot with a role: *first*, *last*, or *representative*.
- **One-take**: each keyframe is assigned a target timestamp along a continuous trajectory.

**Controlled factors** (5 dimensions):
1. **Input format**: multi-image list vs. storyboard grid.
2. **Prompt control level**: minimal (only order, synopsis, structure, duration) vs. segment-specific (temporal placement, subject states, camera language, narrative content).
3. **Video structure**: one-take vs. multi-shot.
4. **Keyframe count**: 3, 6, 9, or 12.
5. **Application domain**: daily capture, product visualization, cinematic narrative.

### Data Construction

Data is sourced from:
- **Narrative datasets**: ViStoryBench, VIST, ROCStories.
- **Real-video transcription**: selected real videos with MLLM-generated narrative captions.

Each story is converted into a **structured scene specification** containing synopsis, video structure, subject descriptions, shot definitions, temporal layout, keyframe positions, and cinematic annotations. Keyframe images are generated using GPT-Image-2 and Nano Banana Pro, then screened via multimodal consistency checks and human review. Video prompts are rewritten by Gemini 3.1 Pro.

### Evaluation Metrics

The evaluation framework has two components:

#### Keyframe Response Metrics

All metrics depend on a **shared matching pipeline** (Figure 4 in paper):
1. Use Gemini 3.1 Pro to infer actual shot structure and assign each input keyframe to a generated segment.
2. Determine expected temporal windows (one-to-one, multi-to-one, unassigned).
3. Within each window, compare keyframe against every generated frame using **semantic** (DINOv3 cosine similarity, threshold $\tau_d$) and **pixel-level** (PSNR $\geq \tau_p$ or SSIM $\geq \tau_s$) criteria.
4. Select the candidate with highest DINOv3 similarity; mark as unmatched if none satisfies thresholds.

**Six metrics**:

- **Hit Rate (HR)** – fraction of input keyframes with a valid match:
  $$
  \text{HR} = \frac{N_{\text{hit}}}{N_{\text{key}}}
  $$

- **Keyframe Similarity (KFS)** – reproduction fidelity of matched frames:
  $$
  q_i = w_{\text{pix}} s_i^{\text{pix}} + w_{\text{dino}} c_i^{\text{dino}}, \quad i \in \mathcal{M}, \quad \text{KFS} = \frac{1}{N_{\text{kf}}} \sum_{i \in \mathcal{M}} q_i
  $$
  where $w_{\text{pix}}=0.4$, $w_{\text{dino}}=0.6$.

- **Keyframe Position Accuracy (KPA)** – temporal placement score:
  $$
  p_i = \begin{cases}
  \max\left(0, 1 - \frac{|t_i^{\text{match}} - t_i^{\text{target}}|}{\epsilon_i}\right), & i \in P \text{ (point-position)} \\
  \mathbf{1}[t_i^{\text{match}} \in [t_i^{\text{start}}, t_i^{\text{end}}]], & i \in R \text{ (representative)}
  \end{cases}
  $$
  $$
  \text{KPA} = \frac{1}{N_{\text{match}}} \sum_{i=1}^{N_{\text{match}}} p_i
  $$

- **Keyframe Order Consistency (KOC)** – Kendall’s $\tau$ between input order and matched timestamps (global, without window constraints), normalized to $[0,1]$:
  $$
  \text{KOC} = \frac{\tau + 1}{2}
  $$

- **Persistence Around Keyframe (PAK)** – detects flash/freeze failures; takes the minimum of a flash score (based on response duration) and a freeze score (based on temporal variation) per keyframe.

- **Response Uniqueness (RU)** – measures whether each keyframe appears in a single coherent temporal region. Clusters matched frames by temporal proximity; score is fraction of matches in the dominant cluster if multiple clusters, else 1.

#### General Quality Metrics

Evaluated with a checklist-based protocol using GPT-5.5 for checklist generation and Gemini 3.1 Pro for scoring, augmented by specialized perception models.

**Four dimensions**:
1. **Video Quality**: Static Visual Quality (DOVER, MUSIQ) and Dynamic Visual Quality (DOVER).
2. **Spatiotemporal Coherence**: Attribute Consistency (using SAM 3.1 tracking, DINOv3, ElasticFace, etc.), Spatial Orientation Consistency (MonST3R camera motion descriptors), Physical Rationality (five commonsense dimensions).
3. **Instruction Adherence**: Video Modality Adherence (camera execution, shot structure, narrative pacing, subject-scene alignment) and Audio Modality Adherence (CLAP similarity fusion with Gemini score):
   $$
   S_{\text{audio}} = \lambda_{\text{audio}} S_{\text{Gemini}}^{\text{audio}} + (1-\lambda_{\text{audio}}) \tilde{C}_{\text{CLAP}}^{\text{audio}}
   $$
4. **Audio-Visual Coordination**: ImageBind Similarity and JavisScore (window-level synchrony averaging least-aligned frames).

All scores normalized to $[0,1]$.

## Empirical Validation / Results

### Experimental Setup

- **Models evaluated**: 4 proprietary (Gemini-Omni-Flash, Kling-3.0-Omni, Seedance 2.0, Wan2.7-I2V) + 5 open-source (LTX-2.3, daVinci-MagiHuman-1080p-I2V, HunyuanVideo1.5-I2V, SkyReels-V2-I2V, Wan2.2-I2V-A14B).
- Short videos (≤10s) generated in single pass; long videos (10–45s) via agent-mode workflows (only Kling-3.0-Omni, Seedance 2.0).
- Input format adapted per model (multi-image or storyboard grid).

### Main Results – Short-Video Leaderboard (Table 3)

| Rank | Model | Keyframe Fidelity | Temporal Organization | Video Quality | Spatiotemporal Coherence | Instruction Adherence | AV Coordination | **Overall** |
|------|-------|-------------------|-----------------------|---------------|--------------------------|-----------------------|-----------------|-------------|
| 1 | Seedance 2.0 | **0.807** | **0.859** | 0.850 | **0.935** | **0.931** | 0.626 | **0.807** |
| 2 | Gemini-Omni-Flash | 0.483 | 0.807 | **0.861** | 0.905 | 0.923 | **0.640** | 0.744 |
| 3 | Kling-3.0-Omni | 0.665 | 0.805 | 0.813 | 0.876 | 0.871 | 0.598 | 0.738 |
| 4 | LTX-2.3 | **0.855** | **0.899** | 0.557 | 0.680 | 0.721 | 0.570 | 0.659 |
| 5 | Wan2.7-I2V | 0.490 | 0.667 | 0.734 | 0.781 | 0.706 | 0.593 | 0.628 |
| 6 | daVinci-MagiHuman-1080p-I2V | 0.295 | 0.627 | 0.212 | 0.292 | 0.152 | 0.577 | 0.284 |

Key insights:
- **Seedance 2.0** ranks first overall (0.807) with balanced performance.
- **LTX-2.3** leads in keyframe fidelity and temporal organization but lags in video quality, revealing the fidelity-quality trade-off.
- **Gemini-Omni-Flash** excels in video quality and AV coordination but has low keyframe fidelity (0.483).

### Per-Metric Results (Table 4 – short video, segment-specific prompt – abridged)

| Model | HR | KFS | KOC | PAK | RU | KPA | SVQ | DVQ | AC | SOC | PR | VMA | AMA |
|-------|----|-----|-----|-----|----|-----|-----|-----|----|-----|----|-----|-----|
| Seedance 2.0 | 0.961 | 0.652 | 0.984 | 0.730 | 0.953 | 0.717 | 0.862 | 0.844 | 0.967 | 0.940 | 0.919 | 0.907 | 0.884 |
| Gemini-Omni-Flash | 0.623 | 0.371 | 0.901 | 0.798 | 0.922 | 0.587 | 0.864 | 0.861 | 0.940 | 0.914 | 0.869 | 0.907 | 0.855 |
| LTX-2.3 | 0.967 | 0.735 | 0.985 | 0.891 | 0.904 | 0.873 | 0.665 | 0.518 | 0.879 | 0.698 | 0.515 | 0.637 | 0.633 |

### Instruction Adherence by Keyframe Density (Table 5)

| Keyframes | Instruction | VMA | AMA |
|-----------|-------------|-----|-----|
| 3 | 0.849 | 0.861 | 0.837 |
| 6 | 0.818 | 0.799 | 0.837 |
| 9&12 | 0.756 | 0.682 | 0.830 |

Instruction adherence declines as keyframe count increases, driven by VMA; AMA remains stable.

### Key Failure Modes

1. **LTX-2.3**: Faithful keyframe reproduction but abrupt, slideshow-like transitions (Figure 6a) and implausible morphing (Figure 6b).
2. **Gemini-Omni-Flash**: Uses keyframes as semantic references rather than visual anchors; re-stages shots with new appearances.
3. **Open-source models (except LTX-2.3)**: Fail to decompose storyboard grids (Figure 6c), outputting grid as static image; contamination from unrelated visual artifacts (Figure 6d); daVinci reads prompt text aloud.

### Human Alignment (Table 6)

Spearman’s rank correlation between human and automatic rankings (n=5 models):

| Dimension | $\rho_s$ | Exact $p$ |
|-----------|----------|-----------|
| Video Quality | 0.90 | 0.083 |
| Spatiotemporal Coherence | 0.80 | 0.133 |
| Instruction Adherence | 1.00 | 0.017 |
| AV Coordination | 0.70 | 0.233 |
| Overall | 0.90 | 0.083 |

High alignment, especially for Instruction Adherence ($\rho_s = 1.00$, $p=0.017$).

## Theoretical and Practical Implications

- **Diagnostic separation of capabilities**: The two-axis framework reveals that keyframe fidelity and general video quality are non-interchangeable. Ranking by fidelity alone favors LTX-2.3; ranking by quality alone favors Gemini-Omni-Flash; joint evaluation favors Seedance 2.0. This demonstrates the need for multi-dimensional evaluation.
- **Fidelity vs. quality trade-off**: Models that adhere strictly to keyframes often produce unnatural transitions (LTX-2.3), while models with high quality may drift from input appearances (Gemini-Omni-Flash). Practical applications must choose the right balance: applications requiring strict visual adherence (e.g., product visualization) may prefer high-fidelity models, whereas those prioritizing narrative flow may favor high-quality models.
- **Open-source gap in multi-keyframe comprehension**: The failure of open-source models (except LTX-2.3) to interpret storyboard grids indicates a training data gap rather than capacity limitation. Future work should include multi-image conditioning or grid decomposition training.
- **Degradation under dense constraints**: Increasing keyframe count reduces instruction adherence, especially in visual and temporal aspects. This suggests that current models cannot effectively handle dense visual plans; generating longer videos with many keyframes remains challenging.
- **Utility for benchmarking**: KeyFrame-Compass provides a standardized benchmark with controlled factors, enabling fair comparison and diagnosing specific weaknesses (e.g., transition synthesis, temporal localization, ordering).

## Conclusion

KeyFrame-Compass establishes the first comprehensive benchmark for keyframe-conditioned video generation. It covers diverse settings (386 samples, 5 controlled factors) and evaluates both keyframe execution (6 metrics) and overall quality (4 dimensions) via an automated pipeline combining MLLM judges and specialized perception models. Experiments on 9 models reveal:

- A consistent trade-off between keyframe fidelity and natural video synthesis.
- Degradation of instruction adherence under denser keyframe

---

_Markdown view of https://picx.dev/p/HQbP63, served by PicX — AI-generated visual whiteboard summaries of research papers._