Summary (Overview)

• VibeVideoBench is introduced as the first benchmark for evaluating general multimodal agents on code-driven video generation with audio, comprising 239 self-contained tasks in English and Chinese across 12 categories, requiring agents to deliver finished MP4 files with sound within one hour.
• The benchmark uses a hierarchical, evidence-based evaluation protocol: program checks (L1), ~13k task-specific rubric checks (L2) traced to specific requirements, and five quality dimensions (L3) scored by three VLM judge panels with modality routing.
• Key findings: GPT-6 Astra and Claude Opus 5.5 rank first and second with both judges and human annotators; the best system satisfies every core requirement on only 76.3% of tasks, indicating the benchmark is far from solved.
• Delivery reliability is a second axis: counting missing videos as zero changes rankings; 75% of lost videos result from final renders starting too late, running too slowly, or never starting.
• Judge reliability: The three-panel mean ranks systems as humans do (Spearman ρ=0.98), but agrees with humans on only 73% of individual video pairs, highlighting limitations for per-video verdicts.

---

Introduction and Theoretical Foundation

The paper addresses a critical gap in video generation evaluation. While diffusion models [7, 21, 32, 45, 51, 56] produce visually striking clips, they cannot reliably render:

• Exact formulas and correct charts
• Rule-consistent games or state changes
• Narration synchronized to on-screen events
• Multi-minute structured content

These are natural properties of programs, not pixel generators. The authors note that humans already make such videos with tools like Manim or Remotion, and coding agents (Claude Code, Codex) can now work autonomously for hours, driving complete toolchains (renderers, headless browsers, TTS engines, ffmpeg).

Why existing evaluations fail:

The task formulation is formalized as a tuple:

$$\tau = (p, \mathcal{A}, d, a, c)$$

where $p$ is a natural-language specification, $\mathcal{A}$ is an optional read-only asset package (data, fonts, sprites, 3D scans, audio), $d$ is a machine-readable delivery spec (resolution, frame rate, duration range $[t_{min}, t_{max}]$, optional exact frame count), $a$ is an audio policy marking four channels—narration, music, sound effects, ambience—as required, optional, or not used, and $c$ is an explicit creative-freedom statement.

---

Methodology

Task Construction

Tasks are seeded from four real-world sources:

1. Real user requests to code-video tools (rewritten)
2. Real videos (seeded from dense captions)
3. Real asset packs (3D scans, game levels, archival records)
4. Lecture notes and research papers

Contract specification: LLM drafters rewrite source material under a shared contract that fixes every fact, number, and object identity, divides videos into stages with dwell times, and states a four-channel audio policy while protecting creative freedom.

Human curation: Humans select checkable tasks, rewrite prompts to be self-contained, verify facts, and remove duplicates. The result: 239 tasks in 12 categories and 43 subcategories, from 8 seconds to 15 minutes.

Evaluation Protocol

Three layers with different executors:

L1 - Program checks (decided by code, no model involved):

• Delivery spec: container, decodability, resolution, frame rate (±0.01), duration within $[t_{min} - 0.5, t_{max} + 0.5]$ seconds
• 205 core tasks add: delivered source project, exact frame count (54 tasks), audible audio stream (peak > −50 dBFS)
• Total: 1,449 L1 checks per system

L2 - Rubric checks (12,985 total, ~54 per task):

• Each check quotes its basis (the requirement it enforces)
• Marked core or detail
• Requires process shown, not result claimed
• Tagged by modality (visual, audio, audio-visual)
• Metrics: L2 pass rate = #pass/(#pass+#fail); L2 strict = no failed core check with ≥1 passed

L3 - Quality scores (five 1–5 dimensions):

• Visual Quality (VQ), Visual Aesthetics (VA), Content Logic (CL), Timing and Motion (TM), Audio Quality and A/V sync (AQ)
• Each dimension anchored at 1, 3, and 5 with task-specific descriptions

Judge Panels

Three independent VLM panels judge each video, never seeing system identity:

Critical design choice: Audio checks are routed only to judges that can hear, since many VLMs cannot process audio.

---

Empirical Validation / Results

Main Results

Failure Analysis (224 non-delivered runs)

Finding: Videos are lost to time management

• 169 failures (75%): final renders that started too late, ran too slowly, or never started
• 45 failures (20%): stalled renders (exclusively in Gemini systems—33 deadlocks from piping frames into ffmpeg without reading the error stream; one stopped at frame 7,744 of 10,102)
• 7 failures (3%): outputs the agent destroyed itself (e.g., DeepSeek re-encoded in place 1 minute before deadline, leaving a truncated 0.26 MB file)
• Successful systems leave margin: Opus finishes a median of 24 minutes before deadline vs. 10–11 minutes for Sol and GLM

Process Analysis

Renderer preferences vary by system:

• Claude agents and Gemini 4 Argon: cairo (42–54%)
• GPT-5.6 Sol, Kimi, GLM: PIL/numpy rasterization (43–55%)
• Astra, DeepSeek, Gemini 3.8 Flash: browser canvases and three.js (52–60%)

Self-inspection does not predict quality: 95% of runs view their own frames, but correlation with quality is ρ=−0.07. Astra looks least (median 3 images/run) and scores best; DeepSeek looks most (median 35).

Judge-Human Agreement

Key insight: Agreement depends on margin—63% when score gap < 0.2, but 86% when gap ≥ 0.5. Judge scores order videos but do not certify acceptability; on pairs humans label "both bad," panels give scores (3.81) identical to other pairs.

Cost Analysis

• Mean API cost per task ranges from $0.32 (GLM-5.3-Flash)** to **$15.38 (Claude Opus 5.5)—a 49× spread
• Prompt-cache writes account for 76% of Opus's cost and 48% of Sonnet's
• Rank correlation between cost and score: ρ=0.55 (permutation p=0.13)
• Pareto frontier: GLM-5.3-Flash ($0.32, 2.79), DeepSeek-V4.1-Flash ($0.47, 3.91), GPT-6 Astra ($8.19, 4.11)
• Output tokens per task: 61k (Astra) to 289k (Sonnet)

---

Theoretical and Practical Implications

For Evaluation Methodology

The paper introduces several significant advances:

1. Evidence-based grading: L2 checks separate "process shown" from "result claimed," preventing judges from crediting unverifiable self-declarations
2. Modality routing: Audio checks never reach judges that cannot hear—a critical correction given many VLMs' audio limitations
3. Dual reporting: Delivered-only scores separate quality from reliability; no-video-as-zero combines them
4. Missing videos handled explicitly: This alone changes rankings substantially

For Agent Design

• Time management is a core competence: The finding that 75% of failures stem from poorly managed final renders suggests agents need better temporal planning heuristics
• Self-inspection alone is insufficient: Looking at frames doesn't predict quality; agents need better verification strategies (Opus's "finish complete deliverable, then improve copies" strategy works)
• Robustness issues identified: Pipes deadlocking (ignoring error streams), background processes being cut off, and in-place re-encoding are actionable failure patterns

For Benchmark Design

• Judge panels can certify system rankings over hundreds of tasks but cannot yet be trusted for per-video verdicts—critical for anyone using VLM judges for training rewards
• The sensitivity of strict metrics to judge strictness (Opus L2 strict: 83.7%, 93.3%, 51.9% across panels) suggests single-panel evaluations may be unreliable

---

Conclusion

VibeVideoBench establishes that code-driven video generation is a distinct and measurable paradigm from diffusion-based generation, one that naturally satisfies requirements for exact text, rule-consistent state changes, multi-minute structure, and synchronized audio. The benchmark's three-layer, evidence-based evaluation protocol with modality routing represents a methodological advance for evaluating perceptual, temporal, and audio-visual agent outcomes where unit tests do not exist.

Key takeaways:

1. The task is far from solved—even the best system fails core requirements on ~25% of tasks
2. Delivery reliability is as important as output quality in real deployments
3. Agent trajectory analysis reveals actionable failure patterns (time management, renderer choice, verification habits)
4. Current VLM judges can rank systems but need margin requirements or multi-panel consensus for per-video decisions

Future directions implied by the work include: developing better temporal planning for agents, improving verification strategies (particularly for audio and timing), and training judges that achieve higher per-video agreement with humans to enable their use in training-time rewards.