RA-Bench: A Benchmark for AI-Generated Video Detection in Real-World Crisis Events

Summary (Overview)

• Novel Benchmark: Introduces RA-Bench, the first benchmark for AI-generated video detection specifically focused on real-world crisis events, containing 17,886 videos (1,830 real-video anchors across 10 social-risk categories and 16,056 generated clips from 9 generators).

• Comprehensive Detector Evaluation: Systematically evaluates 19 detection methods across 3 families (traditional detectors, zero-shot multimodal models, and fine-tuned MLLMs), finding that none generalizes consistently across generation sources, with traditional detector AUC dropping from 67.6–98.6% (public reference) to 43.9–57.3% on RA-Bench.

• Key Vulnerability Discovery: Identifies that fine-tuned MLLM detectors (Skyra) rely heavily on temporal timestamp patterns rather than visual content—replacing timestamps with frame indices causes mean BAcc to drop from 68.5–69.5% to 54.4–54.9%.

• Human-AI Alignment: Demonstrates that videos that mislead human reviewers (RA-Bench-HumanProof, n=633) are also difficult for current detectors, with the best zero-shot model achieving only 54.7% BAcc and traditional detectors averaging 47.5% AUC on these cases.

• Social Dissemination Vulnerability: Introduces RA-Bench-LastMile simulation showing that common social media operations (downsampling, frame-rate reduction, news badges) reduce fine-tuned MLLM fake detection recall from 46.0% to 1.4%, revealing severe fragility in real-world deployment.

---

Introduction and Theoretical Foundation

Background and Motivation

Recent advances in video generation models (e.g., Sora, Seedance, Kling) have produced systems capable of generating highly realistic videos with coherent appearance and motion. While these advances offer creative and productive benefits, they also create significant societal risks when used to fabricate content depicting real-world crises—wars, natural disasters, public health emergencies, and other socially consequential events.

Research Gap

The paper identifies a critical gap in existing benchmarks for AI-generated video detection:

Existing benchmarks primarily focus on general video content, lacking systematic analysis of detector performance and generator behavior in the context of real-world crises where the stakes of misinformation are highest.

Theoretical Foundation

The benchmark is built on the insight that real video anchors from actual crisis events provide a more realistic and challenging testbed than synthetic or generic video collections. By conditioning image-to-video (I2V) generators on real first frames, RA-Bench simulates a common misinformation scenario: using a real still image from a crisis to fabricate a plausible video of subsequent events.

---

Methodology

Benchmark Construction Pipeline

The five-stage RA-Bench construction pipeline:

1. Source Data Collection: 675 publicly available videos depicting real-world crises, organized into 10 social-risk categories (L1) and 44 subcategories (L2).

2. Automated Preprocessing: Scene segmentation using PySceneDetect, producing 5,774 reviewable clips; near-duplicate prefiltering using ResNet-18 features.

3. Manual Review: Two-round human review process:

   • Round 1: Seven reviewers assess visual quality, semantic fit, duration suitability, and source risks
   • Round 2: Four adjudicators resolve disagreements (2,426 clips retained)

4. Postprocessing: Duration bounding (3–15 seconds), H.264 re-encoding, homogeneity pruning, and licensing review yielding 1,830 final clips (mean duration 10.08s).

5. Paired Video Generation: Each real clip paired with I2V-generated counterparts using:

   • Gemini-3.1-Pro-Preview captioning
   • First-frame conditioning
   • Dynamic duration (2–8 seconds, proportional to real clip)
   • Four open-source and five closed-source generators

Detector Evaluation Protocol

Three detector families were evaluated:

Traditional Detectors (7): CNNSpot, NPR, UnivFD, ForgeLens (image/frame-level); DeCoF, D3, ReStraV (video/temporal-level)

Zero-Shot Multimodal Models (10 configurations): Qwen3.5 (0.8B–122B variants), Gemini-3.1-Pro-Preview, GPT-5.5, Qwen3.7-Plus under three prompt settings (Binary, Diagnostic, Rating)

Fine-Tuned MLLMs (2): Skyra-SFT, Skyra-RL (with official timestamp and frame-index variants), BusterX++

---

Empirical Validation / Results

Detector Generalization

Traditional Detectors: Performance drops dramatically from public references:

• Seven-detector mean AUC: 50.9–57.3% (open-source), 43.9–54.0% (closed-source)
• 26 of 63 detector-source pairs fall below 50% AUC
• Spearman correlation between public and RA-Bench rankings: only 0.26

Zero-Shot Multimodal Models:

• Best model (Gemini-3.1-Pro-Preview): 63.4% BAcc (Binary prompt)
• Prompt format can reverse class preference (e.g., Qwen3.5-0.8B: 19.7% → 97.8% FakeR across prompts)
• Scaling does not eliminate prompt sensitivity

Fine-Tuned MLLMs:

Generation Properties

Quality Associations (within-source, per-interquartile increase):

• Condition Fidelity: −14.4 points Gemini Diagnostic FakeR
• Dynamic Degree: +5.5–8.4 points fake evidence (Gemini), +0.1 points (traditional)

Real-Image Conditioning (T2V → I2V → I2V+):

Seed Stability: Source-level patterns stable across seeds (max AUC range: 1.05 points; max FakeR range: 2.46 points)

Human Perception and Social Dissemination

Human Recognition (20 reviewers, source-unaware):

• Open-source generators: 68.6% FakeR
• Closed-source generators: 52.9% FakeR
• Seedance2.0: 40.7% FakeR (most deceptive)

RA-Bench-HumanProof (633 human-deceptive videos):

RA-Bench-LastMile Social Dissemination (Full condition):

• Traditional detectors: mean AUC falls 51.4% → 47.3%
• Fine-tuned MLLMs: mean FakeR falls 46.0% → 1.4% (near-complete collapse)
• Individual operations: downsampling (7.9% FakeR), frame-rate reduction (temporal-grid interaction), news badge (14.0% FakeR)

---

Theoretical and Practical Implications

Theoretical Contributions

1. Conditional Reliability: Demonstrates that detector reliability cannot be captured by a single benchmark score—performance is conditional on generation source, prompt format, protocol, and class prior.

2. Protocol Confounds: Identifies temporal-label representation as a previously unrecognized confound in MLLM-based video detection, highlighting the need for controlled ablations in detector evaluation.

3. Human-AI Overlap: Provides quantitative evidence that human-deceptive videos overlap with detector-difficult videos, suggesting shared underlying difficulty factors.

Practical Implications

1. Deployment Risk: Current detectors are not ready for unsupervised deployment in crisis scenarios—social dissemination operations can collapse fake detection recall to near zero.

2. Evaluation Standards: The benchmark provides concrete protocols for evaluating source generalization, prompt sensitivity, protocol dependence, and class bias separately.

3. Detection Priorities: Results suggest prioritizing:

   • Multi-cue fusion (temporal, spatial, and semantic evidence)
   • Robustness to post-processing operations
   • Active watermarking co-design
   • Audio-visual joint detection

---

Conclusion

RA-Bench represents a significant contribution to AI-generated video detection research by focusing on the critical use case of real-world crisis events. The benchmark's comprehensive evaluation reveals that current detection methods—across all three major families—fail to generalize reliably when faced with realistic, real-anchored generation scenarios.

Key findings:

1. No detector family provides consistent performance across generation sources
2. Generation quality and conditioning affect detector families differently
3. Human-deceptive videos remain difficult for all detector types
4. Social dissemination operations can reduce fine-tuned MLLM FakeR from 46.0% to 1.4%

Future directions identified by the authors include:

• Developing detectors robust to evolving generators and post-processing
• Creating interpretability benchmarks for MLLM-based detection evidence
• Integrating active watermarking with passive detection
• Extending to audio-visual detection settings
• Using detection as a reward signal for improving generation realism

The benchmark, code, and protocols are publicly available, providing the research community with tools to address the growing challenge of AI-generated video misinformation in high-stakes real-world scenarios.