# Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

> Video-DeepResearch-35B-A3B achieves 64.0% accuracy on multi-hop video reasoning, surpassing Claude-4.5-Sonnet by 5 points via decoupled perception-exploration and stage-wise tool training.

- **Source:** [arXiv](https://arxiv.org/abs/2608.03979)
- **Published:** 2026-08-06
- **Permalink:** https://picx.dev/p/I0NEeE
- **Whiteboard:** https://picx.dev/p/I0NEeE/image

## Summary

## Summary (Overview)

- **New task definition**: The paper introduces **Video-DeepResearch (Video-DR)**, a paradigm requiring agents to perform complex, multi-hop reasoning over continuous video streams by combining dense spatiotemporal visual grounding with open-web exploration.
- **Identified failure modes**: A preliminary study reveals two critical bottlenecks in existing agents: (1) **modality bias**—models strongly prefer text search and rarely invoke visual tools (e.g., only 0.10 vision tool calls per task by the best open-source model), and (2) **parametric knowledge leakage**—models like GPT-5 achieve high accuracy with almost zero tool calls, relying on internal memory rather than genuine tool-augmented execution.
- **Proposed framework**: **VIDEO-DEEPRESEARCH** introduces a decoupled perception-exploration pipeline with stage-wise tool unlocking, a scalable data engine producing **30K video-grounded QA pairs** and **7K curated trajectories**, and a two-stage training recipe (**SFT followed by GRPO**).
- **State-of-the-art results**: **Video-DeepResearch-35B-A3B** achieves **64.0%** average accuracy, surpassing Claude-4.5-Sonnet (59.0%) by 5.0 points and outperforming GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%). The smaller **30B-A3B** variant reaches **59.3%**, competitive with Claude-4.5-Sonnet.
- **New benchmark**: **VIDEODR-BENCH**, a 200-instance multi-hop VQA benchmark built via human–AI collaboration, where every question provably requires both visual search and external knowledge reasoning.

## Introduction and Theoretical Foundation

The paper positions Video-DeepResearch as a natural evolution of deep research agents from text-only web agents (e.g., WebGPT, AutoGPT) to image-centric multimodal systems (e.g., Web-Watcher, Vision-DeepResearch). While prior work addresses static images, continuous video streams introduce a far more challenging setting where text-based synthesis and dense visual tracking are deeply intertwined. The authors identify two fundamental bottlenecks:

1. **Data synthesis**: Conventional video datasets focus on localized captions or short-term action labels, but Video-DR requires generative data curation that couples long-horizon decision trajectories with dense, time-varying textual and visual evidence.
2. **Evaluation**: Standard VQA metrics are insufficient for measuring multi-step strategic execution, creating the need for multi-dimensional evaluation protocols that quantify long-term reasoning consistency and error-recovery under continuous temporal dynamics.

A naive formulation treats video as a temporal composition of key entity trajectories. The agent is equipped with two fundamental visual tools:
- **Select_Keyframe**: isolates informative moments from the continuous stream.
- **Crop_Search**: crops salient entities from keyframes to construct precise visual queries.

Formally, given a complex research query $Q$ and visual input $V = \{ v_1, v_2, \ldots, v_T \}$ (a continuous video stream or pre-sampled keyframes), the agent generates action $a_i$ based on historical trajectory $\mathcal{H}_i$:

$$
a_i \sim \pi_{\theta}(a \mid \mathcal{H}_i) \tag{1}
$$

where $\pi_\theta$ denotes the multi-modal policy and observations $o$ come from environment execution. The action space includes $\mathcal{A} = \{\text{Select\_Keyframe}, \text{Crop\_Search}, \text{Text\_Search}, \ldots\}$.

The empirical study on an existing Video-DR benchmark (Table 1) validates the two core hypotheses:

| Model | Acc. | Vision Tool | Text Tools |
|---|---|---|---|
| Qwen3.5-35B-A3B | 41 | 0.04 | 0.58 |
| Qwen3.5-397B-A17B | 58 | 0.10 | 1.27 |
| GPT-5 | 57 | 0.00 | 0.12 |

- **Finding 1**: Severe modality bias—even the strongest open-source model invokes visual tools only 0.10 times per task while using text tools 1.27 times.
- **Finding 2**: Parametric knowledge leakage—GPT-5 scores 57 with virtually zero tool calls.

## Methodology

### 3.1 VQA Generation

The pipeline synthesizes foundational Video QA pairs from raw videos:

- **Step 0: Multi-Domain Video Filtering** — Videos are collected from established video datasets and real-world streaming platforms. A rule-based filter removes instances outside duration thresholds, followed by an agentic filtering stage where Qwen3.5-35B-A3B assesses content complexity to discard uninformative or overly simplistic videos.
- **Step 1: Keyframe Selection and Visual Search** — CLIP-based inter-frame similarity proposes candidate frames; Qwen3.5-397B-A17B finalizes keyframes $v_t$ and predicts bounding boxes $B$ to localize entities $e$. Cropped entities are used for visual search queries, and a secondary model (Qwen3.5-35B-A3B) verifies semantic alignment. Successful instances are stored as metadata tuple $\langle v_t, B, \text{entity name}, \text{search summary} \rangle$.
- **Step 2: VQA Generation and Verification** — QA pairs are synthesized via **single-entity** (fact-based) and **multi-entity** (compositional reasoning) patterns. Superficial attribute queries are penalized. To filter parametric leakage, four tool-free rollouts are performed; if the agent answers correctly in any attempt, the instance is discarded. This yields **30K vqa pairs**.

### 3.2 Trajectory Generation

To overcome modality bias, a **decoupled trajectory construction pipeline** separates visual perception from web exploration:

- Trajectories are generated using Qwen3.5-397B-A17B with rejection sampling.
- **Stage-wise tool unlocking**: In the initial phase, the agent is restricted to vision-only actions (Select_Keyframe, Crop_Search), forcing extensive cross-frame, cross-entity visual retrieval. Once visual context is sufficient (or a maximum perception horizon is reached), the action space expands to include text tools (Search, Visit) for deriving the final answer.
- Only successfully resolved trajectories are retained, yielding **7K correct trajectories**.

### 3.3 Training

Two base models are used with identical training procedures:
- **Video-DeepResearch-30B-A3B**: Qwen3-VL-30B-A3B-Instruct.
- **Video-DeepResearch-35B-A3B**: Qwen3.5-35B-A3B.

**Stage 1 — Supervised Fine-Tuning (SFT)**: The model is trained on the 7K curated trajectories to internalize the decoupled perception-exploration paradigm, plus an additional 7K text-only QA instances from VDR to reinforce fundamental deep research capabilities. The SFT objective is standard autoregressive negative log-likelihood:

$$
\mathcal{L}_{\mathrm{SFT}} = - \mathbb{E}_{(x, y) \sim \mathcal{D}} \left[ \sum_{i=1}^{|y|} \log \pi_{\theta}(y_i \mid x, y_{< i}) \right] \tag{2}
$$

**Stage 2 — Group Relative Policy Optimization (GRPO)**: A 2K moderate-difficulty RL dataset is constructed by executing four rollouts per trajectory and retaining instances with Pass@4 score between 0 and 1. A sparse binary reward ($r=1$ for correct answers judged by Qwen3-VL-30B-A3B-Instruct, else $r=0$) is used. Formatting violations or repetitive loops have their negative gradients down-sampled (applied with 20% probability). The objective is:

$$
\begin{array}{l} 
\mathcal{L}_{\mathrm{GRPO}} = \frac{1}{G} \sum_{i=1}^{G} \left[ \min \left(\frac{\pi_{\theta}(o_i)}{\pi_{\mathrm{old}}(o_i)} \hat{A}_i, \right. \right. \\
\left. \operatorname{clip} \left(\frac{\pi_{\theta}(o_i)}{\pi_{\mathrm{old}}(o_i)}, 1 - \epsilon, 1 + \epsilon\right) \hat{A}_i\right) \bigg ] - \beta \mathbb{D}_{\mathrm{KL}} \end{array} \tag{3}
$$

Experiments are conducted on four NVIDIA H800 (80GB) GPU nodes.

### 3.4 VIDEODR-BENCH

The benchmark is built through a **scalable human-in-the-loop annotation framework**:

1. Human annotators pause at critical timestamps, use Crop_Search to query salient visual entities, and verify consistency between retrieved evidence and original frames.
2. Annotators formulate seed VQA pairs, which are expanded by a multi-agent pipeline:
   - A **Drafting Agent** brainstorms semantic directions (e.g., expanding "LeBron James" to team, spouse, MVP awards) and queries a search engine.
   - A **QA Generation Agent** formulates novel multi-hop questions from retrieved web contexts.
   - **Parametric leakage filtering** removes questions answerable without tools.
   - Human annotators verify answerability; a **Ranking Agent** scores candidates, retaining the highest-rated instance.
   - The top-ranked VQA can recursively serve as a new seed for increasingly complex higher-hop reasoning.

Video length distribution:

| Video Type | Duration (min) | Count | Percentage |
|---|---|---|---|
| Short | ≤ 2 | 92 | 46.0% |
| Medium | 2 ~ 10 | 68 | 34.0% |
| Long | ≥ 10 | 40 | 20.0% |

## Empirical Validation / Results

### Main Results

Evaluation is conducted under both **Direct** (tool-free) and **Agentic** (full tool suite) settings on VideoDR and VIDEODR-BENCH. Correctness is judged by Qwen3-VL-30B-A3B-Instruct using the official judge prompt from Tongyi DeepResearch.

| Model | Video-DR Acc. | KNL | ENT | DLY | G&S | NWS | OTH | Overall | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| **Closed-Source Models** | | | | | | | | | |
| Gemini 2.5 Pro | 62.0 | 54.2 | 52.3 | 51.4 | 51.7 | 54.2 | 57.1 | 53.0 | 57.5 |
| GPT-5 | 57.0 | 50.8 | 45.5 | 48.6 | 48.3 | 45.8 | 42.9 | 48.0 | 52.5 |
| Claude-4.5-Sonnet | 63.0 | 55.9 | 54.5 | 54.1 | 58.6 | 54.2 | 42.9 | 55.0 | 59.0 |
| **Open-Source Models** | | | | | | | | | |
| Qwen3.5-397B-A13B | 58.0 | 49.2 | 61.4 | 40.5 | 55.2 | 29.2 | 14.3 | 47.5 | 52.8 |
| Kimi K2.5 | 61.0 | 54.2 | 52.3 | 51.4 | 51.7 | 54.2 | 57.1 | 53.0 | 57.0 |
| Qwen3-VL-30B-A3B-Instruct | 38.0 | 44.1 | 43.2 | 35.1 | 51.7 | 41.7 | 42.9 | 43.0 | 40.5 |
| **Video-DeepResearch-30B-A3B (Ours)** | 62.0 | 62.7 | 61.4 | 40.5 | 58.6 | 58.3 | 42.9 | 56.5 | 59.3 |
| Improvement↑ | +24.0 | +18.6 | +18.2 | +5.4 | +6.9 | +16.6 | 0.0 | +13.5 | +18.8 |
| Qwen3.5-35B-A3B | 42.0 | 45.8 | 50.0 | 40.5 | 44.8 | 33.3 | 28.6 | 43.5 | 42.8 |
| **Video-DeepResearch-35B-A3B (Ours)** | 68.0 | 66.1 | 65.9 | 56.8 | 62.1 | 41.7 | 42.9 | 60.0 | 64.0 |
| Improvement↑ | +26.0 | +20.3 | +15.9 | +16.3 | +17.3 | +8.4 | +14.3 | +16.5 | +21.2 |

Key observations:

- **Video-DeepResearch-35B-A3B** sets a new state-of-the-art (**64.0%**), surpassing Claude-4.5-Sonnet by 5.0 points and achieving the highest VIDEODR-BENCH score (65.4% overall).
- **Video-DeepResearch-30B-A3B** achieves 59.3%, competitive with Claude-4.5-Sonnet and significantly above GPT-5 and Gemini 2.5 Pro.
- **Scale is not sufficient**: GPT-5 (52.5%) lags significantly, and Qwen3.5-397B-A13B performs comparably to the 30B trained model, showing that specialized training pipelines are essential.
- **Category analysis**: Both models excel in Knowledge (66.1% for 35B) and Entertainment (61.4%/65.9%); the News category (41.7% for 35B vs. 58.3% for 30B) shows potential brittleness on temporally dynamic content.

### Tool Usage Analysis

| Model | VideoDR Visual | VideoDR Text | VideoDR-Bench Visual | VideoDR-Bench Text |
|---|---|---|---|---|
| Claude-4.5-Sonnet | 1.83 | 3.38 | 2.25 | 3.24 |
| GPT-5 | 0.00 | 0.12 | 0.31 | 1.43 |
| Gemini-2.5-Pro | 0.31 | 0.84 | 1.93 | 2.07 |
| Qwen3.5-35B | 0.04 | 0.58 | 0.20 | 0.70 |
| Qwen3.5-397B | 0.10 | 1.27 | 0.04 | 2.77 |
| Base | 1.82 | 2.21 | 1.75 | 2.40 |
| **Ours** | 2.33 | 4.24 | 2.98 | 3.81 |

- VIDEODR-BENCH forces significantly more visual and text operations than VideoDR (e.g., GPT-5's visual calls increase from 0.00 to 0.31, text from 0.12 to 1.43), demonstrating the benchmark successfully circumvents parametric leakage.
- **Video-DeepResearch-30B** uses more tools (2.33 visual, 4.24 text on VideoDR) than even the 397B baseline, confirming that training methodology outweighs raw parameter count.
- The strong correlation between tool usage diversity and task performance validates the core hypothesis that effective Video-DR agents must overcome modality bias.

### Ablation Study

| Setting | VideoDR | VideoDR-Bench | Avg |
|---|---|---|---|
| Base | 38.0 | 43.0 | 40.5 |
| 4k-SFT | 44.0 | 48.0 | 46.0 |
| 7K-SFT | 55.0 | 51.0 | 53.0 |
| 7K-SFT + 7k-text-SFT | 59.0 | 54.5 | 56.8 |
| 14k-SFT + 2K-RL | 62.0 | 56.5 | 59.3 |

- **Video-centric SFT** (Base → 7K-SFT): +12.5% gain, confirming visual grounding data is strictly necessary.
- **Text-augmented SFT** (+7k-text-SFT): additional +3.8% improvement, validating that textual exploration data mitigates tool-invocation bias.
- **RL optimization

---

_Markdown view of https://picx.dev/p/I0NEeE, served by PicX — AI-generated visual whiteboard summaries of research papers._