Summary (Overview)

• New task definition: The paper introduces Video-DeepResearch (Video-DR), a paradigm requiring agents to perform complex, multi-hop reasoning over continuous video streams by combining dense spatiotemporal visual grounding with open-web exploration.
• Identified failure modes: A preliminary study reveals two critical bottlenecks in existing agents: (1) modality bias—models strongly prefer text search and rarely invoke visual tools (e.g., only 0.10 vision tool calls per task by the best open-source model), and (2) parametric knowledge leakage—models like GPT-5 achieve high accuracy with almost zero tool calls, relying on internal memory rather than genuine tool-augmented execution.
• Proposed framework: VIDEO-DEEPRESEARCH introduces a decoupled perception-exploration pipeline with stage-wise tool unlocking, a scalable data engine producing 30K video-grounded QA pairs and 7K curated trajectories, and a two-stage training recipe (SFT followed by GRPO).
• State-of-the-art results: Video-DeepResearch-35B-A3B achieves 64.0% average accuracy, surpassing Claude-4.5-Sonnet (59.0%) by 5.0 points and outperforming GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%). The smaller 30B-A3B variant reaches 59.3%, competitive with Claude-4.5-Sonnet.
• New benchmark: VIDEODR-BENCH, a 200-instance multi-hop VQA benchmark built via human–AI collaboration, where every question provably requires both visual search and external knowledge reasoning.

Introduction and Theoretical Foundation

The paper positions Video-DeepResearch as a natural evolution of deep research agents from text-only web agents (e.g., WebGPT, AutoGPT) to image-centric multimodal systems (e.g., Web-Watcher, Vision-DeepResearch). While prior work addresses static images, continuous video streams introduce a far more challenging setting where text-based synthesis and dense visual tracking are deeply intertwined. The authors identify two fundamental bottlenecks:

1. Data synthesis: Conventional video datasets focus on localized captions or short-term action labels, but Video-DR requires generative data curation that couples long-horizon decision trajectories with dense, time-varying textual and visual evidence.
2. Evaluation: Standard VQA metrics are insufficient for measuring multi-step strategic execution, creating the need for multi-dimensional evaluation protocols that quantify long-term reasoning consistency and error-recovery under continuous temporal dynamics.

A naive formulation treats video as a temporal composition of key entity trajectories. The agent is equipped with two fundamental visual tools:

• Select_Keyframe: isolates informative moments from the continuous stream.
• Crop_Search: crops salient entities from keyframes to construct precise visual queries.

Formally, given a complex research query $Q$ and visual input $V = \{ v_1, v_2, \ldots, v_T \}$ (a continuous video stream or pre-sampled keyframes), the agent generates action $a_i$ based on historical trajectory $\mathcal{H}_i$:

$$a_i \sim \pi_{\theta}(a \mid \mathcal{H}_i) \tag{1}$$

where $\pi_\theta$ denotes the multi-modal policy and observations $o$ come from environment execution. The action space includes $\mathcal{A} = \{\mathrm{Select\_Keyframe}, \mathrm{Crop\_Search}, \mathrm{Text\_Search}, \ldots\}$.

The empirical study on an existing Video-DR benchmark (Table 1) validates the two core hypotheses:

• Finding 1: Severe modality bias—even the strongest open-source model invokes visual tools only 0.10 times per task while using text tools 1.27 times.
• Finding 2: Parametric knowledge leakage—GPT-5 scores 57 with virtually zero tool calls.

Methodology

3.1 VQA Generation

The pipeline synthesizes foundational Video QA pairs from raw videos:

• Step 0: Multi-Domain Video Filtering — Videos are collected from established video datasets and real-world streaming platforms. A rule-based filter removes instances outside duration thresholds, followed by an agentic filtering stage where Qwen3.5-35B-A3B assesses content complexity to discard uninformative or overly simplistic videos.
• Step 1: Keyframe Selection and Visual Search — CLIP-based inter-frame similarity proposes candidate frames; Qwen3.5-397B-A17B finalizes keyframes $v_t$ and predicts bounding boxes $B$ to localize entities $e$. Cropped entities are used for visual search queries, and a secondary model (Qwen3.5-35B-A3B) verifies semantic alignment. Successful instances are stored as metadata tuple $\langle v_t, B, \text{entity name}, \text{search summary} \rangle$.
• Step 2: VQA Generation and Verification — QA pairs are synthesized via single-entity (fact-based) and multi-entity (compositional reasoning) patterns. Superficial attribute queries are penalized. To filter parametric leakage, four tool-free rollouts are performed; if the agent answers correctly in any attempt, the instance is discarded. This yields 30K vqa pairs.

3.2 Trajectory Generation

To overcome modality bias, a decoupled trajectory construction pipeline separates visual perception from web exploration:

• Trajectories are generated using Qwen3.5-397B-A17B with rejection sampling.
• Stage-wise tool unlocking: In the initial phase, the agent is restricted to vision-only actions (Select_Keyframe, Crop_Search), forcing extensive cross-frame, cross-entity visual retrieval. Once visual context is sufficient (or a maximum perception horizon is reached), the action space expands to include text tools (Search, Visit) for deriving the final answer.
• Only successfully resolved trajectories are retained, yielding 7K correct trajectories.

3.3 Training

Two base models are used with identical training procedures:

• Video-DeepResearch-30B-A3B: Qwen3-VL-30B-A3B-Instruct.
• Video-DeepResearch-35B-A3B: Qwen3.5-35B-A3B.

Stage 1 — Supervised Fine-Tuning (SFT): The model is trained on the 7K curated trajectories to internalize the decoupled perception-exploration paradigm, plus an additional 7K text-only QA instances from VDR to reinforce fundamental deep research capabilities. The SFT objective is standard autoregressive negative log-likelihood:

$$\mathcal{L}_{\mathrm{SFT}} = - \mathbb{E}_{(x, y) \sim \mathcal{D}} \left[ \sum_{i=1}^{|y|} \log \pi_{\theta}(y_i \mid x, y_{< i}) \right] \tag{2}$$

Stage 2 — Group Relative Policy Optimization (GRPO): A 2K moderate-difficulty RL dataset is constructed by executing four rollouts per trajectory and retaining instances with Pass@4 score between 0 and 1. A sparse binary reward ($r=1$ for correct answers judged by Qwen3-VL-30B-A3B-Instruct, else $r=0$) is used. Formatting violations or repetitive loops have their negative gradients down-sampled (applied with 20% probability). The objective is:

$$\begin{array}{l} \mathcal{L}_{\mathrm{GRPO}} = \frac{1}{G} \sum_{i=1}^{G} \left[ \min \left(\frac{\pi_{\theta}(o_i)}{\pi_{\mathrm{old}}(o_i)} \hat{A}_i, \right. \right. \\ \left. \operatorname{clip} \left(\frac{\pi_{\theta}(o_i)}{\pi_{\mathrm{old}}(o_i)}, 1 - \epsilon, 1 + \epsilon\right) \hat{A}_i\right) \bigg ] - \beta \mathbb{D}_{\mathrm{KL}} \end{array} \tag{3}$$

Experiments are conducted on four NVIDIA H800 (80GB) GPU nodes.

3.4 VIDEODR-BENCH

The benchmark is built through a scalable human-in-the-loop annotation framework:

1. Human annotators pause at critical timestamps, use Crop_Search to query salient visual entities, and verify consistency between retrieved evidence and original frames.
2. Annotators formulate seed VQA pairs, which are expanded by a multi-agent pipeline:
   • A Drafting Agent brainstorms semantic directions (e.g., expanding "LeBron James" to team, spouse, MVP awards) and queries a search engine.
   • A QA Generation Agent formulates novel multi-hop questions from retrieved web contexts.
   • Parametric leakage filtering removes questions answerable without tools.
   • Human annotators verify answerability; a Ranking Agent scores candidates, retaining the highest-rated instance.
   • The top-ranked VQA can recursively serve as a new seed for increasingly complex higher-hop reasoning.

Video length distribution:

Empirical Validation / Results

Main Results

Evaluation is conducted under both Direct (tool-free) and Agentic (full tool suite) settings on VideoDR and VIDEODR-BENCH. Correctness is judged by Qwen3-VL-30B-A3B-Instruct using the official judge prompt from Tongyi DeepResearch.

Key observations:

• Video-DeepResearch-35B-A3B sets a new state-of-the-art (64.0%), surpassing Claude-4.5-Sonnet by 5.0 points and achieving the highest VIDEODR-BENCH score (65.4% overall).
• Video-DeepResearch-30B-A3B achieves 59.3%, competitive with Claude-4.5-Sonnet and significantly above GPT-5 and Gemini 2.5 Pro.
• Scale is not sufficient: GPT-5 (52.5%) lags significantly, and Qwen3.5-397B-A13B performs comparably to the 30B trained model, showing that specialized training pipelines are essential.
• Category analysis: Both models excel in Knowledge (66.1% for 35B) and Entertainment (61.4%/65.9%); the News category (41.7% for 35B vs. 58.3% for 30B) shows potential brittleness on temporally dynamic content.

Tool Usage Analysis

• VIDEODR-BENCH forces significantly more visual and text operations than VideoDR (e.g., GPT-5's visual calls increase from 0.00 to 0.31, text from 0.12 to 1.43), demonstrating the benchmark successfully circumvents parametric leakage.
• Video-DeepResearch-30B uses more tools (2.33 visual, 4.24 text on VideoDR) than even the 397B baseline, confirming that training methodology outweighs raw parameter count.
• The strong correlation between tool usage diversity and task performance validates the core hypothesis that effective Video-DR agents must overcome modality bias.

Ablation Study

• Video-centric SFT (Base → 7K-SFT): +12.5% gain, confirming visual grounding data is strictly necessary.
• Text-augmented SFT (+7k-text-SFT): additional +3.8% improvement, validating that textual exploration data mitigates tool-invocation bias.
• **RL optimization