OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents

Summary (Overview)

• OpenSearch-VL introduces a fully open-source recipe for training frontier multimodal deep search agents, addressing the reproducibility gap in proprietary multimodal search systems by releasing all data, code, and models.
• The paper proposes a three-stage data curation pipeline based on Wikipedia hyperlink graph sampling, fuzzy entity rewriting, and source-anchor visual grounding that jointly mitigates shortcut learning and one-step retrieval collapse.
• A diverse tool environment is designed that unifies text search, image search, OCR, cropping, sharpening, super-resolution, and perspective correction, enabling agents to combine active perception with external knowledge acquisition.
• The authors develop a multi-turn fatal-aware GRPO algorithm that handles cascading tool failures by masking post-failure tokens while preserving useful pre-failure reasoning through one-sided advantage clamping.
• Experimental results show over 10-point average improvements across seven benchmarks (e.g., from 47.8 to 61.6 average score for the 30B-A3B variant), achieving performance comparable to proprietary commercial models on several tasks.

Introduction and Theoretical Foundation

Background and Motivation

Multimodal deep search has emerged as a critical capability for multimodal large language models (MLLMs), enabling them to evolve from passive visual understanding systems into agents that actively search evidence, verify facts, and reason over knowledge-intensive visual queries. However, frontier multimodal search agents remain difficult to reproduce because their training data, code, and trajectories are often proprietary or insufficiently disclosed.

Key Challenges Addressed

1. Data bottleneck: High-quality training data is a central bottleneck, as the strongest frontier systems are dominated by commercial corporations where data sources, filtering criteria, and tool-use trajectories are kept private.

2. Agentic RL challenges: Training multimodal search agents with agentic reinforcement learning in long-horizon tool-use settings poses unique challenges—a single malformed call, timeout, or repeated failure can invalidate the remaining rollout.

3. Imperfect visual inputs: Real-world visual inputs are often imperfect (blurred photos, low-resolution thumbnails, skewed documents), requiring agents to first crop, enhance, rectify, or parse visual evidence before reliable search can begin.

Problem Formulation

Given an input image $I_0$ and a question $q$, the agent answers $q$ by interleaving reasoning with tool calls over a diverse tool set $\mathcal{T} = \mathcal{T}_v \cup \mathcal{T}_s$, where $\mathcal{T}_v$ contains visual tools and $\mathcal{T}_s$ contains retrieval tools. At step $l$, the model conditions on the accumulated history:

$$h_l = \left(\mathcal{I}_l, q, \mathbf{a}_{<l}, \mathbf{o}_{<l}\right) \tag{1}$$

The interaction unfolds as a multi-turn trajectory:

$$\tau = \left\{\left(h_0, a_0, o_0\right), \left(h_1, a_1, o_1\right), \dots, \left(h_{L-1}, a_{L-1}, o_{L-1}\right), \left(h_L, a_L\right)\right\} \tag{2}$$

Following the ReAct think-then-act convention, each action decomposes as $a_l = [z_l, c_l]$ where $z_l$ is a reasoning trace and $c_l$ denotes a tool invocation (for $l < L$) or the final response (for $l = L$).

Methodology

1. Data Curation Pipeline

The pipeline proceeds in three stages:

Stage 1: High-Quality VQA Construction

Wikipedia Path Sampling: Cast Wikipedia as a directed graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ with articles as nodes and hyperlinks as edges. A constrained random walk of length $h \in \{2, 3, 4\}$ produces a path:

$$P = \left(v_0 \xrightarrow{\rho_1} v_1 \xrightarrow{\rho_2} \dots \xrightarrow{\rho_h} v_h\right) \tag{5}$$

Each node is assigned a functional role: $v_0$ is the anchor (visual entry point), $v_1, \ldots, v_{h-1}$ are bridge nodes (intermediate entities with fuzzified names), and $v_h$ is the answer node.

Fuzzy Entity Rewriting: Progressively rewrites entity names into fuzzy descriptors while preserving answer invariance, uniqueness, and non-leakage:

$$\underbrace{a(q_f) = a(q_t)}_{\text{answer invariance}}, \qquad \underbrace{|\mathcal{R}(q_f)| = 1}_{\text{uniqueness}}, \qquad \underbrace{\left(\bigcup_{j=0}^{h} \text{aliases}(v_j)\right) \cap q_f = \emptyset}_{\text{non-leakage}} \tag{6}$$

Anchor-aware Visual Grounding: Retrieves a representative image of the anchor entity $v_0$ and replaces it with a visual referring expression (e.g., "the person in the image"), substantially reducing single-hop shortcuts.

Stage 2: Filtering and Enhancement

• Two-stage difficulty filter using a frozen Qwen3-VL-32B: discards examples answerable without tools, then discards examples solvable with a single ImageSearch call
• 10% of the filtered pool receives controlled degradations (blur, downsampling, perspective distortion) paired with enhancement tools to induce "think-with-image" behavior

Stage 3: Multi-turn Trajectory Synthesis

• Rolls out Claude Opus 4.6 as expert model with $K=5$ independent rollouts per instance
• Two-stage rejection cascade: answer correctness check, then process-level judge on tool-use quality

This yields 36,592 high-quality expert trajectories with an average of 6.3 tool-invocation turns per trajectory.

2. Tool Environment

3. Training

Stage 1: Supervised Fine-Tuning (SFT)

The SFT objective jointly supervises both reasoning trace and tool invocation:

$$\max_{\theta} \sum_{i=1}^{N} \sum_{l=1}^{L_i} \left[ \log P_{\theta}\left(z_l^{(i)} \mid h_l^{(i)}\right) + \log P_{\theta}\left(c_l^{(i)} \mid h_l^{(i)}, z_l^{(i)}\right) \right] \tag{8}$$

Stage 2: Multi-Turn Search Fatal-Aware GRPO

Composite Reward:

$$r(\tau) = r_{\text{fmt}}(\tau) \cdot \left[ \alpha r_{\text{acc}}(\tau) + (1 - \alpha) r_{\text{query}}(\tau) \right] \tag{9}$$

where $\alpha = 0.8$, $r_{\text{fmt}}$ is a format reward enforcing structural integrity, $r_{\text{acc}} \in \{0, 1\}$ is accuracy reward from a GPT-4o judge, and $r_{\text{query}} \in [0, 1]$ is a process-level query-quality reward from GPT-5.4.

Fatal-Aware Token Masking:

$$M\left(y_{i,t}\right) = M_{\text{gen}}\left(y_{i,t}\right) \cdot \mathbb{1}\left[s(t) < f_i\right] \tag{10}$$

where $f_i$ is the earliest step where $K=3$ consecutive tool-execution errors commence.

One-Sided Advantage Clamping:

$$\hat{A}_i = \begin{cases} \widetilde{r}_i & \text{if } f_i = L_i + 1 \text{ (non-fatal)}, \\ \max(\widetilde{r}_i, 0) & \text{if } f_i \leq L_i \text{ (fatal)}. \end{cases} \tag{11}$$

Final GRPO Objective:

$$\mathcal{J}(\theta) = \mathbb{E}_{\substack{(I_0,q)\sim\mathcal{D}\\ \{\tau_i\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|I_0,q;\mathcal{E})}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{\sum_{t}M_{i,t}}\sum_{t=1}^{|\tau_i|}M_{i,t}\min\left(\rho_{i,t}(\theta)\hat{A}_i, \text{clip}_{1-\epsilon}^{1+\epsilon}(\rho_{i,t}(\theta))\hat{A}_i\right)\right] \tag{12}$$

Empirical Validation / Results

Main Results (Table 2)

Key findings:

• OpenSearch-VL-8B achieves 56.6 average, surpassing SenseNova-MARS-8B by 3.9 points
• OpenSearch-VL-32B outperforms Gemini-2.5-Pro on direct reasoning tasks
• Large gains on VDR (+13.3), MMSearch (+24.5), FVQA (+10.2), and InfoSeek (+16.2)

Ablation Studies (Table 3)

Data Pipeline Ablation (relative to full pipeline):

• w/o source-anchor grounding: -11.5 points average
• w/o fuzzy entity rewriting: -10.3 points average
• w/o staged filtering: -8.2 points average
• w/o enhancement subset: -1.3 points average

RL Recipe Ablation:

The full method achieves a 4.2-point gain over vanilla GRPO, demonstrating that fatal-aware masking with one-sided clamping is crucial for effective RL.

Statistical Analysis of Clamping

Aggregated over 10,000 groups: 91.8% of fatal rollouts fall on the negative side of the clamp threshold (mean $\overline{\widetilde{r}}_{--} = -0.68$) and are zeroed out; the remaining 8.2% are preserved (mean $\overline{\widetilde{r}}_{+} = +0.57$) and overlap with the positive mode of the non-fatal reference distribution.

Theoretical and Practical Implications

Theoretical Contributions

1. Fatal-aware GRPO framework: The paper generalizes search-augmented GRPO to multimodal environments with diverse tools, introducing a principled way to handle cascading tool failures through token masking and one-sided advantage clamping. This extends the theoretical foundations of RL for agentic reasoning.

2. Data curation theory: The three-stage pipeline formalizes the requirements for effective multimodal search training data: answer invariance, uniqueness, and non-leakage (Eq. 6), providing a theoretical framework for understanding what makes training data genuinely tool-demanding.

3. Credit assignment in long-horizon tasks: The one-sided clamping mechanism provides a principled solution to the credit assignment problem in partially successful trajectories, preserving useful pre-failure reasoning without amplifying noise from failed rollouts.

Practical Implications

1. Reproducibility: All data, code, and models are released, lowering the reproducibility barrier for frontier multimodal search agent research.

2. Active perception: The tool environment demonstrates that combining search with visual pre-processing tools (sharpening, super-resolution, perspective correction) is essential for real-world robustness.

3. Scalability: The recipe scales effectively from 8B to 32B models, suggesting broad applicability across model sizes.

Conclusion

OpenSearch-VL provides a fully open recipe for training multimodal deep search agents with agentic reinforcement learning. The key components are:

• A Wikipedia-based data curation pipeline producing SearchVL-SFT-36k and SearchVL-RL-8k datasets
• A diverse tool environment spanning retrieval, image enhancement, and attention-and-parsing tools
• A multi-turn fatal-aware GRPO algorithm preserving useful pre-failure reasoning through one-sided advantage clamping

The approach achieves over 10-point average gains across seven multimodal deep search benchmarks, with competitive performance on representative tasks compared with strong proprietary reasoning models.

Limitations and Future Work

• Tool environment instability: Search ranking drift, fetch failures, and summarization hallucinations inflate reward variance
• Proprietary judge dependence: The composite reward relies on GPT-4o judges, which are costly and version-dependent
• Limited visual action scoring: Current process rewards only score textual queries, ignoring intermediate visual operations (e.g., Crop)
• Reproducibility challenges: Reliance on external APIs (Serper, PaddleX OCR) and prohibitive multi-seed evaluation costs

Future directions include replacing proprietary judges with open process reward models covering the full visual action space $\mathcal{T}_v$, and developing on-policy reliability estimation techniques.