Summary

• Video2GUI is a fully automated framework that extracts grounded GUI interaction trajectories directly from unlabeled Internet videos, addressing the data scarcity problem in GUI agent training.
• The framework employs a coarse-to-fine filtering strategy (metadata-based classification followed by content-based video quality scoring) to identify high-quality GUI tutorial videos from 500 million video metadata entries.
• The authors construct WildGUI, the largest open-source GUI pre-training dataset to date, containing 12.7 million interaction trajectories and 124.5 million screenshots spanning over 1,500 applications and websites across web, mobile, and desktop platforms.
• Pre-training Qwen2.5-VL and Mimo-VL on WildGUI yields consistent improvements of 5–20% across multiple GUI grounding and action benchmarks, matching or surpassing state-of-the-art performance.
• The pipeline uses a VLM-driven trajectory extraction approach with sliding-window context memory, followed by a multi-frame action spatial grounding stage to map actions to precise screen coordinates.

---

Introduction and Theoretical Foundation

Background and Motivation

Recent advances in multimodal large language models (MLLMs) have driven growing interest in GUI agents capable of autonomously interacting with graphical user interfaces. These agents automate tasks by perceiving visual interface states and emulating human actions (clicking, typing, dragging) across diverse platforms including web, desktop, and mobile applications.

A key prerequisite for developing generalized GUI agents is access to large-scale and diverse trajectory data that precisely documents GUI interactions. However, existing approaches face fundamental limitations:

1. Manual annotation (e.g., Deka et al., 2017; Rawles et al., 2023) provides high-quality supervision but entails substantial costs and limited scalability.
2. Simulated environments (e.g., Shvo et al., 2021; Lee et al., 2024) suffer from domain gaps and restricted diversity.
3. Internet videos constitute a rich repository of real-world GUI usage but lack explicit interaction annotations and require sophisticated filtering.

Theoretical Formulation

The authors formulate GUI agent interaction as a Partially Observable Markov Decision Process (POMDP), defined by the tuple $(U, S, A, O, T)$:

• $U$: space of high-level user instructions (task descriptions)
• $S$: possible environment states
• $A$: action space of atomic GUI operations (clicking, typing)
• $O$: observation space (multimodal feedback such as screenshots)
• $T$: transition function $\mathcal{T}: \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{S}$

At each time step $t$, the agent $\pi_\theta$ receives observation $o_{t-1} \in \mathcal{O}$ and selects action $a_t \in \mathcal{A}$ based on policy $\pi_\theta(\cdot \mid e_{t-1})$, where $e_{t-1} = (u, a_1, o_1, \ldots, a_{t-1}, o_{t-1})$ denotes the interaction history. Each action is parameterized as a tuple $(\tau_t, b_t)$ where $\tau_t$ specifies the action type and $b_t$ denotes action parameters (coordinates, target elements, or input text).

---

Methodology

3.1 Coarse-to-Fine Video Filtering

Meta Info Filtering: The pipeline begins with metadata-based textual filtering using over 500 million YouTube video metadata entries. The authors:

• Use DeepSeek-V3 to annotate relevance labels for 10K samples
• Fine-tune a lightweight Qwen2.5-7B with a classification head for scalable inference
• Apply the classifier to all collected metadata, reducing candidates to ~20 million videos

The classifier is trained with cross-entropy loss:

$$\mathcal{L}_{\mathrm{CE}} = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i) \right]$$

Video Quality Scoring: A content-based scorer using Qwen2.5-Omni evaluates videos along three dimensions:

1. Topic Relevance — whether the video focuses on teaching GUI operations
2. Instruction Clarity — clarity and coherence of instructional narration
3. Recording Quality — visual clarity, completeness, and stability

The scorer is trained with MSE loss:

$$\mathcal{L}_{\mathrm{MSE}} = \frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{3} (y_{ij} - \hat{y}_{ij})^2$$

This stage retains 4.16 million videos (~300,000 hours of high-quality GUI instructional content).

3.2 Trajectory Extraction

The goal is to transform unstructured videos into task-oriented instruction–trajectory pairs:

$$\mathcal{D}(V) = \{(u^{(k)}, e^{(k)})\}_{k=1}^{N}$$

where each pair $(u^{(k)}, e^{(k)})$ corresponds to the $k$-th independent task instance. The approach uses:

• Gemini-3-Pro as the annotation model
• A sliding-window strategy with historical context memory (4-minute segments)
• Long-range memory across segments for tasks spanning boundaries
• Output of visually grounded textual descriptions (low-level instructions) for each interaction step

3.3 Action Spatial Grounding

For each interaction action at timestamp $t$, a triplet of high-resolution frames is retrieved:

$$O_t = \{o_{t-0.5s}, o_t, o_{t+0.5s}\}$$

The grounded action is defined as:

$$b_t = g_\phi(o_{t-0.5s}, o_t, o_{t+0.5s}, \tau_t)$$

The model selects the first frame yielding a valid spatial grounding result. Manual verification confirms over 95% accuracy on 200 randomly sampled actions.

3.4 Agent Training

Stage 1: Continual Pre-training — Three complementary tasks with mixed objective:

$$\mathcal{L}_{\text{pretrain}} = \mathcal{L}_{\text{ground}} + \mathcal{L}_{\text{action}} + \mathcal{L}_{\text{traj}}$$

1. GUI grounding: localize target UI elements by predicting coordinates/bounding boxes
2. GUI action prediction: predict actions from a single screenshot conditioned on task instruction
3. GUI trajectory modeling: autoregressively predict actions from chronologically arranged screenshots

Training covers approximately 200 billion tokens for one epoch.

Stage 2: Post-training — Fine-tuning on curated open-source datasets (Rico, SeeClickWeb, WebUI, OS-Atlas, AITW, AITZ, AndroidControl, AMEX, GUI-Odyssey) for 3 epochs (~15 billion tokens).

---

Empirical Validation / Results

GUI Grounding Evaluation

Table 2: Performance on ScreenSpot-Pro and OSWorld-G

Mimo-VL-7B trained on WildGUI achieves state-of-the-art performance on OSWorld-G (67.6), surpassing Qwen3-VL-32B (60.6) and Seed1.5-VL (62.9).

Offline GUI Agent Evaluation

Table 3: Performance on AndroidControl and CAGUI

Online GUI Agent Evaluation

On AndroidWorld, the full Stage 1 + Stage 2 pipeline achieves 31.9% success rate, nearly doubling the base model's 16.4% and outperforming the Stage-2-only baseline of 23.3%. On OSWorld, the model reaches 12.3% compared to 10.4% for Stage-2-only training.

Scaling Effects

Performance consistently improves with increasing pre-training tokens:

• ScreenSpot-Pro: from ~41% to 56.9% at 200B tokens
• OSWorld-G: from ~55% to 67.6% at 200B tokens
• Model surpasses Stage-2-only baseline at ~50B tokens with no evident saturation

Ablation Studies

Table 4: Ablation on training tasks for Mimo-VL-7B

Key findings:

• Removing $\mathcal{L}_{traj}$ causes significant drops on AndroidWorld (31.9 → 24.1), confirming trajectory modeling is crucial for long-horizon planning
• Removing $\mathcal{L}_{ground}$ causes substantial degradation on ScreenSpot-Pro (56.9 → 49.8)
• Removing Stage 2 causes catastrophic drops, especially on AndroidWorld (6.0)

Data Quality Assessment

Human evaluation (5 expert participants, Krippendorff's α = 0.84) shows:

• Video quality scores improve from 1.22 → 2.12 → 4.45 through the filtering pipeline
• WildGUI achieves the highest trajectory quality score of 4.62, outperforming TongUI (3.35) and VideoAgentTrek (4.05)

---

Theoretical and Practical Implications

Dataset Scale and Diversity

WildGUI represents a paradigm shift in GUI agent training data:

Key Insights

1. Data scale matters: The scaling study demonstrates a strong positive correlation between pre-training data quantity and agent performance, with no evident saturation at 200B tokens.

2. Diverse offline data generalizes online: Despite WildGUI consisting entirely of offline GUI interaction data, models show strong generalization to online, open-ended environments.

3. Complementary training objectives: Different pre-training tasks contribute to distinct capabilities — grounding loss for spatial accuracy, trajectory modeling for long-horizon planning, and action prediction for atomic execution.

4. Two-stage training is essential: Stage 1 provides broad GUI knowledge while Stage 2 aligns the model for complex instruction-following scenarios.

Cost-Efficiency

The total API cost is approximately **$0.0763 per sample** (trajectory extraction:$0.0653, spatial grounding: $0.011), with video quality scoring handled by a self-deployed open-source model at negligible cost. Dataset construction is a one-time cost since WildGUI and the pipeline are released to the community.

---

Conclusion

This paper addresses the data scarcity challenge in GUI agent training by introducing Video2GUI, a fully automated framework that synthesizes high-quality interaction trajectories from unlabeled internet videos. The key contributions are:

1. Video2GUI pipeline: A scalable, coarse-to-fine filtering approach combined with VLM-driven trajectory extraction and multi-frame spatial grounding.

2. WildGUI dataset: The largest GUI pre-training dataset with 12 million trajectories across 1,500+ applications and websites, covering web, mobile, and desktop platforms.

3. Empirical validation: Pre-training on WildGUI yields consistent improvements of 5–20% across GUI grounding and agent benchmarks for both Qwen2.5-VL and Mimo-VL, matching or surpassing state-of-the-art performance.

4. Open resources: The WildGUI dataset and Video2GUI pipeline will be released to facilitate future research on scalable GUI agent training and evaluation.

The results validate that scaling training with diverse, offline video data provides a promising pathway toward generalized GUI agents. Future directions include exploring reinforcement learning on top of the pre-trained foundation and extending the framework to additional platforms and interaction modalities.