Summary (Overview)

• Qwen-UI-Agent is a real-world-centric foundation GUI agent developed by Alibaba Token Hub that operates across mobile, computer-use, web, and DeepSearch environments, achieving state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol.
• The system introduces six key transitions: from simulated to real-device execution, from isolated domains to cross-platform workflows, from GUI-only to hybrid GUI+CLI and batched actions, from short-horizon to reliable long-horizon task completion, from human-intensive to AutoResearch-style training pipelines, and from reactive execution to proactive service initiation.
• Qwen-UI-Agent achieves strong benchmark results: 82.1% on MobileWorld, 92.2% on MobileWorld-Real, 97.5% on AndroidDaily, 79.5% on OSWorld-Verified, 73.6% on WebArena, and 81.5% on ScreenSpot-Pro (zoom-in).
• The system combines scalable sandbox environments (supporting up to 10,000 concurrent rollouts) with a real-device mobile runtime comprising over 100 physical devices and 150+ applications, using an AutoResearch-style data flywheel and online RL for long-horizon trajectory optimization.
• A lightweight harness layer enables proactive service initiation (based on mobile notifications) and cross-platform task execution with stateful workflows across mobile and computer environments.

Introduction and Theoretical Foundation

The paper establishes that while GUI agents have made rapid progress through multimodal foundation models, they remain largely optimized for simulated benchmarks, leaving a substantial gap between benchmark performance and real-world utility. The authors identify six key transitions needed to bridge this gap:

1. From simulated environments to real-device execution: Simulated benchmarks cannot capture the complexity of real mobile devices—varied UI layouts, interruptions, permissions, and changing environments.

2. From isolated domains to cross-domain, cross-platform workflows: Real-world tasks span mobile, web, desktop, and information-seeking systems, requiring unified execution.

3. From GUI-only actions to hybrid GUI+CLI and batched actions: GUI provides universal visual access, while CLI enables efficient structured operations (data processing, file manipulation). Combining these with batched actions improves efficiency.

4. From short-horizon tasks to reliable long-horizon task completion: Practical value grows with the complexity of tasks an agent can complete, requiring sustained planning, state tracking, and error recovery.

5. From human-intensive training pipelines to AutoResearch-style ones: As agent capabilities improve, the development process should become increasingly automated.

6. From reactive execution to proactive service initiation: Agents should recognize actionable moments from digital signals (e.g., notifications) and propose appropriate next steps without explicit user instruction.

The task formulation is defined as:

$$\tau = (I, E_\tau)$$

where $I$ is the user instruction and $E_\tau$ is the set of digital environments available. Observations are multichannel:

$$o_t = (o_t^{GUI}, o_t^{CLI}, o_t^{API})$$

Methodology

Environment Infrastructure

The system's environment infrastructure has four core components:

Scaling Sandbox Environments: Supports up to 10,000 isolated sandbox environments running in parallel across four domains:

• Mobile Use: Built on MobileWorld with Android on redroid (container-based, no KVM)
• Computer Use: Ubuntu VM environment from OSWorld, extended with direct bash execution
• Browser Use: Self-contained browser runtime with FastAPI, Playwright, and Chromium
• DeepSearch: Structured information access via Serper (search) and Jina Reader (document extraction)

Real-Device Mobile Runtime: Comprises over 100 physical devices and over 150 applications. Features include:

• Health-aware scheduler that tracks device/application/account/network health
• Virtual-screen mechanism allowing one phone to host multiple application sessions concurrently
• Dedicated User Agent for sensitive operations (CAPTCHA, payments, permissions)
• VLM-based judge to distinguish model failures from environment failures

Hybrid GUI+CLI Action Space: The unified action space supports:

• GUI actions: click, double_click, long_press, type, open, drag, system_button, wait
• CLI actions: cli_command (bash)
• API actions: api_call
• Interaction/Control: ask_user, terminate

Unified Interface: Standardizes environment lifecycle (acquire, reset, step, evaluate, tear_down, release) across heterogeneous environments.

Agent-Driven Data Flywheel

Organized into two stages:

Stage I: Domain Capability Bootstrapping:

• Uses strong foundation models to analyze domain knowledge and capabilities
• Agents generate initial task pool and environment contexts
• Multiple rounds of rejection sampling produce a unified SFT corpus

Stage II: Iterative Refinement Loop:

1. Evaluation and performance assessment
2. Automated failure diagnosis (model vs. environment failures)
3. Weakness-driven task synthesis
4. Trajectory collection
5. Step-level VLM judge + trajectory-level state-based verifier
6. Iterative model training

Key design choices include: knowledge- and capability-aware task synthesis, agent-driven environment and verifier synthesis, step-level judging for SFT, and failure analysis-driven iteration.

Training Framework

Supervised Fine-Tuning (SFT):

• Domain-conditioned expert training with model merging
• In-distribution data to preserve general and agentic capabilities
• Sliding-window training (windows of n=5 steps, 4-step stride) for efficient long-trajectory SFT

Action RL (Step-Level): Targets six recurring error patterns:

• Confusable-element grounding
• Sorting and ranking
• Quantity and multi-target completeness
• Premature completion
• Repetitive action loops
• Long-tail action selection failures

The action-aware reward function:

$$r_t = F_t [w_{\text{type}} C_t + w_{\text{arg}} C_t Q_t - \lambda_{\text{sens}} S_t - \lambda_{\text{rep}} L_t]$$

where $F_t \in \{0,1\}$ indicates format validity, $C_t$ measures action-type correctness, $Q_t$ measures argument quality, and $S_t$, $L_t$ penalize sensitive actions and repetitions.

Online RL (Trajectory-Level):

• Uses GRPO (Group Relative Policy Optimization) adapted for GUI trajectories

• Up to 10,000 concurrent rollouts across sandbox environments

• Model-adaptive task curriculum: tasks with intermediate success rates receive full rollout budget; mastered/reactivated tasks monitored with smaller budget

• Reward computation:

  $$r_i = v_{\text{x}}(s_{\text{final}}^{(i)}) \in \{0,1\}, \quad \bar{r}_{\text{x}} = \frac{1}{K}\sum_{j=1}^K r_j, \quad \hat{A}_i = \frac{r_i - \bar{r}_{\text{x}}}{\text{Std}(r_1,\dots,r_K) + \epsilon}$$

• Automatic synthesis of ~10,000 validated task–verifier pairs via environment state synthesis, task synthesis, and verifier synthesis with execution validation

Harness Layer

Proactive Service: Builds on mobile notifications through:

• Event perception: parses notifications into structured events
• Affair state and memory: persistent representation of ongoing real-world matters
• Affair-level reasoning and task formation
• Proactive preparation with user confirmation for consequential operations
• Personalization through profile memory and feedback memory

Cross-Platform Task Execution: Implements a hierarchical planner-executor system with:

• Global perception and device grounding (tagged observations)
• Dependency-aware planning and executor selection
• Parallel multi-agent execution (independent subtasks run concurrently)
• Hybrid execution with Qwen-UI-Agent

Empirical Validation / Results

Mobile-Use Evaluation

Table 2: MobileWorld (GUI-only subset, 117 tasks)

Table 3: Real-Device Mobile Benchmarks

Computer-Use Evaluation

Table 4: OSWorld-Verified

Table 5: OSWorld-v2

Browser Use and DeepSearch

Table 6: WebArena

Table 7: DeepSearch Benchmarks

GUI Grounding

Table 8: GUI Grounding Benchmarks (key results)

General and Agentic Capabilities

Summary (Table 9): Qwen-UI-Agent (27B) preserves its base model's (Qwen3.5-27B) general reasoning while outperforming specialized GUI models across all benchmarks. Key scores: MMMU-Pro 72.4, RealWorldQA 83.1, MathVision 82.8, MMLU-Pro 86.5, Terminal-Bench 2.0 50.1, Claw-Eval 73.5, BFCL-v4 74.2.

Behavioral Analysis Findings

• Action RL effects (Table 12): Improves performance on error-pattern-specific tests by 3.8–9.5 percentage points (e.g., Confusable-Element Grounding: +6.3%, Repetitive Action Loops: +9.5%)
• CLI usage on computer-use: 40.7% of actions on OSWorld-Verified, 55.1% on OSWorld-v2; batched actions constitute 39.6% and 41.6% respectively
• Online RL effects: 14.7% increase in trajectories with verification actions, 11.2% decrease in false-stop rate, 10.6% increase in trajectories employing both GUI and Bash

Theoretical and Practical Implications

Theoretical Implications:

• The paper demonstrates that GUI agents benefit fundamentally from a systems-level approach where models, environments, data flywheels, training, and harness are co-designed around real-world usage rather than optimized for isolated benchmarks.
• The AutoResearch-style data flywheel provides a framework for automated capability development that could generalize beyond GUI agents to other interactive AI systems.
• The behavioral analysis reveals that simulated-to-real gaps are not merely performance differences but manifest as specific, diagnosable failure patterns (exploration failure, erroneous action loops, pop-up interference, physical widget control) that require targeted training interventions.

Practical Implications:

• Real-device mobile execution is feasible at scale with the health-aware scheduler and virtual-screen mechanisms, enabling practical deployment.
• Hybrid GUI+CLI action spaces offer substantial efficiency gains: CLI actions constitute ~40-55% of computer-use actions, and batched execution reduces the number of model inference steps per task.
• Proactive service initiation based on notifications represents a paradigm shift from reactive to anticipatory assistance, with the harness layer providing a practical implementation.
• Cross-platform task execution with state preservation enables complex workflows that span mobile and desktop environments, addressing a real user need.
• The model-adaptive curriculum for online RL provides a practical method for efficiently allocating computational resources across tasks of varying difficulty.

Conclusion

Qwen-UI-Agent represents a significant step toward next-generation real-world-centric GUI agents. The system integrates scalable sandbox environments with a real-device mobile runtime, unified GUI+CLI action spaces, an agent-driven data flywheel, a training framework combining SFT, action RL, and online RL, and a harness layer for proactive services and cross-platform execution.

Key achievements include:

• State-of-the-art mobile-use performance (82.1% MobileWorld, 92.2% MobileWorld-Real, 97.5% AndroidDaily)
• Competitive computer-use performance (79.5% OSWorld-Verified, 40.0% OSWorld-v2)
• Leading browser-use and DeepSearch results (73.6% WebArena, 64.1%/75.0% BrowseComp)
• Strong GUI grounding (81.5% ScreenSpot-Pro zoom-in)
• Preservation of general reasoning and agentic capabilities

Limitations acknowledged by the authors include: reliance on AutoJudge for real-device evaluation (92.8% accuracy), incomplete CUA/DeepSearch training at 35B-A3B scale, and the system being agent-driven rather than fully autonomous (requiring human oversight).

Future directions include: improving execution efficiency (latency reduction), harness-assisted long-horizon workflows, large-scale cross-domain online RL, scalable synthesis of high-fidelity environments, and systematic safety evaluations with personalization.