# Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

> Qwen-UI-Agent achieves state-of-the-art mobile-use performance (92.2% MobileWorld-Real) via real-device execution, hybrid GUI+CLI actions, and an AutoResearch-style data flywheel with online RL.

- **Source:** [arXiv](https://arxiv.org/abs/2607.28227)
- **Published:** 2026-08-01
- **Permalink:** https://picx.dev/p/5xnDqo
- **Whiteboard:** https://picx.dev/p/5xnDqo/image

## Summary

## Summary (Overview)

- **Qwen-UI-Agent** is a real-world-centric foundation GUI agent developed by Alibaba Token Hub that operates across mobile, computer-use, web, and DeepSearch environments, achieving state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol.
- The system introduces six key transitions: from simulated to real-device execution, from isolated domains to cross-platform workflows, from GUI-only to hybrid GUI+CLI and batched actions, from short-horizon to reliable long-horizon task completion, from human-intensive to AutoResearch-style training pipelines, and from reactive execution to proactive service initiation.
- Qwen-UI-Agent achieves strong benchmark results: **82.1%** on MobileWorld, **92.2%** on MobileWorld-Real, **97.5%** on AndroidDaily, **79.5%** on OSWorld-Verified, **73.6%** on WebArena, and **81.5%** on ScreenSpot-Pro (zoom-in).
- The system combines scalable sandbox environments (supporting up to 10,000 concurrent rollouts) with a real-device mobile runtime comprising over 100 physical devices and 150+ applications, using an AutoResearch-style data flywheel and online RL for long-horizon trajectory optimization.
- A lightweight harness layer enables proactive service initiation (based on mobile notifications) and cross-platform task execution with stateful workflows across mobile and computer environments.

## Introduction and Theoretical Foundation

The paper establishes that while GUI agents have made rapid progress through multimodal foundation models, they remain largely optimized for simulated benchmarks, leaving a substantial gap between benchmark performance and real-world utility. The authors identify six key transitions needed to bridge this gap:

1. **From simulated environments to real-device execution**: Simulated benchmarks cannot capture the complexity of real mobile devices—varied UI layouts, interruptions, permissions, and changing environments.

2. **From isolated domains to cross-domain, cross-platform workflows**: Real-world tasks span mobile, web, desktop, and information-seeking systems, requiring unified execution.

3. **From GUI-only actions to hybrid GUI+CLI and batched actions**: GUI provides universal visual access, while CLI enables efficient structured operations (data processing, file manipulation). Combining these with batched actions improves efficiency.

4. **From short-horizon tasks to reliable long-horizon task completion**: Practical value grows with the complexity of tasks an agent can complete, requiring sustained planning, state tracking, and error recovery.

5. **From human-intensive training pipelines to AutoResearch-style ones**: As agent capabilities improve, the development process should become increasingly automated.

6. **From reactive execution to proactive service initiation**: Agents should recognize actionable moments from digital signals (e.g., notifications) and propose appropriate next steps without explicit user instruction.

The task formulation is defined as:
$$\tau = (I, E_\tau)$$
where $I$ is the user instruction and $E_\tau$ is the set of digital environments available. Observations are multichannel:
$$o_t = (o_t^{GUI}, o_t^{CLI}, o_t^{API})$$

## Methodology

### Environment Infrastructure
The system's environment infrastructure has four core components:

**Scaling Sandbox Environments**: Supports up to **10,000** isolated sandbox environments running in parallel across four domains:
- **Mobile Use**: Built on MobileWorld with Android on redroid (container-based, no KVM)
- **Computer Use**: Ubuntu VM environment from OSWorld, extended with direct bash execution
- **Browser Use**: Self-contained browser runtime with FastAPI, Playwright, and Chromium
- **DeepSearch**: Structured information access via Serper (search) and Jina Reader (document extraction)

**Real-Device Mobile Runtime**: Comprises **over 100 physical devices** and **over 150 applications**. Features include:
- Health-aware scheduler that tracks device/application/account/network health
- Virtual-screen mechanism allowing one phone to host multiple application sessions concurrently
- Dedicated User Agent for sensitive operations (CAPTCHA, payments, permissions)
- VLM-based judge to distinguish model failures from environment failures

**Hybrid GUI+CLI Action Space**: The unified action space supports:
- GUI actions: click, double_click, long_press, type, open, drag, system_button, wait
- CLI actions: cli_command (bash)
- API actions: api_call
- Interaction/Control: ask_user, terminate

**Unified Interface**: Standardizes environment lifecycle (acquire, reset, step, evaluate, tear_down, release) across heterogeneous environments.

### Agent-Driven Data Flywheel
Organized into two stages:

**Stage I: Domain Capability Bootstrapping**:
- Uses strong foundation models to analyze domain knowledge and capabilities
- Agents generate initial task pool and environment contexts
- Multiple rounds of rejection sampling produce a unified SFT corpus

**Stage II: Iterative Refinement Loop**:
1. Evaluation and performance assessment
2. Automated failure diagnosis (model vs. environment failures)
3. Weakness-driven task synthesis
4. Trajectory collection
5. Step-level VLM judge + trajectory-level state-based verifier
6. Iterative model training

Key design choices include: knowledge- and capability-aware task synthesis, agent-driven environment and verifier synthesis, step-level judging for SFT, and failure analysis-driven iteration.

### Training Framework

**Supervised Fine-Tuning (SFT)**:
- Domain-conditioned expert training with model merging
- In-distribution data to preserve general and agentic capabilities
- Sliding-window training (windows of n=5 steps, 4-step stride) for efficient long-trajectory SFT

**Action RL (Step-Level)**:
Targets six recurring error patterns:
- Confusable-element grounding
- Sorting and ranking
- Quantity and multi-target completeness
- Premature completion
- Repetitive action loops
- Long-tail action selection failures

The action-aware reward function:
$$r_t = F_t [w_{\text{type}} C_t + w_{\text{arg}} C_t Q_t - \lambda_{\text{sens}} S_t - \lambda_{\text{rep}} L_t]$$

where $F_t \in \{0,1\}$ indicates format validity, $C_t$ measures action-type correctness, $Q_t$ measures argument quality, and $S_t$, $L_t$ penalize sensitive actions and repetitions.

**Online RL (Trajectory-Level)**:
- Uses GRPO (Group Relative Policy Optimization) adapted for GUI trajectories
- Up to **10,000 concurrent rollouts** across sandbox environments
- Model-adaptive task curriculum: tasks with intermediate success rates receive full rollout budget; mastered/reactivated tasks monitored with smaller budget
- Reward computation:
  $$r_i = v_{\text{x}}(s_{\text{final}}^{(i)}) \in \{0,1\}, \quad \bar{r}_{\text{x}} = \frac{1}{K}\sum_{j=1}^K r_j, \quad \hat{A}_i = \frac{r_i - \bar{r}_{\text{x}}}{\text{Std}(r_1,\dots,r_K) + \epsilon}$$

- Automatic synthesis of ~10,000 validated task–verifier pairs via environment state synthesis, task synthesis, and verifier synthesis with execution validation

### Harness Layer

**Proactive Service**: Builds on mobile notifications through:
- Event perception: parses notifications into structured events
- Affair state and memory: persistent representation of ongoing real-world matters
- Affair-level reasoning and task formation
- Proactive preparation with user confirmation for consequential operations
- Personalization through profile memory and feedback memory

**Cross-Platform Task Execution**: Implements a hierarchical planner-executor system with:
- Global perception and device grounding (tagged observations)
- Dependency-aware planning and executor selection
- Parallel multi-agent execution (independent subtasks run concurrently)
- Hybrid execution with Qwen-UI-Agent

## Empirical Validation / Results

### Mobile-Use Evaluation

**Table 2: MobileWorld (GUI-only subset, 117 tasks)**

| Model | Access/Size | Success Rate (%) |
|---|---|---|
| Seed 2.1 Pro | Closed-source | 73.2 |
| GPT-5.6 Sol | Closed-source | 70.1 |
| Claude Opus 4.8 | Closed-source | 67.5 |
| Qwen 3.7 Plus | 397B-A17B | 62.3 |
| **Qwen-UI-Agent (27B)** | **27B** | **82.1** |
| **Qwen-UI-Agent (35B-A3B)** | **35B-A3B** | **65.0** |

**Table 3: Real-Device Mobile Benchmarks**

| Model | MobileWorld-Real (%) | AndroidDaily (%) |
|---|---|---|
| Seed 2.1 Pro | 88.7 | 95.2 |
| Gemini 3.1 Pro | 86.2 | 93.8 |
| GPT-5.6 Sol | 85.4 | 92.6 |
| Claude Opus 4.8 | 84.7 | 93.0 |
| **Qwen-UI-Agent (27B)** | **92.2** | **97.5** |
| **Qwen-UI-Agent (35B-A3B)** | **87.4** | **93.9** |

### Computer-Use Evaluation

**Table 4: OSWorld-Verified**

| Model | Access/Size | Success Rate (%) |
|---|---|---|
| Claude Opus 4.8 | Closed-source | 83.4 |
| **Qwen-UI-Agent (27B)** | **27B** | **79.5** |
| Seed 2.1 Pro | Closed-source | 78.8 |
| GPT-5.5 | Closed-source | 78.7 |

**Table 5: OSWorld-v2**

| Model | Partial (%) | Binary (%) | Steps/Task |
|---|---|---|---|
| Claude Opus 4.8 | 54.8 | 20.6 | 103.0 |
| GPT-5.5 | 49.5 | 13.0 | 95.2 |
| **Qwen-UI-Agent (27B)** | **40.0** | **13.9** | **135.8** |
| MiniMax M3 | 22.3 | 4.6 | 326.7 |

### Browser Use and DeepSearch

**Table 6: WebArena**

| Model | Success Rate (%) |
|---|---|
| **Qwen-UI-Agent (27B)** | **73.6** |
| Claude Opus 4.8 | 71.9 |
| GPT-5.5 | 69.5 |
| Gemini 3.1 Pro | 65.3 |
| Human Performance | 78.2 |

**Table 7: DeepSearch Benchmarks**

| Model | BrowseComp (%) | BrowseComp-ZH (%) |
|---|---|---|
| GPT-5.5 | 90.1 | – |
| **Qwen-UI-Agent (27B)** | **64.1** | **75.0** |
| Qwen3.5-397B-A17B | 78.6 | 70.3 |
| UI-TARS-2 | 29.6 | 50.5 |

### GUI Grounding

**Table 8: GUI Grounding Benchmarks (key results)**

| Model | SS-Pro (zoom-in) | SS-V2 | MM-GUI-L2 | OSW-G-R | UI-Vision |
|---|---|---|---|---|---|
| **Qwen-UI-Agent (27B)** | **81.5** | **97.5** | **92.6** | **78.5** | **70.0** |
| Qwen 3.7 Plus | 79.0 | 96.6 | 90.5 | 78.2 | 68.0 |
| Seed 2.1 Pro | 80.7 | 96.6 | 90.9 | 78.0 | 62.0 |

### General and Agentic Capabilities

**Summary (Table 9)**: Qwen-UI-Agent (27B) preserves its base model's (Qwen3.5-27B) general reasoning while outperforming specialized GUI models across all benchmarks. Key scores: MMMU-Pro 72.4, RealWorldQA 83.1, MathVision 82.8, MMLU-Pro 86.5, Terminal-Bench 2.0 50.1, Claw-Eval 73.5, BFCL-v4 74.2.

### Behavioral Analysis Findings

- **Action RL effects** (Table 12): Improves performance on error-pattern-specific tests by 3.8–9.5 percentage points (e.g., Confusable-Element Grounding: +6.3%, Repetitive Action Loops: +9.5%)
- **CLI usage on computer-use**: 40.7% of actions on OSWorld-Verified, 55.1% on OSWorld-v2; batched actions constitute 39.6% and 41.6% respectively
- **Online RL effects**: 14.7% increase in trajectories with verification actions, 11.2% decrease in false-stop rate, 10.6% increase in trajectories employing both GUI and Bash

## Theoretical and Practical Implications

**Theoretical Implications**:
- The paper demonstrates that GUI agents benefit fundamentally from a systems-level approach where models, environments, data flywheels, training, and harness are co-designed around real-world usage rather than optimized for isolated benchmarks.
- The AutoResearch-style data flywheel provides a framework for automated capability development that could generalize beyond GUI agents to other interactive AI systems.
- The behavioral analysis reveals that simulated-to-real gaps are not merely performance differences but manifest as specific, diagnosable failure patterns (exploration failure, erroneous action loops, pop-up interference, physical widget control) that require targeted training interventions.

**Practical Implications**:
- Real-device mobile execution is feasible at scale with the health-aware scheduler and virtual-screen mechanisms, enabling practical deployment.
- Hybrid GUI+CLI action spaces offer substantial efficiency gains: CLI actions constitute ~40-55% of computer-use actions, and batched execution reduces the number of model inference steps per task.
- Proactive service initiation based on notifications represents a paradigm shift from reactive to anticipatory assistance, with the harness layer providing a practical implementation.
- Cross-platform task execution with state preservation enables complex workflows that span mobile and desktop environments, addressing a real user need.
- The model-adaptive curriculum for online RL provides a practical method for efficiently allocating computational resources across tasks of varying difficulty.

## Conclusion

Qwen-UI-Agent represents a significant step toward next-generation real-world-centric GUI agents. The system integrates scalable sandbox environments with a real-device mobile runtime, unified GUI+CLI action spaces, an agent-driven data flywheel, a training framework combining SFT, action RL, and online RL, and a harness layer for proactive services and cross-platform execution.

Key achievements include:
- State-of-the-art mobile-use performance (82.1% MobileWorld, 92.2% MobileWorld-Real, 97.5% AndroidDaily)
- Competitive computer-use performance (79.5% OSWorld-Verified, 40.0% OSWorld-v2)
- Leading browser-use and DeepSearch results (73.6% WebArena, 64.1%/75.0% BrowseComp)
- Strong GUI grounding (81.5% ScreenSpot-Pro zoom-in)
- Preservation of general reasoning and agentic capabilities

**Limitations acknowledged by the authors** include: reliance on AutoJudge for real-device evaluation (92.8% accuracy), incomplete CUA/DeepSearch training at 35B-A3B scale, and the system being agent-driven rather than fully autonomous (requiring human oversight).

**Future directions** include: improving execution efficiency (latency reduction), harness-assisted long-horizon workflows, large-scale cross-domain online RL, scalable synthesis of high-fidelity environments, and systematic safety evaluations with personalization.

---

_Markdown view of https://picx.dev/p/5xnDqo, served by PicX — AI-generated visual whiteboard summaries of research papers._