OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use

Summary (Overview)

• Definition of GCUA: The paper introduces the concept of a Generalist Computer-Use Agent (GCUA) — an agent system where the model decides what to observe, how to act, and how to organize its work across visual (GUI) and programmatic (CLI, API, accessibility) interfaces, rather than being restricted to GUI-only interaction.

• Key design contributions: Two central mechanisms are proposed: (1) code execution — allowing models to write programs that combine GUI actions with shell commands, loops, and conditionals; and (2) harness-to-model decision transfer — moving selected workflow decisions (e.g., when to request screenshots, how to track task state) from fixed harness rules to the model itself.

• Benchmark results: On the OSWorld-v2 offline set (82 tasks), the full OpenGCUA configuration improves task scores across all eight evaluated frontier models (including GPT-6 Astra and Claude Opus 5), raising the mean partial score by +19.6 points and reducing reported steps per task by an average of 53.0% relative to batched GUI-only execution.

• Real-device findings: On four real macOS tasks, a large gap exists between models: Qwen3.8-Max uses 19.4× more reasoning tokens than GPT-6 Astra (50.2K vs. 2.6K tokens per task) while completing fewer tasks, highlighting substantial differences in real-device execution efficiency.

• Open-source release: The project releases implementation code, prompts, evaluation trajectories from all eight models, a preliminary real-device harness, and a cross-device SDK, providing the community with a full-stack open foundation for GCUA research.

---

Introduction and Theoretical Foundation

Background and Motivation

Computer use research has reached a turning point: the focus is expanding from GUI-only interaction to systems that complete tasks on users' computers through both visual and programmatic interfaces. Frontier implementations such as GPT-6 Astra's computer-use system remain largely closed, limiting systematic study of their design.

The paper defines a Generalist Computer-Use Agent (GCUA) as:

"An agent system designed to complete diverse computer tasks through both visual interaction and programmatic access, in which the model decides how to observe, act, and organize its work as the task requires."

Formal Definition

A GCUA makes three decisions at each step, formalized as:

$$(q_{t}, p_{t}, m_{t+1}) \sim \pi_{\theta}(\cdot | m_{t}, o_{t}). \tag{1}$$

Where:

• $q_t \subseteq \mathcal{Q} = \{\text{screenshot}, \text{AX tree}\}$ — what additional views to observe ($q_t = \emptyset$ means no additional view)
• $p_t$ — a program whose statements call primitives in $\mathcal{A} = \mathcal{A}_{\text{GUI}} \cup \mathcal{A}_{\text{AX}} \cup \mathcal{A}_{\text{CLI}} \cup \mathcal{A}_{\text{API}}$, potentially using loops, branches, and variables
• $m_{t+1}$ — the updated working state (plan, progress record, observations kept in context)

The next observation is:

$$o_{t+1} = \big(e_{t}, \{v_{q}(s_{t+1})\}_{q \in q_{t}}\big), \tag{2}$$

where $v_q$ renders view $q$ of the computer state and $e_t$ is the execution output.

Relation to Existing Agents

The paper shows that existing agent families are special cases of GCUA where some decisions are fixed in advance:

The Route Flip to Code Execution

The paper identifies a 2026 shift from clicking toward programming, visible in vendor documentation. With code execution, each model call yields a program, and control flow is decided while the program runs. Three things become possible in a single call:

1. Conditions — if the dialog is visible, dismiss it
2. Loops — for each row, fill the three fields
3. Verification — assert the status now reads Saved

This contrasts with batched actions, where all actions must be fixed when the call is generated — a batch is essentially "a program without control flow."

---

Methodology

Code Execution over GUI and CLI

Implementation:

• System prompt specifies PyAutoGUI for desktop interaction and bash() for shell commands
• Model outputs one Python program per step in a <code> block
• Harness extracts, syntax-checks, and executes the program
• Execution feedback (stdout, stderr, exit status) is returned to the model

Five recurring code-execution patterns identified from GPT-6 Astra trajectories:

1. Cross-Interface Composition — using information from one interface to guide operations through another (e.g., clicking GUI buttons while checking output files programmatically)
2. Loop-Based Execution — specifying a procedure once and repeating it across multiple targets (e.g., iterating over seven webpage sections with DOM checks)
3. Result-Dependent Execution — conditional logic responding to execution results (e.g., comparing draft text before clicking Send)
4. Observation Preparation — gathering and organizing evidence for the next model decision (e.g., assembling 25 views into a numbered contact sheet)
5. Executable Verification — checking both intended changes and preserved content (e.g., verifying background images while confirming other content unchanged)

Harness-to-Model Decision Transfer

Four mechanisms implement this transfer:

1. On-Demand Observation — the screenshot() function lets the model decide whether the next input needs a new screenshot
2. Task-State Tracking — a todo interface maintains persistent records of progress with statuses (pending, changed_unverified, claimed_verified)
3. Action Intent — an <intent> block states each program's purpose before execution, aiding user understanding
4. Harness-Managed Context — rolling screenshot buffer and micro-compaction policies manage context length while preserving prompt caching

Experimental Setup

• Benchmark: OSWorld-v2 offline subset (82 tasks, excluding 26 requiring internet)
• Models: Claude Opus 5, GPT-6 Astra, Gemini 3.8 Flash, Qwen3.8-Max, Qwen3.8-Flash, Kimi K3, DeepSeek-V4.1-Flash, GLM-5.3-Flash
• Baseline: GUI-only with batched actions
• Metrics: Partial score, binary success, steps/task, output tokens/task

---

Empirical Validation / Results

CUA versus GCUA (Table 2)

Code-Execution Ablation

Relative to batched GUI+CLI (same interfaces, no code execution):

• GLM-5.3-Flash: +7.4 points (55.8 → 63.2)
• Qwen3.8-Max: −17.0% steps (66.1 → 54.9)

Harness-to-Model Decision Transfer Ablation (Table 4)

Real-Device Results (Table 5)

Key finding: Qwen3.8-Max uses 19.4× GPT-6 Astra's reasoning tokens (50.2K vs. 2.6K per task) but completes fewer tasks.

---

Theoretical and Practical Implications

Theoretical Implications

1. Execution design matters beyond interface access: The ablation showing code execution adds +6.1 points over GUI+CLI access with the same interfaces demonstrates that how operations are coordinated (programs with control flow vs. fixed batches) is as important as which interfaces are available.

2. Decision allocation is a design dimension: The GCUA framework formalizes that the question is not just what components an agent has (the five-layer view) but who decides — the model or the harness — at each step.

3. Efficiency is a first-class metric: The paper argues that success rate alone is insufficient; steps per task, output tokens, reasoning tokens, and cost must be tracked alongside task completion.

Practical Implications

1. Resource-aware model selection: Money and time favor different models — Qwen3.8-Flash reaches 60.5 points at ~$0.38/task, while GPT-6 Astra reaches 75.7 at ~$15.26/task. Under tight budgets, lower-scoring models may be preferable.

2. Real-device challenges require dedicated infrastructure: Benchmarks fail to capture shared control, background windows, evolving requests, and existing user environments. The five-layer harness architecture addresses these systematically.

3. Reasoning efficiency is a critical optimization target: The 19.4× gap in reasoning tokens between models on real devices suggests substantial room for improvement through "concise reasoning" training and system-level optimizations.

---

Conclusion

Main takeaways:

1. GCUA generalizes existing GUI, CLI, and hybrid agents by making three decisions model-driven: what to observe, how to act, and how to organize work.

2. Code execution and harness-to-model decision transfer improve task scores across all eight tested frontier models (+19.6 mean points) while cutting steps per task by 53.0% on OSWorld-v2.

3. On real devices, the largest gap lies in the model itself — not the harness — with substantial differences in reasoning efficiency and code-execution capability.

Future directions:

• Model release: Training a dedicated GCUA model that unifies computer and browser use, with code execution and decision transfer as learned capabilities
• Efficiency research: More concise reasoning for routine operations, mitigating overthinking, and system-level improvements
• Real-device benchmarks: Development of evaluation environments on real macOS devices covering wider application ranges, user interactions, and changing device states
• Cross-device expansion: Extending the harness and SDK toward tasks spanning computers, browsers, and mobile devices

The project invites community contributions to the harness, SDK, real-device evaluation, and research agenda, positioning OpenGCUA as a shared foundation for generalist computer-use research.