# StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

> StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% on OSWorld 2.0 while cutting cost ninefold by grounding action in program state rather than screenshots.

- **Source:** [arXiv](https://arxiv.org/abs/2607.22798)
- **Published:** 2026-07-29
- **Permalink:** https://picx.dev/p/TdLQoX
- **Whiteboard:** https://picx.dev/p/TdLQoX/image

## Summary

## Summary (Overview)

- **StateAct** is a code-first, multi-agent harness for computer-use agents that grounds action, verification, and memory in **program state** (files, application backends, DOM) rather than screenshots, while retaining a dedicated GUI subagent for the few visual-only subgoals (1.1% of main-agent steps, 28 of 108 tasks).
- On OSWorld 2.0 (108 long-horizon tasks), StateAct lifts Claude Opus 4.8 from **20.6% → 26.9%** binary success and **54.8% → 61.6%** mean partial success, while reducing cost ~9× (from ~$72 to ~$7.8 per task).
- An independent **finish gate** verifies completion by inspecting persisted state without reading the agent’s narration, catching structural defects (e.g., missing file, wrong path) but not value errors, which remain the dominant failure mode.
- Ablation shows the gain comes from **state-grounding** (observing state rather than pixels), not from added agentic depth; a code-only variant without the GUI subagent scores only 45.9% partial, below the screenshot-based baseline (54.8%).
- The bottleneck shifts from **perception to reasoning**: failures now depend more on what the agent thinks than on what it sees.

## Introduction and Theoretical Foundation

### Background and Motivation
Computer-use agents operate real desktop software (file managers, spreadsheets, calendars, browsers). Most prior work improves agents by strengthening screen perception—better models for reading rendered pixels and choosing clicks. However, a screenshot is a **lossy rendering** of the underlying program state. Different states can produce the same pixels, and code can inspect and modify that state directly. This gap worsens over long horizons: a per-step lossy proxy accumulates state drift across hundreds of dependent operations.

### The State-Grounding Principle
The paper formalizes two channels for observing computer state:

- **Pixel channel**: $o_{\text{pix}} = f_{\text{render}}(s)$, where $s \in \mathcal{S}$ is the program state.
- **State channel**: $o_{\text{state}} = g(s)$ (e.g., shell command, workbook read, DOM serialization).

Two structural facts drive the approach:

1. **Rendering is lossy and non-injective.** Many distinct states render to indistinguishable screenshots. Formally, $f_{\text{render}}$ is not injective, so $f_{\text{render}}^{-1}$ does not exist; no perceptual model can recover $s$ from $o_{\text{pix}}$ in general. Over the substate a task touches, $g$ is effectively invertible (e.g., reading a cell formula directly from the workbook).

2. **The deliverable is the state.** A desktop task’s success is determined by program state $G(s)$. For tasks depending on non-rendered content (formulas, hidden rows, backend state), $G(s)$ cannot be recovered from $f_{\text{render}}(s)$ alone.

The principle distinguishes three task types: **state-addressable** (code path exists), **hybrid** (mostly state, one visual subgoal), and **render-only** (pixel/appearance judgments). State-grounding helps the first two and gives no advantage on the third.

## Methodology

### StateAct Architecture
StateAct (Figure 2 in the paper) has three components:

**1. Act on State** – Main agent action space is code and structured operations: persistent bash, file editor, read-only image viewer, plan checklist, finish action, and agent delegation tool. No live screen actuation (mouse/keyboard) is exposed to the main agent. The main agent discovers state through model priors and active probing (find, ls, grep, sqlite3).

**2. Verify on State** – An independent **finish gate** spawns when the main agent calls `finish` (permitted only after ≥3 non-finish steps). The gate’s context contains only the verbatim task instruction and machine access (bash, file reads, DOM queries), with editor mutations blocked. It never sees the main agent’s message history, plan, or rationale. Properties:
- **Narration-blind** – cannot be biased by the trajectory
- **State-grounded** – checks the requested result in the exact persisted artifact
- **Anti-capture** – rejects evidence found only in side files the agent created
- **Bounded correction** – on rejection, the main agent is sent back to fix the gap (up to 3 rounds)

**3. Sustain State** – Three mechanisms for long episodes (up to 200 main-agent turns + ~98 subagent turns):
- **Fresh-context specialists**: each delegation runs in its own context window
- **Auto-compaction**: near context limit, oldest prefix is summarized, images stripped
- **Externalized plan**: task checklist persisted outside message history, re-injected each turn

**Delegation rule**: The main agent uses the GUI subagent when a subgoal is irreducibly visual (no file/backend/DOM path found, or effect only expressible as rendered interaction). Measured GUI-use rate: 1.1% of main-agent steps (28 of 108 tasks). Browser work is handled by a dedicated web subagent that uses DOM serialization (structured state).

### Experimental Setup
- **Benchmark**: OSWorld 2.0 (108 long-horizon tasks)
- **Backbone**: Claude Opus 4.8 with adaptive thinking, 200-turn main-agent budget
- **Metrics**: Binary success, mean partial score (fractional credit), cost per task (USD)
- **Turn accounting**: Main agent averages ~57 model turns; each subagent delegation adds ~23 interior turns (capped at 50), for ~155 total model turns per task.

## Empirical Validation / Results

### Main Result
On the identical backbone (Claude Opus 4.8), replacing the computer-use-agent harness with StateAct raises binary success by **+6.3 points** and mean partial by **+6.8 points**, while cutting output tokens (224K → 100K) and cost (~$72 → ~$7.8 per task, ~9× reduction).

StateAct (26.9%/61.6%) is the best-performing entry, exceeding the same-backbone reference (20.6%/54.8%), Opus-4.7 (18.2%/48.9%), and GPT-5.5 (13.0%/49.5%).

**Table 1: Per-capability performance on OSWorld 2.0** (binary / mean-partial %)

| Capability | StateAct (Opus 4.8) | Opus 4.8 (ref. CUA) | GPT-5.5 | Opus-4.7 | Sonnet-4.6 | GLM-5V | MiniMax M3 | Qwen3.7 |
|------------|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| Multi-item state | 27.9/66.7 | – | 14.0/50.6 | – | 11.6/46.7 | 0.0/11.2 | 7.0/23.2 | 2.3/20.2 |
| Streaming | 66.7/66.7 | – | 50.0/57.8 | – | 0.0/4.7 | 0.0/1.2 | 0.0/6.4 | 0.0/0.0 |
| Cross-source | 26.1/64.9 | – | 13.0/52.4 | – | 10.9/45.8 | 0.0/11.7 | 6.5/24.9 | 6.5/26.3 |
| Conflict disambig. | 30.8/64.5 | – | 20.5/51.4 | – | 12.8/42.4 | 0.0/12.4 | 7.7/24.3 | 7.7/29.1 |
| Visual-spatial | 31.1/61.3 | – | 11.1/51.2 | – | 8.9/36.5 | 0.0/4.6 | 4.4/19.8 | 2.2/19.8 |
| Implicit state | 30.2/60.1 | – | 14.0/47.3 | – | 9.3/37.0 | 0.0/11.0 | 4.7/24.4 | 2.3/24.1 |
| Tutorial | 13.6/55.3 | – | 9.1/37.5 | – | 13.6/43.5 | 0.0/7.8 | 4.5/15.0 | 4.5/15.7 |
| Multimodal edit | 20.0/54.2 | – | 6.7/47.0 | – | 6.7/37.5 | 0.0/6.2 | 6.7/22.3 | 0.0/20.6 |
| Dynamic env. | 30.0/53.5 | – | 30.0/46.2 | – | 10.0/22.0 | 0.0/5.8 | 0.0/17.9 | 0.0/16.3 |
| Human-in-the-loop | 0.0/43.9 | – | 16.7/43.1 | – | 16.7/51.9 | 0.0/12.5 | 0.0/16.8 | 0.0/22.5 |
| **Overall (/108)** | **26.9/61.6** | 20.6/54.8 | 13.0/49.5 | 18.2/48.9 | 8.3/41.5 | 0.0/9.0 | 4.6/22.3 | 2.8/21.5 |

### Ablation
**Table 3a: Component ablation** (binary / mean-partial %)

| Config | Binary | Partial |
|--------|:------:|:-------:|
| StateAct (full) | 26.9 | 61.6 |
| – verify (no finish gate) | 23.1 | 57.5 |
| – sustain (plan off) | 21.3 | 58.7 |
| – act (GUI; cua blocked) | 18.5 | 51.3 |

Removing act-on-state produces the largest drop (partial 61.6% → 51.3%, below the reference’s 54.8%), showing code-first action on state is the largest single contributor.

### Additional Designs
- **Bash-only** (no GUI, subagents, or finish gate): 45.9% partial, below the reference (54.8%)
- **Sonnet 4.6 backbone**: StateAct lifts binary from 8.3% → 11.1%
- **OSWorld-Verified** (short-horizon): StateAct and reference perform similarly (78.4% vs. 77.3% binary), confirming long-horizon advantage

### Finish-Gate Analysis
**Table 5: Finish-gate operating characteristics**

| Final verdict vs. grader | Binary success | Non-perfect | Total |
|:------------------------:|:--------------:|:-----------:|:-----:|
| PASS | 28 | 68 | 96 |
| FAIL | 1 | 8 | 9 |
| No finish called | 0 | 3 | 3 |
| Total | 29 | 79 | 108 |

**Operating rates**: Correct rejections / all rejections = 88.9%; Correct rejects / non-perfect reached = 10.5%; Correct passes / binary successes = 96.6%; Tasks retried = 26.7%; Retries ending in binary success = 21.4%.

The gate correctly rejects only 8 of 76 non-perfect tasks that reached it (error rate ≈ 90%). This is because the gate checks only **structure**; most failures are value or reasoning errors no structural check can catch.

### Is a Strong GUI Subagent Necessary?
On four of five benchmarks, substituting a compact 31B in-house GUI model (SFR-CUA) for Claude Opus 4.8 as the GUI subagent barely moves the score (OSWorld-Verified: 81.1 vs. 81.9; WindowsAgentArena: 51.2 vs. 50.6; AndroidWorld: 84.1 vs. 81.9; MobileWorld: 68.4 vs. 70.1). The exception is OSWorld 2.0 (longest horizon), where the weaker subagent causes a drop (43.2% vs. 61.6% partial). A frontier GUI model is unnecessary on shorter-horizon benchmarks but still helps on the hardest long-horizon desktop tasks.

## Theoretical and Practical Implications

**Theoretical implications**: The paper formalizes the distinction between pixel and state channels, showing that rendering is non-injective and that task success depends on state, not pixels. This shifts the theoretical framing from improving perception to **grounding action, verification, and memory in state**. The bottleneck moves from *what the agent sees* (perception) to *what the agent thinks* (reasoning).

**Practical implications**: 
- State-grounding simultaneously improves accuracy (~6-7 points) and reduces cost (~9×), showing that code-first approaches are not just better but cheaper.
- A dedicated GUI subagent is needed for only 1.1% of steps, and a compact model suffices for most benchmarks.
- The finish gate provides a principled, narration-blind verification mechanism, but its ceiling is structural correctness; value errors require ground-truth labels or per-task verifiers.
- For practitioners, this suggests building agents that operate on APIs, files, and databases whenever possible, reserving GUI interaction for the irreducible visual subgoals.

## Conclusion

StateAct makes program state the primary interface for a computer-use agent's main agent while retaining a dedicated GUI subagent for visual interaction. On OSWorld 2.0, Claude Opus 4.8 rises from 20.6% to 26.9% binary success (54.8% to 61.6% partial) at ~9× lower cost, without any model change. Ablation and diagnostic analyses localize the gain to **what the agent observes** (state rather than screenshots), not added depth, and the limit to **value correctness**, which a self-verifier without ground-truth labels cannot close. State-grounding moves the accuracy wall from perception to reasoning; it does not remove it. For long-horizon computer use, the bottleneck is now what the agent thinks, not what it sees.

---

_Markdown view of https://picx.dev/p/TdLQoX, served by PicX — AI-generated visual whiteboard summaries of research papers._