# KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

> KnowAct-GUIClaw achieves 64.1% success rate on MobileWorld via self-evolving memory and skill libraries that generalize across models and platforms.

- **Source:** [arXiv](https://arxiv.org/abs/2607.12625)
- **Published:** 2026-07-17
- **Permalink:** https://picx.dev/p/CFXbVN
- **Whiteboard:** https://picx.dev/p/CFXbVN/image

## Summary

# KnowAct-GUIClaw: Know Deeply, Act Perfectly – Personal GUI Assistant with Self-Evolving Memory and Skill

## Summary (Overview)

- **Proposes KnowAct-GUIClaw**, a cross-platform GUI agent framework built on the “Know Deeply, Act Perfectly” paradigm, addressing two core limitations of OpenClaw-style agents: insufficient cross-platform GUI interaction and lack of self-evolution mechanisms.
- **Introduces a four-stage Know–Route–Act–Reflect closed-loop pipeline** that decouples high-level task orchestration (host agent) from low-level visual device manipulation (GUI subagent), enabling efficient delegation and reusable knowledge.
- **Implements two persistent reusable storage modules**: attribution-aware experience memory (textual policies from past trajectories) and state-validated self-evolving skill libraries (parameterized, checkable action sequences including deeplink/intent shortcuts).
- **Achieves state-of-the-art performance** on the MobileWorld GUI-Only benchmark: **64.1% success rate (SR)** with open-source Kimi-K2.6 models, surpassing all agential frameworks and closed-source models (e.g., Seed-2.0-Pro at 63.2%, GPT-5.5 at 62.4%).
- **Demonstrates cross-model transferability**: equipping a Qwen3.5-35B-A3B executor with memory and skills distilled from Kimi-K2.6 trajectories improves SR by 16.2 points (24.8 → 41.0), and cross-platform usability on HarmonyOS (76.2%) and Windows (70.0%).

## Introduction and Theoretical Foundation

The paper motivates the need for GUI-capable personal assistants by noting that real-world user tasks often require interaction with graphical user interfaces (GUIs) rather than clean APIs. Existing OpenClaw-style agents excel at tool-based automation but suffer from four drawbacks:

1. **Cross-app information loss**: High-level instructions spanning multiple apps lose intermediate data values.
2. **Incomplete GUI observations**: Screen captures, accessibility trees, and action histories each reveal only fragments of the device state.
3. **No persistent learning**: Successful and failed trajectories are discarded, forcing agents to re-learn from scratch.
4. **Non-visual shortcuts ignored**: Faster methods like Android deep links, intents, and reusable action sequences are not integrated or validated.

**Theoretical foundation**: The paper models GUI automation as a **partially observable Markov decision process (POMDP)**:

$$
\mathcal{M} = \langle S, A, O, \mathcal{T}, \Omega, R, \gamma \rangle
$$

where $S$ is the hidden device state, $A$ the action space, $O$ the observation space, $\mathcal{T}$ and $\Omega$ the transition and observation functions, $R$ the task reward, and $\gamma$ the discount factor. Partial observability is the defining property—the agent never observes $S$ directly but only an observation $o_t$ (screenshot, foreground app, action history) that projects a subset of hidden state (navigation stacks, login status, loading states, off-screen values).

The framework adopts a **host-centric multi-agent system** design, where a central orchestrator (host) maintains conversation state, memory, and tool access, and delegates subtasks to specialized agents (GUI subagent).

## Methodology

KnowAct-GUIClaw organizes execution as a **four-stage loop**: Know, Route, Act, Reflect. Two persistent stores—memory/history store and skill/shortcut store—provide advisory context at every stage.

### 1. Know: Context Gathering and Host Control
- **Active retrieval**: Prior GUI memories and candidate skills are retrieved by semantic similarity; policy memory is injected directly.
- **Host-held context**: The host agent keeps session context, agent memory, and user profile, proposing defaults for fuzzy instructions.
- **Host-centric control**: The host resolves eligible subtasks itself (using conversation, memory, tools) and delegates to GUI subagent only when visual interaction, cross-app manipulation, or on-device verification is required.

### 2. Route: Task Decomposition and Information Contracts
- **Task decomposition**: The router emits either a single GUI task or an ordered multi-app workflow. Each subtask is a goal-level tuple $(g_i, h_i, I_i, O_i)$: app-scoped goal $g_i$, optional app name $h_i$, required inputs $I_i$, and declared outputs $O_i$.
- **Information transfer via blackboard**: Let $\mathcal{G}$ denote GUIClaw executing a subtask over the POMDP, $E$ the evidence-to-value mapping, and $B_i$ the blackboard after subtask $i$. The transfer rule is:

$$
\mathcal{G}(g_i, h_i, B_{i-1}[I_i]) \rightarrow \tau_i, \quad E(\tau_i, O_i) \rightarrow B_i[O_i]
$$

Missing inputs or outputs cause the workflow to fail closed, preventing fabrication.

### 3. Act: Hybrid GUI–Fast Path Execution
- **Hybrid action space**: $A = A_{\text{gui}} \cup A_{\text{skill}} \cup A_{\text{shortcut}} \cup A_{\text{ask}}$
  - $A_{\text{gui}}$: human-like primitives (tap, swipe, scroll, text input, navigation, app open/close, wait).
  - $A_{\text{skill}}$: reusable skills distilled from trajectories (parameterized, state-validated).
  - $A_{\text{shortcut}}$: Android deeplinks/intents that bypass navigation when verified.
  - $A_{\text{ask}}$: intervention actions for user input/authorization.
- **Observe–Reason–Act loop**: $p_t \rightarrow o_t \rightarrow a_t \rightarrow e_t \rightarrow o_{t+1}$, where $p_t$ is the prompt carrying subtask, blackboard, policy memory, hints, and available skills.

### 4. Reflect: Trajectory Distillation and Skill Evolution
- **Post-run summarization**: Each GUI task trajectory is condensed into a checkpoint note for resumption.
- **Skill extraction**: For selected trajectories, a vision LLM rewrites GUI events into structured evidence (task, platform, apps, action sequence, parameters, screenshots, state contracts). Accepted steps inherit state contracts from the trajectory.
- **Skill evolution**: When a reused skill fails, reflection records the failure and updates the skill in-place (narrowing description, adding obstacle handling, refreshing targets) rather than extracting a new skill.
- **Shortcut validation**: Candidates from app manifests are validated by launching and checking foreground app, ADB output, UI tree, and screenshot; only page-validated records are promoted.
- **Experience memory**: Trajectories are distilled into textual policies (success/failure lessons) using separate prompts, stored with per-app deduplication, and fed to both routing and GUI execution.

## Empirical Validation / Results

**Benchmarks**: MobileWorld (117 GUI-Only tasks, deterministic evaluators) and AndroidDaily (194 resolved tasks, 235 all tasks, LLM judge + human expert). **Default config**: Qwen3.5-397B-A17B host + Qwen3.5-35B-A3B GUI executor; Kimi-K2.6 configurations use Kimi as both host and executor.

### Main Results on MobileWorld

**Table 1: MobileWorld GUI-Only success rate (pass@1).** Selected rows:

| Model / Configuration | SR (%) |
|-----------------------|--------|
| **General models** | |
| Seed-2.0-Pro | 63.2 |
| GPT-5.5 | 62.4 |
| Gemini-3.1-Pro-Preview | 58.1 |
| Claude-Opus-4.7 | 56.4 |
| **Kimi-K2.6 (ours)** | |
| + host & memory | 61.5 |
| + host, memory & skills | **64.1** |
| **Qwen3.5-35B-A3B (ours)** | |
| executor only | 24.8 |
| + host & memory | 34.5 |
| + host, memory & skills | 37.9 |
| + host, Kimi-derived memory & skills | 41.0 |
| **Qwen3.5-397B-A17B (ours)** | |
| host acts directly (F) | 46.2 |

Key findings:
- **Kimi-K2.6 with host, memory & skills achieves 64.1%**, the highest overall, beating all closed-source models.
- **Memory and skills provide consistent improvements**: +9.7 points (35B executor only → +host&mem), +3.4 points (35B +host&mem → +skills), and +5.5 points (397B host only → F).
- **Cross-model transfer**: Kimi-derived memory & skills on 35B executor yields 41.0% (vs. 37.9% with same-model artifacts).

### Ablation and Efficiency Analysis

**Table 2: MobileWorld ablation and efficiency (our runs).** Key rows:

| Config | Setting | SR (%) | GUI steps | GUI tasks | Total tokens | Host tokens |
|--------|---------|--------|-----------|-----------|--------------|-------------|
| **35B executor** | | | | | | |
| A | exec. only | 24.8 | 26.7 | 1.0 | 281,266 | – |
| B | + host & mem | 34.5 | 26.8 | 2.3 | 279,211 | 65,224 |
| C | + skills | 37.9 | 25.1 | 2.5 | 278,289 | 63,792 |
| **397B executor** | | | | | | |
| D | exec. only | 40.7 | 26.1 | 1.0 | 254,459 | – |
| E | + host & mem | 43.3 | 26.8 | 1.4 | 273,352 | 10,096 |
| F | + skills | 46.2 | 23.7 | 1.7 | 260,516 | 10,982 |

- **Host and memory raise accuracy at modest cost**: A→B (+9.7 SR, flat tokens); D→E (+2.6 SR, +7% tokens).
- **Skills cut steps and tokens**: B→C (-1.7 steps, -0.9% tokens); E→F (-3.1 steps, -4.7% tokens).
- **Host-centric control with 397B host (F) is most efficient**: lowest GUI-trace tokens (260,516) and host tokens (10,982) among augmented configurations.

**Table 3: Skill-reuse effect on skill-using tasks (83 tasks with 35B, 87 with 397B).**

| Metric | 35B +skills | 35B no skills | ∆ | 397B +skills | 397B no skills | ∆ |
|--------|-------------|---------------|----|--------------|----------------|----|
| GUI steps/task | 25.7 | 29.0 | -3.3 | 22.3 | 25.6 | -3.3 |
| total tokens/task | 284,279 | 303,014 | -6.2% | 242,930 | 258,084 | -5.9% |
| single-run SR (%) | 40.6 | 35.7 | +4.9 | 50.2 | 48.3 | +1.9 |

### AndroidDaily Results

**Table 4: AndroidDaily (resolved) – 78.61% total** (vs. UI-TARS-1.5 at 56.64%, Step-GUI-8B at 52.50%). Largest margins on Analyze and complex tasks.

### Cross-Platform Checks

- **HarmonyOS**: 48/63 tasks (76.2%) solved. Failures due to inertial wheel pickers (e.g., 8:25 alarm) and premature termination (e.g., TaoBao payment page misinterpretation).
- **Windows**: 21/30 tasks (70.0%) solved. Failures due to semantic grounding (e.g., “laughing emoji” misselection) and weak temporal grounding (short-lived notifications).

## Theoretical and Practical Implications

- **Theoretical contribution**: Formalizes GUI automation as a POMDP and introduces a decomposed host-executor architecture that separates reasoning from perception, enabling efficient task allocation and reusable knowledge. The “Know Deeply, Act Perfectly” paradigm emphasizes that accumulated experience directly improves execution accuracy—a principle transferable to other multimodal agent systems.
- **Practical implications**: 
  - **Cross-platform adaptability**: The framework works on Android, iOS, HarmonyOS, and Windows using the same interface, demonstrating that GUI subagents can be platform-agnostic with proper memory and skill stores.
  - **Self-evolution without retraining**: Memory and skills are updated via reflection, enabling continuous improvement from user interactions without model fine-tuning.
  - **Efficiency gains**: Skills reduce GUI steps by ~3 per task on average, cutting token costs (especially from visual observations) by 5-6%. Host-centric control further reduces overhead by resolving non-visual subtasks directly.
  - **Model transferability**: Memory and skills distilled from one model (Kimi-K2.6) improve another (Qwen3.5-35B) by 16.2 points, suggesting that knowledge can be portable across model families.

## Conclusion

The paper proposes **KnowAct-GUIClaw**, a cross-platform GUI agent framework that addresses OpenClaw-style agents’ limitations via a four-stage Know–Route–Act–Reflect pipeline. Key components include:
- Attribution-aware persistent experience memory and state-validated self-evolving skill libraries.
- Typed blackboard information transfer protocol for cross-application data flow.
- Host-centric control that selectively delegates GUI tasks.

The framework achieves **state-of-the-art 64.1% on MobileWorld** (with Kimi-K2.6) and demonstrates cross-model transfer, cross-platform usability, and measurable efficiency gains.

**Future work**: Tighter native integration of the Knowledge module, external tools, and the GUI subagent to eliminate rigid sequential pipeline handoffs. Design a unified joint planner that concurrently weighs knowledge retrieval, non-visual tool calls, and GUI actions within one decision cycle to reduce host-subagent communication overhead.

---

_Markdown view of https://picx.dev/p/CFXbVN, served by PicX — AI-generated visual whiteboard summaries of research papers._