# MMSkills: Towards Multimodal Skills for General Visual Agents

> MMSkills introduces multimodal skill packages that combine textual procedures with visual state evidence, enabling visual agents to shift from trial-and-error to grounded, state-aware execution and consistently improve performance across GUI and game benchmarks.

- **Source:** [arXiv](https://arxiv.org/abs/2605.13527)
- **Published:** 2026-05-19
- **Permalink:** https://picx.dev/p/R7xs3T
- **Whiteboard:** https://picx.dev/p/R7xs3T/image

## Summary

# MMSkills: Towards Multimodal Skills for General Visual Agents

## Summary (Overview)
*   **Introduces Multimodal Skill Packages**: A novel representation for reusable skills in visual agents, combining textual procedures, runtime state cards, and multi-view visual evidence into state-conditioned procedural knowledge.
*   **Proposes an Automated Skill Generator**: An agentic pipeline that transforms public, non-evaluation interaction trajectories into multimodal skill packages through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing.
*   **Develops Branch-Loaded Utilization**: A runtime mechanism where a temporary branch inspects selected skill evidence, aligns it with the live environment, and returns structured guidance to the main agent, avoiding context pollution and visual over-anchoring.
*   **Demonstrates Consistent Performance Gains**: Experiments across GUI (OSWorld, macOSWorld) and game (VAB-Minecraft, Super Mario Bros) benchmarks show MMSkills improve both frontier and smaller multimodal agents over no-skill and text-only skill baselines.
*   **Shifts Agent Behavior**: MMSkills reduce low-level action load, suppress repetitive trajectories, and improve completion awareness, moving agents from exploratory trial-and-error towards grounded, state-aware execution.

## Introduction and Theoretical Foundation
Reusable skills are a core abstraction for building capable agents, often stored as textual prompts, executable code, or learned routines. However, for **visual agents**, procedural knowledge is inherently **multimodal**. Success depends not only on *what* operation to perform, but also on *recognizing the relevant visual state*, *interpreting visual evidence of progress or failure*, and *deciding what to do next*. Text-only skills become verbose yet underspecified, while raw demonstrations are lengthy and instance-specific.

This gap motivates the need for **multimodal procedural knowledge**: reusable guidance that binds action procedures to the visual evidence and state-dependent decisions required for applying them. The paper formalizes three central challenges:
1.  **Representation**: What should a multimodal skill package contain?
2.  **Generation**: Where can such packages be derived from public, non-evaluation interaction data?
3.  **Utilization**: How can an agent consult multimodal evidence at inference time without excessive image context or over-anchoring to reference screenshots?

MMSkills is proposed as a framework to address these challenges, with the core thesis that **external multimodal procedural knowledge complements model-internal priors**, especially for weaker visual agents.

## Methodology

### 1. Multimodal Skill Package Representation
Each MMSkill is a state-conditioned procedure package represented as:
$$
M = (D, P, S, K)
$$
where:
*   $D$: Compact descriptor.
*   $P$: Reusable textual procedure.
*   $S = \{S_j\}_{j=1}^m$: Set of **runtime state cards**.
*   $K = \{K_j\}_{j=1}^m$: Set of **keyframe bundles** aligned with the state cards.

A **runtime state card** $S_j$ is an agent-facing node that links a point in the procedure to decision rules:
$$
S_j = (\text{when\_to\_use}_j, \text{when\_not\_to\_use}_j, \text{visible\_cues}_j, \text{verification\_cue}_j, V_j)
$$
where $V_j = \text{available\_views}_j$.

Each key state is grounded by a **multi-view keyframe bundle**. Let $V = \{\text{full\_frame}, \text{focus\_crop}, \text{before}, \text{after}\}$. Then:
$$
K_j = \{ K^v_j : v \in V_j, v \in V \}
$$
The `full_frame` preserves global context, `focus_crop` localizes the visual cue, and optional `before`/`after` views expose useful transitions.

### 2. Skill Generator from Public Trajectories
An automated, meta-skill-guided pipeline `G_F` transforms a public trajectory pool $T_d$ for domain $d$ into a skill library $M_d$:
$$
G_F: T_d \mapsto M_d
$$
The pipeline comprises five phases:
$$
T_d \xrightarrow[\text{Phase 0}]{\text{embed+cluster}} C_d \xrightarrow[\text{Phase 1}]{\text{cluster plan}} A_d \xrightarrow[\text{Phase 2}]{\text{merge}} R_d \xrightarrow[\text{Phase 3}]{\text{text draft}} \tilde{M}_d \xrightarrow[\text{Phase 4}]{\text{image ground+audit}} M_d
$$
For a merged skill $r \in R_d$, finalization is:
$$
\tilde{M}_r = (D_r, P_r, \tilde{S}_r, \tilde{K}_r) \xrightarrow{\text{ground+audit}} M_r = (D_r, P_r, S_r, K_r)
$$
The generator uses a conservative visual grounding policy, adding views only for state recognition, transition comparison, or completion verification.

### 3. Branch-Loaded Multimodal Skills Agent
To avoid the pitfalls of direct skill loading (context pressure, visual over-anchoring), MMSkills uses a **branch-loading** mechanism. The main agent can either act directly or consult a temporary skill branch:
$$
\text{direct}: A_t = \pi_{\text{main}}(O_t, H_t, C_I), \quad \text{branch}: G_t = \text{Branch}(O_t, H_t, M_t), \quad A_t = \pi_{\text{main}}(O_t, H_t, C_I, G_t)
$$
The branch output is a **structured guidance tuple**:
$$
G_t = (\text{applicable}_t, \text{subgoal}_t, \text{plan}_t, \text{do\_not\_do}_t, \text{verify}_t)
$$
The branch operates in two stages:
1.  **Gated View Selection**: Selects relevant state cards and view types based on the live observation:
    $$(J_t, R_t) = \text{SelectViews}(O_t, H_{t-1}, P_t, S_t), \quad V_t = \{ K^v_j : j \in J_t, v \in R_{t,j} \}$$
2.  **Branch Planning**: Aligns selected evidence with the live state and returns $G_t$:
    $$G_t = \text{PlanBranch}(O_t, H_{t-1}, P_t, \{S_j : j \in J_t\}, V_t)$$
The main agent uses $G_t$ as decision support but chooses the final grounded action from the live screenshot.

## Empirical Validation / Results
Experiments were conducted across GUI (OSWorld, macOSWorld) and game (VAB-Minecraft, Super Mario Bros) benchmarks, evaluating frontier and smaller multimodal models under no-skill, text-only skill, and MMSkills conditions.

### RQ1: Overall Performance on GUI and Game Tasks
**OSWorld Results (Table 1)**: MMSkills improved overall success rates across all model families.
*   **Gemini 3.1 Pro**: 44.08% → 50.11%
*   **Gemini 3 Flash**: 36.65% → 47.97%
*   **Qwen3-VL-235B**: 21.34% → 39.17%
*   **Qwen3-VL-8B-Instruct**: 10.78% → 25.40%

**Auxiliary Benchmark Results (Table 2)**: Gains transferred beyond OSWorld.
*   **macOSWorld**: MMSkills improved completed large-model runs (e.g., Gemini 3 Flash: 55.94% → 65.73%).
*   **VAB-Minecraft**: Consistent gains in success rate and average score across all models.
*   **Super Mario Bros**: Higher total performance and reward under MMSkills.

**Key Finding**: External multimodal procedural knowledge is especially valuable for weaker visual agents, compensating for limited model-internal priors.

### RQ2: Ablations of Skill Content and Branch Loading
**Skill Package Components (Figure 3A)**: The complete MMSkills package (with state cards *and* images) performed best. Removing state cards weakened state discrimination; removing images preserved decision rules but removed visual grounding evidence.

**Branch Loading Mechanism (Figure 3B)**:
*   **Direct-full loading** hurt performance due to context pollution.
*   **Branch loading** provided clear gains, even for text-only skills.
*   The full **two-stage design** (branch loading + view selection) performed best.

### RQ3: Skill Usage and Interaction Dynamics (Table 3)
*   **Invocation**: MMSkills were invoked more often than text-only skills (e.g., Qwen3-VL-235B on OSWorld: 37.50% → 65.28%), suggesting multimodal evidence makes external knowledge easier to recognize as relevant.
*   **Trajectory Length**: MMSkills **reduced** average steps per task, while text-only skills sometimes added overhead. This indicates multimodal skills help agents find shorter, more efficient paths.
*   **Selected Views**: `focus_crop` views dominated selected visual evidence, with `full_frame`, `before`, and `after` views providing supplementary context when needed.

### RQ4: Behavioral Shift Analysis (Figures 4 & 6)
*   **Lower Action Load**: MMSkills reduced the mean number of low-level primitives (clicks, keyboard actions) per task.
*   **Reduced Repetition**: Exact repeated actions and the longest same-mode run decreased significantly (e.g., Qwen3-VL-235B exact repeats: 21.8% → 6.2%).
*   **Improved Completion Awareness**: `DONE` actions increased, indicating state cards and verification cues help agents decide when a task is complete.

**Key Finding**: MMSkills reshape agent behavior from exploratory trial-and-error towards grounded, state-aware execution.

## Theoretical and Practical Implications
*   **Theoretical**: Formalizes **multimodal procedural knowledge** as a crucial component for visual agent competence, bridging the gap between abstract text procedures and instance-specific visual demonstrations.
*   **Representation**: Introduces a structured, **state-conditioned skill package** that explicitly binds procedures to visual decision rules (`when_to_use`, `verification_cue`) and evidence (`state cards`, `keyframes`).
*   **Generation**: Demonstrates that usable multimodal skills can be **automatically mined from public interaction data**, not requiring hand-crafted examples or evaluation-task trajectories.
*   **Utilization**: **Branch loading** addresses practical LLM limitations (long-context reliability, visual over-anchoring) by isolating evidence inspection from action generation.
*   **Practical**: Provides a **scalable framework** to enhance visual agents across diverse domains (GUI, games). The gains for smaller models suggest MMSkills can democratize access to capable visual automation.

## Conclusion
MMSkills presents a comprehensive framework for representing, generating, and utilizing reusable multimodal skills for visual agents. By coupling textual procedures with runtime state cards and multi-view visual evidence, and employing a branch-loaded consultation mechanism, MMSkills consistently improve agent performance across benchmarks and model families. The work demonstrates that **external multimodal procedural knowledge effectively complements model-internal priors**, leading to more efficient, reliable, and state-aware visual agents.

**Limitations and Future Work**: The approach depends on source-trajectory coverage, may inherit errors from the generation pipeline, and incurs extra inference cost from branch loading. Future directions include extending MMSkills to broader embodied or safety-critical settings, which will require stronger verification and online skill repair mechanisms.

---

_Markdown view of https://picx.dev/p/R7xs3T, served by PicX — AI-generated visual whiteboard summaries of research papers._