Summary (Overview)

• Resource2Skill is a framework that automatically distills executable skills from multimodal human-created resources (tutorial videos, source repositories, articles, and reference artifacts) into a hierarchical Skill Wiki for software agents.
• The Skill Wiki organizes skills as multimodal entries containing structured text, executable/adaptable code, visual examples, metadata, and provenance, enabling agents to retrieve and compose relevant skills at inference time.
• The same resource-to-skill operator is reused both offline (to build a large-scale library) and online (to fill capability gaps when the offline pool is insufficient), making the library maintainable and incrementally extensible.
• Across seven practical authoring domains (Web, Excel, Reaper, PPT, Blender, CAD, UE5) and four agent backends, Resource2Skill improves the average overall score by +11.9 percentage points over no-skill agents and outperforms strong agentic-harness baselines (ClaudeCode-H, Codex-H) in 26 of 28 main-aggregate model–domain cells.
• Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity (especially tutorial videos), library scaling, the hierarchy-then-LM selection strategy, and online acquisition.

Introduction and Theoretical Foundation

The paper addresses the growing need for software agents to perform complex, tool-augmented tasks beyond simple question answering—tasks such as creating slide decks, spreadsheets, web pages, 3D scenes, CAD designs, and audio projects. Success in these domains depends on reusable procedural knowledge (called skills): how to decompose a goal, which tool patterns to use, what intermediate states to inspect, and how to recover from failures.

Existing skill libraries are limited in how they are created:

• Hand-written by experts (expensive and not scalable)
• Accumulated from an agent’s own interaction traces (narrow, domain-specific)
• Mined from text/code resources (miss temporal and perceptual information)

A major underused source is multimodal human-created resources, especially tutorial videos. Videos capture temporal order of operations, visual effects, and tacit design choices that are difficult to express in text alone. However, directly using raw videos at inference time is expensive and impractical. The paper formulates a new problem: resource-to-skill learning—distilling reusable executable skills from multimodal human resources, with tutorial videos as a key source of procedural and perceptual knowledge.

Theoretical basis: The framework treats skill construction and skill use as a unified pipeline. Offline, large-scale resources are distilled into a domain wiki. At inference, an agent navigates the hierarchical index, retrieves relevant multimodal entries, and composes them during execution. When coverage is insufficient, the same construction operator is invoked online to acquire new skills.

Methodology

Hierarchical Multimodal Skill Wiki

A skill is defined as a tuple:

$$s = (p, x_{\text{text}}, x_{\text{visual}}, x_{\text{code}}, m)$$

• $p$: path of $s$ in a domain-specific taxonomy $T_D$
• $m$: metadata (filtering, auditing, provenance)
• $x_{\text{text}}$: name, mechanism, applicability, inputs, expected effects
• $x_{\text{visual}}$: thumbnails, screenshots, rendered previews, diagrams
• $x_{\text{code}}$: executable or adaptable procedure fragments

The full library for domain $D$ is:

$$\Sigma_D = \{ s : s \text{ accepted by the construction operator} \}$$

Resource-to-Skill Construction

A construction operator $f_\theta$ distills multimodal resources $R_D$ (from four families: tutorial videos, source repositories, articles, reference artifacts) into candidate skills $\tilde{s}_{1:k} = f_\theta(r, D)$. A domain-specific acceptance predicate $A_D$ enforces five checks: completeness, traceable provenance, deduplication, modality consistency, and structural executability of the code field.

$$\Sigma_D = \{ s = \text{normalize}(\tilde{s}) : \tilde{s} = f_\theta(r, D), r \in R_D, A_D(\tilde{s}) = 1 \}$$

Selecting and Composing Skills

Given a brief $q$, the agent uses MetaBrowse, a two-stage selection process:

1. Lexical scorer (BM25) narrows candidates to a topically relevant region of the taxonomy: $$C_K(q) = \text{TopK}_{s \in \Sigma_D} \text{BM25}(q, \text{name}(s) \oplus \text{tags}(s) \oplus \text{applicability}(s) \oplus p(s))$$
2. Language model selects a subset to compose: $$S(q) = \pi_\phi(q, \{ \Phi(s) : s \in C_K(q) \})$$ where $\Phi(s)$ contains metadata and exposed text/visual/code views.

Execution and Online Acquisition

The agent and domain adapter share an MCP (Model Context Protocol) tool surface. Selected skill code executes directly against the live MCP server. When $C_K(q)$ contains no adequate candidate, the same operator $(f_\theta, A_D)$ is invoked online to search for new resources, distill temporary candidates, and expose them as a separate online pool.

Empirical Validation / Results

Setup

• Domains: Web (HTML/CSS/JS), Excel, Reaper (audio), PPT, Blender, CAD, UE5
• Benchmarks: 80 task briefs per domain, no overlap with resource corpora
• Agent backends: GPT-5.5, GPT-5.4, GPT-5.4 Mini, GPT-5.4 Nano
• Judge: GPT-5.4 vision judge (non-audio), GPT-4o audio-capable judge (Reaper)
• Metrics: Five-axis rubric (0–10), overall score = unweighted mean of five axes; failures treated as 0%.

Main Comparison

Table 1: Main comparison, overall score (%).

• w Skills beats w/o Skills in all 28 cells, average lift +11.9 pp (56.8% vs. 45.0%).
• w Skills beats the stronger harness baseline in 26 of 28 cells (exceptions within 1 pp).
• Largest gains in UE5 (+30–40 pp), Blender, Web; smallest in Reaper.

Studies

Skill-Library Scaling

Performance rises monotonically with library size and saturates near 200 skills. The 0→200 slice carries the largest gains (+3.1 to +14.2 pp per domain); beyond 200 the curve flattens.

Offline and Online Acquisition

Table 2: Offline and online skill acquisition, overall score (%).

Online acquisition acts as a gap-filler: negligible on standard tasks, but +21.6 pp on novel tasks not covered by the offline pool.

Ablations

Skill + Wiki vs Skill Only

• Flat (pure text) outperforms no-skill in every domain.
• Our Wiki beats Flat by 2.5–8.2 pp, with largest gains in Excel, Web, Blender.

Wiki Construction: Source Ablation

Table 3: Ablation: resource-source mix, overall score (%).

• Video is non-substitutable: holding it out drops avg. from 68.9% to 59.4% (video-only outscores three-source no-video by 7.4 pp).
• All-source pool stays ahead of strongest two-source variant by 0.3–0.9 pp per domain.

Matched-Budget Representation Ablation

Table 4: Matched-budget representation ablation, overall score (%).

• Text alone (strong curated memory) achieves 65.0%.
• Visuals add +1.9 pp, code adds +2.0 pp.
• Full multimodal entry ranks first in every domain (68.9%).

Selection Strategy Ablation

Table 5: Ablation: selection strategy, overall score (%).

• MetaBrowse (hierarchy-then-LM) outperforms all retrieval-only baselines (BM25, Embed, BM25+Embed) and random/no-skill by large margins.

Theoretical and Practical Implications

• Theoretical: The paper establishes resource-to-skill learning as a new paradigm for grounding procedural knowledge in multimodal human-created resources. It shows that tutorial videos, repositories, articles, and artifacts each contribute complementary signals: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles/artifacts provide conceptual grounding. The hierarchical wiki structure encodes domain structure and narrows search space, making retrieval more efficient than flat retrieval.
• Practical: Resource2Skill provides a scalable, maintainable way to build skill libraries for software agents without manual authoring. The unified offline-online pipeline allows libraries to grow incrementally. The system is immediately applicable to seven commercial authoring tools, and the MCP-based interface is domain-agnostic. The empirical results (up to +40