# Demystifying Agent Skills: Why They Work-Until They Don't

> Skills improve LLM agents primarily as procedural anchors that stabilize execution routines, not by injecting knowledge, with retrieval accuracy largely decoupled from downstream task success.

- **Source:** [arXiv](https://arxiv.org/abs/2608.14036)
- **Published:** 2026-08-20
- **Permalink:** https://picx.dev/p/whkUoq
- **Whiteboard:** https://picx.dev/p/whkUoq/image

## Summary

## Summary (Overview)

- This paper systematically investigates **when** skills help LLM agents, **why** they work, and **where** they fail, moving beyond aggregate success-rate evaluations to mechanistic understanding.
- Through controlled experiments across Terminal-Bench and SkillsBench with multiple agent–model pairings (Codex + GPT-5.3-Codex, Gemini CLI + Gemini-3.1-Pro-Preview), the authors isolate four research questions: representation effects, outcome annotation, cross-framework transfer, and retrieval difficulty.
- A contrastive trajectory-analysis methodology is introduced: 8,135 trial records are normalized, 240 trajectories are open-coded, and 238 valid labels are consolidated into a taxonomy of **3 high-level categories and 12 skill-use modes** (human validation: 95.8% exact agreement, Cohen's κ = 0.952).
- Key finding: Skills primarily work as **procedural anchors** (65.7% of mechanisms) rather than explicit knowledge injection (4.5%), improving over Workflow Memory by **+6.06 percentage points** (95% CI: [+0.76, +11.36]).
- Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6% to 3.3%, yet downstream success remains stable (~36–39%), showing that exact ground-truth invocation is neither sufficient nor necessary for success.

## Introduction and Theoretical Foundation

### Background

Large language model (LLM) agents are increasingly expected to improve through experience rather than solve each task from scratch. Recent agent systems store and reuse traces of prior execution: environment setup sequences, tool-use patterns, debugging routines, and verification steps. Among proposed memory forms, **skills** (Anthropic, 2025) have emerged as a compelling abstraction—not merely a record of past execution, but a compact description of *what to do, what to check, and what pitfalls to avoid*.

Skills promise three advantages over raw traces or direct workflow memories:
1. **Compression** of noisy experience into shorter context
2. **Standardization** of procedural knowledge into a stable format
3. **Transferability** of knowledge across related tasks

### Motivation

Existing evaluations measure only whether skill-augmented agents solve more tasks. This reveals little about:
- What fundamentally changes in agent behavior before/after skill loading
- Which parts of execution are stabilized by skill guidance
- Why the same skill can help one task while harming another

The field lacks a principled understanding of what makes a skill genuinely valuable, leaving skill design to heuristic iteration, prompt tuning, or benchmark-specific trial and error.

### Theoretical Foundation

The paper frames skill utility as a **pipeline problem** along four stages:
1. **Representation** — how prior experience is packaged (workflow memory vs. distilled skill)
2. **Outcome signals** — whether explicit success/failure annotations matter
3. **Transfer** — whether distilled guidance survives framework changes
4. **Retrieval and invocation** — how pool size and confusability affect identification and use

## Methodology

### Experimental Setup

**Models and benchmarks:**
- Two agent–model pairings for RQ1–RQ2: Codex + GPT-5.3-Codex and Gemini CLI + Gemini-3.1-Pro-Preview
- RQ3: cross-framework transfer from Codex to Gemini CLI
- RQ4: Qwen3-Embedding-0.6B for retrieval; Gemini CLI + Gemini-3.1-Pro-Preview and Codex + GPT-5.4 (GPT-5.3 no longer available)
- Benchmarks: Terminal-Bench, SkillsBench, Terminal-Bench-Pro
- Downstream execution: Harbor evaluation workflow, n = 5 unique trials per task, parallelism 20

### Experimental Design

**RQ1–RQ2: Controlled study of procedural experience representation**

Three conditions compared:
- **Raw**: no prior experience
- **Workflow Memory**: cleaned procedural traces from prior executions
- **Skill**: standardized SKILL.md distilled from the same workflows

A fixed-budget composition grid varies source evidence from success-only to failure-only (5s0f through 0s5f). For each composition, Workflow Memory and Skill are built from the same trajectories, holding underlying experience constant while varying representation only.

**No-hint variants**: success/failure identities are hidden from the skill creator while preserving the same trajectories.

**RQ3: Cross-framework transfer**
Skills and workflow memories constructed in Codex are evaluated in Gemini CLI, isolating framework transfer effects.

**RQ4: Skill retrieval and downstream execution**

Three independent experiments on matched candidate pools (each containing ground-truth skill + distractors; pool size 5–100; distractors random, similar, or dissimilar):
- **Arm 1**: Embedding-based ranking without task execution
- **Arm 2**: Explicit agent selection without execution
- **Arm 3**: Full-pool real execution with skill-use parsing after verification

Outputs from Arms 1–2 are **not** passed to Arm 3.

### Taxonomy Construction

- 8,135 trial records normalized into a shared manifest; 7,837 contain agent transcripts
- Open-coding over 240 sampled trajectories; 238 valid unique labels retained
- 12-mode canonical taxonomy organized into 3 high-level categories (SC1: successful procedural anchoring; SC2: execution-layer/verification failures; SC3: invocation/applicability/boundary failures)
- Human validation: 714 trajectory–label checks (all confirmed); 95.8% exact agreement with Cohen's κ = 0.952 on taxonomy aggregation
- 528 paired triples constructed (SkillsBench: 144, Terminal-Bench 2.0: 186, Terminal-Bench-Pro: 198)

## Empirical Validation / Results

### RQ1: Skills as Procedural Anchors

**Aggregate results:**
- Skill: 61.9% oracle-status success rate
- Raw: 59.1%
- Workflow Memory: 55.9%
- Skill improves over Workflow Memory by **+6.06 percentage points** (95% bootstrap CI: [+0.76, +11.36])

**Mechanism analysis:**
- `procedural_anchor` accounts for **65.7%** of skill mechanisms
- `knowledge_injection` accounts for only **4.5%**
- Skill-guided success: 61.6% vs. workflow-guided success: 54.5%

**Table 1: Task success rates (excerpt)**

| Trajectory Mix | Terminal-Bench-2 Workflow | Terminal-Bench-2 Skill | SkillsBench Workflow | SkillsBench Skill |
|---|---|---|---|---|
| 5s0f | 0.4452 | 0.7548 | 0.5250 | 0.7250 |
| 4s1f | 0.4000 | 0.7290 | 0.5667 | 0.6167 |
| 3s2f | 0.4194 | 0.7806 | 0.6417 | 0.6250 |
| 2s3f | 0.3677 | 0.6839 | 0.6083 | 0.7083 |
| 1s4f | 0.2710 | 0.7097 | 0.5167 | 0.6167 |
| 0s5f | 0.2839 | 0.5161 | 0.5833 | 0.4500 |

### RQ2: Outcome Annotations

- Withholding outcome labels has little effect when source pools contain only successful trajectories
- Impact grows sharply once failed trajectories enter the pool
- Example (Gemini on Terminal-Bench-2 at 3s2f): 0.7462 with hints vs. 0.4000 without

### RQ3: Cross-Framework Transfer

- Skills constructed in Codex remain useful when evaluated in Gemini CLI
- Distilled skills provide more portable representation than direct workflow memory across framework boundaries

### RQ4: Retrieval and Downstream Use

**Key finding**: Retrieval accuracy does **not** determine downstream success.

- Actual-use precision falls from 29.6% (pool size 5) to 3.3% (pool size 100)
- Downstream success changes only from 36.4% to 39.3%
- Arm 3 recall remains 54.3–73.6% at k = 100 despite precision of only 0.7–8.1%
- Similar distractors are the dominant stressor for offline identification (Arm 1 top-1 precision on similar pools: 70.5% at k = 5 → 53.4% at k = 100)

**Table 4: Effect of pool composition and size on SkillsBench (excerpt)**

| Pool | Metric | k=5 | k=10 | k=20 | k=50 | k=100 |
|---|---|---|---|---|---|---|
| Random | Arm 1 P | 97.7 | 95.5 | 95.5 | 92.0 | 84.1 |
| Random | Arm 3 P | 25.9 | 23.2 | 19.5 | 8.6 | 4.4 |
| Similar | Arm 1 P | 70.5 | 63.6 | 60.2 | 56.8 | 53.4 |
| Similar | Arm 3 P | 34.5 | 22.3 | 15.7 | 7.3 | 3.7 |
| Dissimilar | Arm 1 P | 96.6 | 96.6 | 96.6 | 94.3 | 93.2 |
| Dissimilar | Arm 3 P | 28.6 | 19.2 | 9.0 | 4.4 | 1.7 |

### Execution-Layer Improvements

- SC2 execution-layer failures: 37.3% (raw) → 33.3% (workflow) → 23.5% (skill)
- `environment_infrastructure_failure`: 5.3% (raw) → 1.7% (workflow) → 0.2% (skill)
- `output_format_schema_mismatch`: 7.4% (raw) → 3.2% (skill)
- `background_service_lifecycle_failure`: 2.7% (raw) → 0.8% (skill)
- Persistent failures: `algorithmic_logic_error` (7.4% with skills), `static_verification_without_runtime` (11.7% with skills)

### Invocation Failures

- `skill_guidance_misapplied_or_ignored`: 10.0% (skill) vs. 0.8% (raw) vs. 0.4% (workflow)
- `timeout_budget_exhaustion`: 10.6% (workflow) vs. 1.7% (raw) vs. 4.4% (skill)

## Theoretical and Practical Implications

### Theoretical Implications

1. **Skills are procedural, not factual**: The dominant mechanism (65.7% procedural anchoring vs. 4.5% knowledge injection) reframes skills as action-stabilizing abstractions rather than information repositories.

2. **Skill utility is a lifecycle problem**: Success depends on a pipeline—distillation, transfer/retrieval, invocation, and adaptation—not on a single memory-injection mechanism.

3. **Retrieval accuracy ≠ task success**: Exact ground-truth skill invocation is neither sufficient nor strictly necessary. Related non-ground-truth skills can provide partial procedural support.

4. **Representation matters**: The same underlying trajectories produce different outcomes depending on whether they are packaged as workflow memory or distilled skills, isolating representation as a causal factor.

### Practical Implications

1. **Skill design**: Focus on compressing noisy trajectories into clean procedural guidance (setup steps, tool sequences, verification checks) rather than injecting domain facts.

2. **Evaluation methodology**: Beyond aggregate success rates, use paired trajectory analysis with mechanism labels to attribute gains/failures to concrete behaviors.

3. **Retrieval systems**: Semantic confusability (similar distractors) is a more important stressor than pool size alone; systems should be tested with hard negatives.

4. **Agent design**: Skills should be treated as *adaptable guidance*, not rigid scripts—agents must decide applicability, adapt, and know when to abandon.

## Conclusion

This paper shifts skill evaluation from aggregate success rates to mechanistic understanding, asking **when** skills help, **why** they work, and **where** they fail. Through controlled experiments and contrastive trajectory analysis:

- Skills are most effective as **procedural anchors** that stabilize execution
- Skills improve execution robustness (setup, output formats, service management, shell commands)
- Skills fail under **brittle assumptions**, **incompatible contexts**, or **insufficient adaptation**
- Retrieval failures and invocation failures are distinct bottlenecks

The findings motivate a **lifecycle view** of skill use: building better self-evolving agents requires not only generating more skills, but also improving how agents represent, retrieve, and leverage procedural knowledge.

### Limitations

- Focus on terminal/tool-using benchmarks; does not cover long-horizon web interaction or open-ended collaboration
- Limited number of agent–model configurations
- Taxonomy derived from stratified open-coding sample (~3% of normalized records); rare behavioral modes may be underrepresented

---

_Markdown view of https://picx.dev/p/whkUoq, served by PicX — AI-generated visual whiteboard summaries of research papers._