Summary (Overview)

• This paper systematically investigates when skills help LLM agents, why they work, and where they fail, moving beyond aggregate success-rate evaluations to mechanistic understanding.
• Through controlled experiments across Terminal-Bench and SkillsBench with multiple agent–model pairings (Codex + GPT-5.3-Codex, Gemini CLI + Gemini-3.1-Pro-Preview), the authors isolate four research questions: representation effects, outcome annotation, cross-framework transfer, and retrieval difficulty.
• A contrastive trajectory-analysis methodology is introduced: 8,135 trial records are normalized, 240 trajectories are open-coded, and 238 valid labels are consolidated into a taxonomy of 3 high-level categories and 12 skill-use modes (human validation: 95.8% exact agreement, Cohen's κ = 0.952).
• Key finding: Skills primarily work as procedural anchors (65.7% of mechanisms) rather than explicit knowledge injection (4.5%), improving over Workflow Memory by +6.06 percentage points (95% CI: [+0.76, +11.36]).
• Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6% to 3.3%, yet downstream success remains stable (~36–39%), showing that exact ground-truth invocation is neither sufficient nor necessary for success.

Introduction and Theoretical Foundation

Background

Large language model (LLM) agents are increasingly expected to improve through experience rather than solve each task from scratch. Recent agent systems store and reuse traces of prior execution: environment setup sequences, tool-use patterns, debugging routines, and verification steps. Among proposed memory forms, skills (Anthropic, 2025) have emerged as a compelling abstraction—not merely a record of past execution, but a compact description of what to do, what to check, and what pitfalls to avoid.

Skills promise three advantages over raw traces or direct workflow memories:

1. Compression of noisy experience into shorter context
2. Standardization of procedural knowledge into a stable format
3. Transferability of knowledge across related tasks

Motivation

Existing evaluations measure only whether skill-augmented agents solve more tasks. This reveals little about:

• What fundamentally changes in agent behavior before/after skill loading
• Which parts of execution are stabilized by skill guidance
• Why the same skill can help one task while harming another

The field lacks a principled understanding of what makes a skill genuinely valuable, leaving skill design to heuristic iteration, prompt tuning, or benchmark-specific trial and error.

Theoretical Foundation

The paper frames skill utility as a pipeline problem along four stages:

1. Representation — how prior experience is packaged (workflow memory vs. distilled skill)
2. Outcome signals — whether explicit success/failure annotations matter
3. Transfer — whether distilled guidance survives framework changes
4. Retrieval and invocation — how pool size and confusability affect identification and use

Methodology

Experimental Setup

Models and benchmarks:

• Two agent–model pairings for RQ1–RQ2: Codex + GPT-5.3-Codex and Gemini CLI + Gemini-3.1-Pro-Preview
• RQ3: cross-framework transfer from Codex to Gemini CLI
• RQ4: Qwen3-Embedding-0.6B for retrieval; Gemini CLI + Gemini-3.1-Pro-Preview and Codex + GPT-5.4 (GPT-5.3 no longer available)
• Benchmarks: Terminal-Bench, SkillsBench, Terminal-Bench-Pro
• Downstream execution: Harbor evaluation workflow, n = 5 unique trials per task, parallelism 20

Experimental Design

RQ1–RQ2: Controlled study of procedural experience representation

Three conditions compared:

• Raw: no prior experience
• Workflow Memory: cleaned procedural traces from prior executions
• Skill: standardized SKILL.md distilled from the same workflows

A fixed-budget composition grid varies source evidence from success-only to failure-only (5s0f through 0s5f). For each composition, Workflow Memory and Skill are built from the same trajectories, holding underlying experience constant while varying representation only.

No-hint variants: success/failure identities are hidden from the skill creator while preserving the same trajectories.

RQ3: Cross-framework transfer Skills and workflow memories constructed in Codex are evaluated in Gemini CLI, isolating framework transfer effects.

RQ4: Skill retrieval and downstream execution

Three independent experiments on matched candidate pools (each containing ground-truth skill + distractors; pool size 5–100; distractors random, similar, or dissimilar):

• Arm 1: Embedding-based ranking without task execution
• Arm 2: Explicit agent selection without execution
• Arm 3: Full-pool real execution with skill-use parsing after verification

Outputs from Arms 1–2 are not passed to Arm 3.

Taxonomy Construction

• 8,135 trial records normalized into a shared manifest; 7,837 contain agent transcripts
• Open-coding over 240 sampled trajectories; 238 valid unique labels retained
• 12-mode canonical taxonomy organized into 3 high-level categories (SC1: successful procedural anchoring; SC2: execution-layer/verification failures; SC3: invocation/applicability/boundary failures)
• Human validation: 714 trajectory–label checks (all confirmed); 95.8% exact agreement with Cohen's κ = 0.952 on taxonomy aggregation
• 528 paired triples constructed (SkillsBench: 144, Terminal-Bench 2.0: 186, Terminal-Bench-Pro: 198)

Empirical Validation / Results

RQ1: Skills as Procedural Anchors

Aggregate results:

• Skill: 61.9% oracle-status success rate
• Raw: 59.1%
• Workflow Memory: 55.9%
• Skill improves over Workflow Memory by +6.06 percentage points (95% bootstrap CI: [+0.76, +11.36])

Mechanism analysis:

• procedural_anchor accounts for 65.7% of skill mechanisms
• knowledge_injection accounts for only 4.5%
• Skill-guided success: 61.6% vs. workflow-guided success: 54.5%

Table 1: Task success rates (excerpt)

RQ2: Outcome Annotations

• Withholding outcome labels has little effect when source pools contain only successful trajectories
• Impact grows sharply once failed trajectories enter the pool
• Example (Gemini on Terminal-Bench-2 at 3s2f): 0.7462 with hints vs. 0.4000 without

RQ3: Cross-Framework Transfer

• Skills constructed in Codex remain useful when evaluated in Gemini CLI
• Distilled skills provide more portable representation than direct workflow memory across framework boundaries

RQ4: Retrieval and Downstream Use

Key finding: Retrieval accuracy does not determine downstream success.

• Actual-use precision falls from 29.6% (pool size 5) to 3.3% (pool size 100)
• Downstream success changes only from 36.4% to 39.3%
• Arm 3 recall remains 54.3–73.6% at k = 100 despite precision of only 0.7–8.1%
• Similar distractors are the dominant stressor for offline identification (Arm 1 top-1 precision on similar pools: 70.5% at k = 5 → 53.4% at k = 100)

Table 4: Effect of pool composition and size on SkillsBench (excerpt)

Execution-Layer Improvements

• SC2 execution-layer failures: 37.3% (raw) → 33.3% (workflow) → 23.5% (skill)
• environment_infrastructure_failure: 5.3% (raw) → 1.7% (workflow) → 0.2% (skill)
• output_format_schema_mismatch: 7.4% (raw) → 3.2% (skill)
• background_service_lifecycle_failure: 2.7% (raw) → 0.8% (skill)
• Persistent failures: algorithmic_logic_error (7.4% with skills), static_verification_without_runtime (11.7% with skills)

Invocation Failures

• skill_guidance_misapplied_or_ignored: 10.0% (skill) vs. 0.8% (raw) vs. 0.4% (workflow)
• timeout_budget_exhaustion: 10.6% (workflow) vs. 1.7% (raw) vs. 4.4% (skill)

Theoretical and Practical Implications

Theoretical Implications

1. Skills are procedural, not factual: The dominant mechanism (65.7% procedural anchoring vs. 4.5% knowledge injection) reframes skills as action-stabilizing abstractions rather than information repositories.

2. Skill utility is a lifecycle problem: Success depends on a pipeline—distillation, transfer/retrieval, invocation, and adaptation—not on a single memory-injection mechanism.

3. Retrieval accuracy ≠ task success: Exact ground-truth skill invocation is neither sufficient nor strictly necessary. Related non-ground-truth skills can provide partial procedural support.

4. Representation matters: The same underlying trajectories produce different outcomes depending on whether they are packaged as workflow memory or distilled skills, isolating representation as a causal factor.

Practical Implications

1. Skill design: Focus on compressing noisy trajectories into clean procedural guidance (setup steps, tool sequences, verification checks) rather than injecting domain facts.

2. Evaluation methodology: Beyond aggregate success rates, use paired trajectory analysis with mechanism labels to attribute gains/failures to concrete behaviors.

3. Retrieval systems: Semantic confusability (similar distractors) is a more important stressor than pool size alone; systems should be tested with hard negatives.

4. Agent design: Skills should be treated as adaptable guidance, not rigid scripts—agents must decide applicability, adapt, and know when to abandon.

Conclusion

This paper shifts skill evaluation from aggregate success rates to mechanistic understanding, asking when skills help, why they work, and where they fail. Through controlled experiments and contrastive trajectory analysis:

• Skills are most effective as procedural anchors that stabilize execution
• Skills improve execution robustness (setup, output formats, service management, shell commands)
• Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation
• Retrieval failures and invocation failures are distinct bottlenecks

The findings motivate a lifecycle view of skill use: building better self-evolving agents requires not only generating more skills, but also improving how agents represent, retrieve, and leverage procedural knowledge.

Limitations

• Focus on terminal/tool-using benchmarks; does not cover long-horizon web interaction or open-ended collaboration
• Limited number of agent–model configurations
• Taxonomy derived from stratified open-coding sample (~3% of normalized records); rare behavioral modes may be underrepresented