# SPT: Skills as Pre-Training Data for Agentic Language Models

> Skill pre-training on public multi-file skill packages improves agentic performance by up to 25 points over direct SFT while preserving general capabilities.

- **Source:** [arXiv](https://arxiv.org/abs/2608.26563)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/uTDjU3
- **Whiteboard:** https://picx.dev/p/uTDjU3/image

## Summary

## Summary (Overview)

- **Skill Pre-Training (SPT)** is a mid-training method that applies causal language modeling to public multi-file skill packages (SkillCorpus) before behavior-oriented post-training, improving agentic capabilities in language models.
- **SkillCorpus** contains 38,040 cleaned and decontaminated ClawHub packages with 218,277 files (~347.8 million tokens), providing reusable tool semantics and workflows as scalable pre-training data.
- **Reference Insert** is a novel reference-aware assembly strategy that places each supporting file near its first mention in the primary instruction, preserving cross-file relationships within packages.
- SPT consistently outperforms direct SFT and mid-training on general (Dolmino) or trajectory (AgentBank) data across three model scales (1.6B, 3B, 7B) and two SFT recipes (Tulu 3, xLAM-FC), while largely preserving general performance.
- Mixed training with 20%–50% skill data outperforms both single-source endpoints, and the advantage persists under SFT-then-RL (GRPO) pipelines.

## Introduction and Theoretical Foundation

### Background and Motivation

Large language models increasingly act as agents that use external tools for multi-step tasks. Tool use requires understanding tool functions and carrying out interactions over multiple steps. However, complete tool-use processes rarely appear in naturally collected corpora.

Most existing work fills this data gap with synthetic function calls and agent trajectories, but producing valid trajectories requires:
- Task environments
- Execution and verification
- Cost that grows with tool and task coverage

A trajectory records one execution rather than a reusable workflow, making it difficult to collect at the scale required for pre-training.

### Key Insight: Skills as Data

Skills are reusable instructions written by humans for AI agents. Unlike trajectories, skills describe *how* an agent should perform a task and can include references, scripts, templates, and configuration files. Public skill availability is growing rapidly—the number of skills on npm rose from 170,226 in February 2026 to 1,640,440 in June 2026 (9.6× growth).

### Theoretical Foundation

SPT is situated in the **mid-training framework**: a stage between broad pre-training and post-training that adapts a checkpoint on curated or domain-weighted corpora. The key distinction:

> "A skill explains *when* a capability applies, *how* to complete a task, *which* constraints to follow, and *which* supporting resources to use. Skills provide reusable task knowledge, whereas calls and trajectories provide direct behavioral supervision."

SPT uses the standard causal language-modeling objective but changes the training unit to reusable, multi-file workflow specifications.

## Methodology

### SkillCorpus Construction

1. **Source**: May 1, 2026 snapshot of public ClawHub packages from trusted, identity-verified, and independently audited publishers.
2. **Quality screening**: Iterative package-level quality screen removing ~2% of collected packages (deduplication, near-duplicate detection, autogenerated content rejection).
3. **Cleaning**: Removal of empty/short content, encoded blobs, lock files, binaries; redaction of sensitive fields; path normalization while preserving package boundaries.
4. **Decontamination**: Manual audit and DeepSeek-V4-Flash screening against API-Bank, MetaTool, APT-Bench, ToolEyes, and OLMES (~0.3% removed).

### Reference Insert Assembly

For each package:
1. Write a header with package name and description.
2. Scan SKILL.md in original order; when a line contains an unambiguous reference to a supporting file, insert a `<referenced_file_sep>` marker and the referenced file immediately after that line.
3. Each supporting file is inserted at most once, at its first mention.
4. Append remaining files with `<unreferenced_file_sep>` markers.
5. Pack assembled stream into 4,096-token training blocks.

This produces 84,905 training blocks (~347.8 million tokens).

### Data Mixture Formulation

Given a fixed mid-training token budget $B$, the training corpus $D_\alpha$ is constructed as:

$$
\begin{array}{c} 
p_{\alpha}(x) = \alpha p_{\text{skill}}(x) + (1 - \alpha) p_{\text{gen}}(x), \\
D_{\alpha} \sim p_{\alpha}, \qquad |D_{\alpha}|_{\text{tok}} = B, \qquad 0 \leq \alpha \leq 1. 
\end{array}\tag{1}
$$

Where:
- $\alpha$ = fraction of skill data
- $p_{\text{skill}}$ = empirical distribution over assembled skill blocks
- $p_{\text{gen}}$ = empirical distribution over general-data blocks
- $\alpha = 1$: pure-skill SPT
- $0 < \alpha < 1$: mixed skill and general data
- $\alpha = 0$: general-data mid-training control

### Experimental Setup

**Backbones** (all single-stage pre-training checkpoints, no annealing or post-training):
- MeCo-1.6B-DCLM-160B
- Instella-3B-Stage1
- OLMo-3-1025-7B

**Mid-training conditions**: Dolmino (general), AgentBank (trajectories), SkillCorpus (skills), or none.

**Post-training recipes**: Tulu 3 (general instruction tuning) or xLAM-FC (function-calling SFT).

**Training configuration**: 4×A100 80GB, BF16, sequence length 4,096, learning rate $2 \times 10^{-5}$, weight decay 0.1, cosine schedule. SFT: 3 epochs, sequence length 2,048, assistant-only supervision.

## Empirical Validation / Results

### Main Results

Across all six backbone/SFT settings, SkillCorpus achieves the best agentic performance:

| Mid-train | Post-train | API | Meta | APT | Eyes | Avg. | Gen. Avg. |
|---|---|---|---|---|---|---|---|
| **MeCo-1.6B** | | | | | | | |
| No mid-training | xLAM-FC | 9.82 | 12.50 | 10.22 | 22.87 | 13.85 | 59.18 |
| Dolmino | xLAM-FC | 15.76 | 16.16 | 17.47 | 23.43 | 18.20 | 58.91 |
| AgentBank | xLAM-FC | 18.39 | 17.13 | 19.47 | 27.49 | 20.62 | 58.99 |
| **SKILLCORPUS** | xLAM-FC | **29.13** | **21.30** | **23.02** | **28.14** | **25.40** | 59.00 |
| **Instella-3B** | | | | | | | |
| No mid-training | xLAM-FC | 20.77 | 17.85 | 14.72 | 32.40 | 21.44 | 72.28 |
| Dolmino | xLAM-FC | 31.42 | 18.02 | 17.60 | 36.09 | 25.78 | 72.05 |
| AgentBank | xLAM-FC | 33.54 | 22.46 | 19.72 | 38.37 | 28.52 | 70.34 |
| **SKILLCORPUS** | xLAM-FC | **45.34** | **31.02** | **23.97** | **53.40** | **38.43** | 71.43 |
| **OLMo-3-7B** | | | | | | | |
| No mid-training | xLAM-FC | 28.15 | 20.93 | 25.27 | 39.64 | 28.50 | 75.31 |
| Dolmino | xLAM-FC | 36.45 | 22.87 | 30.07 | 43.92 | 33.33 | 75.16 |
| AgentBank | xLAM-FC | 48.82 | 49.24 | 31.80 | 46.34 | 44.05 | 74.38 |
| **SKILLCORPUS** | xLAM-FC | **53.23** | **53.10** | **39.96** | **67.54** | **53.46** | 75.07 |

**Key findings**:
- SPT improves the four-benchmark agentic score by 9.11–24.96 over direct SFT, with general score change of only −0.85 to +0.51.
- Skill data adds 6.49–20.13 over Dolmino mid-training alone, showing corpus content (not just the extra training stage) drives most gains.
- SkillCorpus outperforms AgentBank by 4.07–9.91 under the same token budget.
- Gains are larger after xLAM-FC than Tulu 3 (e.g., 24.96 vs. 19.33 at 7B), showing SPT complements tool-focused SFT.

### Mixture Ratio Results

- **30% SkillCorpus** gives the highest agentic score: 38.60 (with Dolmino) and 37.58 (with SmolLM), vs. 15.56/9.07 for general-only and 22.06 for pure-skill.
- General scores at this ratio remain close to general-only baselines (58.64/58.83 vs. 59.29/59.26).
- Mixtures containing 20%–50% skill data outperform both endpoints; above 50%, agentic scores fall rapidly.

### Assembly Strategy Ablation

| Strategy | API-Bank | MetaTool | APT | ToolEyes | Gen. |
|---|---|---|---|---|---|
| DeepSeek-Coder | 24.40 | 15.22 | 16.14 | 25.01 | 57.30 |
| Random File | 17.76 | 10.55 | 14.72 | 21.34 | 57.34 |
| Metadata | 11.46 | 15.03 | 13.64 | 25.86 | 57.35 |
| Skill-FIM | 15.43 | 17.06 | 13.37 | 26.06 | 57.33 |
| **Reference Insert** | **26.47** | **17.39** | **17.97** | **26.39** | **58.20** |

Reference Insert beats DeepSeek-Coder by 1.87 and Random File by 5.96 (same files, same budget—purely from arrangement). Generic structural transformations (Metadata, Skill-FIM) do not provide the same benefit.

### RL Post-Training Robustness

Under identical Tulu 3 SFT + GRPO (with RLVR-GSM-MATH-IF-Mixed-Constraints, containing no tool-use supervision):

| Mid-training | API-Bank | MetaTool | APT | ToolEyes | Gen. |
|---|---|---|---|---|---|
| None | 5.44 | 1.16 | 4.54 | 8.64 | 40.10 |
| Dolmino | 7.69 | 4.75 | 7.66 | 8.90 | 40.96 |
| **SKILLCORPUS** | **15.93** | **10.88** | **9.43** | **11.52** | **41.49** |

SPT retains its advantage (margins of 4.69 and 6.99 over Dolmino and none) even when RL provides no tool-use supervision.

## Theoretical and Practical Implications

### Theoretical Implications

- **Skills vs. trajectories**: Skills state tool applicability, constraints, and reusable procedures explicitly, whereas trajectories record actions and observations from a particular execution. During causal language-modeling mid-training, skill data exposes reusable workflow knowledge directly instead of requiring the model to infer it from individual executions.
- **Complementarity with SFT**: Tool-focused SFT makes greater use of workflow knowledge learned during SPT, suggesting skill descriptions and function-call supervision contribute complementary signals at different training stages.
- **Reference-aware organization matters**: The 5.96-point gap between Reference Insert and Random File (identical content, same budget) demonstrates that *arrangement* of cross-file dependencies within a causal sequence significantly affects training value.

### Practical Implications

- **Data acquisition**: Skill packages can be collected at scale without task environments, tool execution, or synthetic trajectory generation, dramatically reducing the cost of agentic pre-training data.
- **Mixture guidance**: Skill data should be incorporated into a general-data mixture (20%–50%) rather than used standalone; general data contributes more than benchmark retention—its broader language coverage helps skills transfer to varied agentic tasks.
- **Robustness**: SPT works across model scales (360M–7B), post-training recipes (general SFT, function-calling SFT, SFT+RL), and general-data sources (Dolmino, SmolLM).

## Conclusion

This work demonstrates that skill packages are a valuable source of pre-training data for agentic language models. SPT consistently improves agentic performance over direct post-training and mid-training on general or trajectory data across multiple backbones while largely preserving general performance. The gains remain robust under different corpus compositions, package organizations, and post-training methods.

**Future directions** (implied by limitations):
- Extending beyond a single repository (ClawHub) to improve domain and language coverage
- Testing larger models and alternative compute regimes
- Measuring failures in specific agentic components (tool selection, argument construction, error recovery, unsafe actions)
- Evaluating interactive long-horizon behavior
- Addressing residual risks from executable workflows, personal data, and unsafe instructions

---

_Markdown view of https://picx.dev/p/uTDjU3, served by PicX — AI-generated visual whiteboard summaries of research papers._