# Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills

> Distilling GitHub repositories into verified operational skills boosts autonomous ML research agents by up to 134% on MLE-bench, with gains growing on harder tasks.

- **Source:** [arXiv](https://arxiv.org/abs/2609.02749)
- **Published:** 2026-09-04
- **Permalink:** https://picx.dev/p/rLnciK
- **Whiteboard:** https://picx.dev/p/rLnciK/image

## Summary

# Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills

## Summary (Overview)

- **Core contribution**: The paper identifies **operational knowledge** as a missing layer in autonomous ML research agents, complementing the conventional two-component view of (model backbone + harness). This knowledge layer captures the domain-specific know-how that separates knowing a method from making it work.

- **DisCo framework**: Introduces DisCo, a skill-powered research agent that both **creates skills** (creator mode) and **uses them during research** (researcher mode), operating through a four-stage distillation pipeline: scope → ground → construct → verify.

- **AREX-Skill Library**: Scales task-agnostic distillation across the open ecosystem to produce **5,000+ verified skills** distilled from **1,000 widely used ML repositories**, organized into **20 areas** and **178 capability families** with a library-level router.

- **Benchmark gains**: With the GPT-5.5 backbone, research harness, and downstream execution budget held fixed, the skill-equipped agent scores **134.3% higher on MLE-bench**, **34.4% higher on PaperBench**, **9.2% higher on FrontierCS**, and **14.0% higher on PassNet** compared to the same agent without skills.

- **Key insight**: Distilled skills provide selective operating context that replaces trial-and-error exploration with reusable, verified procedures, with gains that grow with task difficulty.

---

## Introduction and Theoretical Foundation

### The Missing Layer: Operational Knowledge

The paper formalizes a research task as:

$$\tau = (q, \mathcal{D}, \mathcal{E}, g), \tag{1}$$

where $q$ states the problem, $\mathcal{D}$ is the data and material, $\mathcal{E}$ is the environment (including tools and budget), and $g$ is the target outcome. Solving $\tau$ means producing artifacts $y$ that fulfill $g$.

A conventional agentic system is described by two components:

$$\mathcal{A} = (M_\theta, H), \tag{2}$$

where $M_\theta$ is the LLM backbone (supplying understanding, reasoning, planning, execution) and $H$ is the harness (supplying orchestration, memory, verification, iterative refinement). At step $t$:

$$a_t \sim \pi_\theta(a \mid \tau, h_t, H), \qquad o_t = \mathcal{E}(a_t), \tag{3}$$

### Defining Operational Knowledge

The paper's central theoretical contribution is formalizing the research agent as:

$$\mathcal{A}_{\mathrm{res}} = (M_\theta, H, \mathcal{K}), \tag{4}$$

where $\mathcal{K}$ is operational knowledge made available as explicit operating context, so Eq. (3) becomes $a_t \sim \pi_\theta(a \mid \tau, h_t, H, \mathcal{K})$.

**Operational knowledge has two constituents**:
1. **Capability**: Methods, code, models, and APIs packaged into units the agent can invoke
2. **Usage policy**: Conditions, reasons, and procedures governing when and how to use each unit

This distinguishes $\mathcal{K}$ from $H$: the harness specializes *how* the agent explores, while $\mathcal{K}$ specializes *what* the agent knows to consider.

**Declarative vs. Operational knowledge**: Papers, repositories, and blogs state facts (declarative), but operational knowledge translates those facts into task-level actions. The central methodological problem is producing operational knowledge **automatically and at scale** from declarative sources.

---

## Methodology

### Skills and Skill Graphs

Operational knowledge is instantiated as a set of skills:

$$\mathcal{K} = \{S_1, \dots, S_m\}, \tag{5}$$

Each skill has a three-layer structure:

$$S = (\underbrace{\text{SKILL.md}}_{\text{knowledge interface}}, \underbrace{\text{references/}}_{\text{knowledge substrate}}, \underbrace{\text{scripts/}}_{\text{execution interface}}), \tag{6}$$

- **SKILL.md**: The knowledge interface—read up front, states goals, procedures, tool usage, failure modes
- **references/**: Knowledge substrate—deeper material loaded only when needed (progressive disclosure)
- **scripts/**: Execution interface—executable wrappers with defined inputs/outputs

Skills from one source form a **skill graph**:

$$\mathcal{G} = (\mathcal{S}, \mathcal{L}), \quad \mathcal{S} = \{S_i\}_{i=1}^n, n \geq 1, \quad \mathcal{L} \subseteq \{(S_i, S_j) \in \mathcal{S} \times \mathcal{S} | i \neq j\}, \tag{7}$$

### Skill Distillation Pipeline

All distillation follows a four-stage process:

$$z \xrightarrow{\text{scope}} \mathcal{Q} \xrightarrow{\text{ground}} \mathcal{X} \xrightarrow{\text{construct}} \tilde{\mathcal{G}} \xrightarrow{\text{verify}} (\mathcal{G}, R), \tag{8}$$

where $z$ is the anchor, $\mathcal{Q}$ is the capability set, $\mathcal{X}$ is the evidence, $\tilde{\mathcal{G}}$ is the candidate graph, and $(\mathcal{G}, R)$ is the verified graph with construction record $R$.

**Two forms of distillation**:

| Form | Anchor | Scope | Grounding | Construction | Verification |
|------|--------|-------|-----------|--------------|--------------|
| **Task-agnostic** | Source $z = c$ | Source Understanding + Capability Identification | Knowledge Extraction | Tool Encapsulation + Skill Packaging | Skill Verification |
| **Task-oriented** | Task $z = \tau$ | Task Decomposition + Capability Gap Analysis | Source Discovery | Skill Generation | Skill Verification |

### Creator and Researcher Modes

- **Creator mode**: Carries out distillation, deposits verified graphs in the AREX-Skill Library (paid once per source, amortized across tasks)
- **Researcher mode**: Solves tasks with $\mathcal{K}$ drawn from the library, following progressive disclosure—reads entry points, follows only relevant links

---

## Empirical Validation / Results

### Setup

- **Harness**: Codex (fixed)
- **Backbone**: GPT-5.5 with xhigh reasoning effort (fixed)
- **Only variable**: Whether DisCo-distilled skills are provided

### MLE-bench (Full 75 Competitions)

**Table 1: Main results on MLE-bench (Any Medal %)**

| Agent | Backbone | Low (n=22) | Medium (n=38) | High (n=15) | All (n=75) |
|-------|----------|------------|---------------|-------------|------------|
| Famou-Agent 2.0 | Gemini-3-Pro-Preview | 80.30 ± 1.52 | 64.04 ± 2.32 | 42.22 ± 2.22 | 64.44 ± 1.18 |
| AIBuildAI | Claude-Opus-4.6 | 77.27 ± 0.00 | 61.40 ± 0.88 | 46.67 ± 0.00 | 63.11 ± 0.44 |
| Codex (no skills) | GPT-5.5 | 42.42 ± 6.60 | 31.58 ± 1.52 | 13.33 ± 3.85 | 31.11 ± 2.22 |
| **Codex + AREX-Skill** | **GPT-5.5** | **86.36 ± 2.62** | **69.30 ± 3.16** | **62.22 ± 2.22** | **72.89 ± 1.18** |

Key findings:
- Overall improvement: 31.11% → 72.89% (+134.3% relative)
- High-difficulty gains are largest: 13.33% → 62.22% (+366.8% relative, 4.67×)
- Surpasses strongest public baseline (64.44% → 72.89%) without a custom harness

### PaperBench (Full 20 Papers)

**Table 2: PaperBench replication scores (selected highlights)**

| Paper | GPT-5.5 Codex | Codex + AREX-Skill | Δ |
|-------|---------------|--------------------|----|
| rice | 7.94 | 48.51 | **+40.57** |
| sequential-neural | 41.67 | 65.37 | +23.70 |
| what-will-my-model-forget | 9.35 | 30.45 | +21.10 |
| pinn | 40.64 | 58.10 | +17.46 |
| **Average Score** | **29.45** | **39.59** | **+10.14** |

Key findings:
- Average replication score: 29.45% → 39.59% (+34.4% relative)
- Improvements on 18 of 20 tasks; regressions only on sample-specific-masks (−5.07) and stay-on-topic (−4.52)
- Largest relative gains on low-baseline tasks (ftrl: 1.50 → 17.17, 11.4×)

### FrontierCS (Agent Track, 188 Tasks)

**Table 3: FrontierCS Agent Track results**

| Agent | Backbone | Score | Avg. Steps | Avg. Tool Calls | Avg. Tokens |
|-------|----------|-------|------------|-----------------|-------------|
| Claude Code | Claude Opus 4.8 | 74.5 | 355.4 | 145.7 | 14.72M |
| Codex (no skills) | GPT-5.5 | 70.63 | 55.9 | 64.7 | 2.46M |
| **Codex + AREX-Skill** | **GPT-5.5** | **77.14** | 88.7 | 105.0 | 4.47M |

Key findings:
- Score: 70.63 → 77.14 (+9.22% relative)
- 95% CI for mean improvement: [3.41, 9.83] via paired bootstrap
- Largest lift on tasks below 50: mean rises from 19.43 to 45.99 (+26.56)
- Additional usage (tokens, steps, tool calls) is uncorrelated with gains (Spearman's ρ ≈ 0.006–0.015)
- Pareto-dominates Claude Code configurations on Score, tokens, steps, and tool calls

### PassNet (200 Samples)

**Table 4: PassNet eval list results**

| Method | AS Score | G-Mean Speedup | Correctness | Fast_1 | Failed samples |
|--------|----------|----------------|-------------|--------|----------------|
| TorchInductor | 1.419 | 1.505 | 79.70% | 23.60% | 0 |
| Codex + GPT-5.5 | 1.343 | 1.5891 | 81.35% | 28.48% | 14 |
| **Codex + GPT-5.5 + AREX-Skill** | **1.5313** | **1.6688** | **90.76%** | 26.72% | **5** |

Key findings:
- AS Score: 1.343 → 1.5313 (+14.0% relative)
- Failed samples reduced 64.3% (14 → 5)
- Correctness improved from 81.35% to 90.76%
- Surpasses TorchInductor's AS Score (1.5313 vs. 1.419)

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Operational knowledge as a distinct layer**: The paper formalizes a three-component view of research agents $\mathcal{A}_{\mathrm{res}} = (M_\theta, H, \mathcal{K})$, arguing that the two-component view leaves domain-specific knowledge unspecified and unaddressed.

2. **Capability vs. policy distinction**: The separation of capability (executable units) from usage policy (conditions and procedures) provides a principled framework for understanding what makes knowledge operational rather than merely declarative.

3. **Scalability through cost asymmetry**: Creator mode is paid once per source and amortized across tasks; researcher mode pays only for what a task opens. This asymmetry makes the knowledge layer scalable without crowding the context window.

4. **Verification as distillation's differentiator**: Verification is what separates distillation from summarization—no skill is admitted on source strength alone, and remaining gaps are recorded in the construction record $R$.

### Practical Implications

1. **Harness-agnostic knowledge**: Distilled skills work with any compatible harness (demonstrated with Codex), separating knowledge content from control-loop design.

2. **Gains grow with difficulty**: The largest improvements occur on high-difficulty tasks (MLE-bench High: +366.8%), suggesting skills are most valuable when trial-and-error is most costly.

3. **Recovery of low-performing tasks**: On FrontierCS, tasks below 50 see mean gains of +26.56 points, with 30 tasks crossing the 50-point threshold—skills rescue tasks that would otherwise stall.

4. **Cost-effectiveness**: Codex + AREX-Skill Pareto-dominates Claude Code configurations using 3.29× more tokens, achieving higher scores with fewer resources.

---

## Conclusion

### Main Takeaways

1. **Operational knowledge is the missing layer** for autonomous ML research agents, complementing the model and harness. The harness governs *how* an agent researches; distilled skills determine *what* it knows when research begins.

2. **DisCo demonstrates automatic, scalable distillation**: The four-stage pipeline (scope → ground → construct → verify) converts declarative sources (repositories, papers) into verified, operational skills in both task-agnostic and task-oriented forms.

3. **The AREX-Skill Library scales to the ecosystem**: 5,000+ verified skills from 1,000 widely used ML repositories, organized into 20 areas and 178 capability families with a router for progressive disclosure.

4. **Consistent, substantial benchmark gains**: 134.3% (MLE-bench), 34.4% (PaperBench), 9.2% (FrontierCS), and 14.0% (PassNet) relative improvements under matched backbone, harness, and budget—demonstrating that adding operational knowledge, not just stronger control loops, improves autonomous research agents.

### Future Directions

- **Routing precision**: Two PaperBench regressions suggest a precision-recall trade-off in skill retrieval; better routing or explicit fallback to unguided reasoning when skills are a poor match may reduce this failure mode.

- **Extended verification**: The paper-derived skill workflow can be extended to additional papers as verification budget permits.

- **Broader source anchors**: Beyond repositories and papers, tutorials and technical blogs could serve as additional task-agnostic sources.

- **Harness compatibility**: The same distilled skills can serve any compatible harness that exposes agent-readable skills, suggesting further evaluation across diverse agent architectures.

---

_Markdown view of https://picx.dev/p/rLnciK, served by PicX — AI-generated visual whiteboard summaries of research papers._