Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills

Summary (Overview)

• Core contribution: The paper identifies operational knowledge as a missing layer in autonomous ML research agents, complementing the conventional two-component view of (model backbone + harness). This knowledge layer captures the domain-specific know-how that separates knowing a method from making it work.

• DisCo framework: Introduces DisCo, a skill-powered research agent that both creates skills (creator mode) and uses them during research (researcher mode), operating through a four-stage distillation pipeline: scope → ground → construct → verify.

• AREX-Skill Library: Scales task-agnostic distillation across the open ecosystem to produce 5,000+ verified skills distilled from 1,000 widely used ML repositories, organized into 20 areas and 178 capability families with a library-level router.

• Benchmark gains: With the GPT-5.5 backbone, research harness, and downstream execution budget held fixed, the skill-equipped agent scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet compared to the same agent without skills.

• Key insight: Distilled skills provide selective operating context that replaces trial-and-error exploration with reusable, verified procedures, with gains that grow with task difficulty.

---

Introduction and Theoretical Foundation

The Missing Layer: Operational Knowledge

The paper formalizes a research task as:

$$\tau = (q, \mathcal{D}, \mathcal{E}, g), \tag{1}$$

where $q$ states the problem, $\mathcal{D}$ is the data and material, $\mathcal{E}$ is the environment (including tools and budget), and $g$ is the target outcome. Solving $\tau$ means producing artifacts $y$ that fulfill $g$.

A conventional agentic system is described by two components:

$$\mathcal{A} = (M_\theta, H), \tag{2}$$

where $M_\theta$ is the LLM backbone (supplying understanding, reasoning, planning, execution) and $H$ is the harness (supplying orchestration, memory, verification, iterative refinement). At step $t$:

$$a_t \sim \pi_\theta(a \mid \tau, h_t, H), \qquad o_t = \mathcal{E}(a_t), \tag{3}$$

Defining Operational Knowledge

The paper's central theoretical contribution is formalizing the research agent as:

$$\mathcal{A}_{\mathrm{res}} = (M_\theta, H, \mathcal{K}), \tag{4}$$

where $\mathcal{K}$ is operational knowledge made available as explicit operating context, so Eq. (3) becomes $a_t \sim \pi_\theta(a \mid \tau, h_t, H, \mathcal{K})$.

Operational knowledge has two constituents:

1. Capability: Methods, code, models, and APIs packaged into units the agent can invoke
2. Usage policy: Conditions, reasons, and procedures governing when and how to use each unit

This distinguishes $\mathcal{K}$ from $H$: the harness specializes how the agent explores, while $\mathcal{K}$ specializes what the agent knows to consider.

Declarative vs. Operational knowledge: Papers, repositories, and blogs state facts (declarative), but operational knowledge translates those facts into task-level actions. The central methodological problem is producing operational knowledge automatically and at scale from declarative sources.

---

Methodology

Skills and Skill Graphs

Operational knowledge is instantiated as a set of skills:

$$\mathcal{K} = \{S_1, \dots, S_m\}, \tag{5}$$

Each skill has a three-layer structure:

$$S = (\underbrace{\text{SKILL.md}}_{\text{knowledge interface}}, \underbrace{\text{references/}}_{\text{knowledge substrate}}, \underbrace{\text{scripts/}}_{\text{execution interface}}), \tag{6}$$

• SKILL.md: The knowledge interface—read up front, states goals, procedures, tool usage, failure modes
• references/: Knowledge substrate—deeper material loaded only when needed (progressive disclosure)
• scripts/: Execution interface—executable wrappers with defined inputs/outputs

Skills from one source form a skill graph:

$$\mathcal{G} = (\mathcal{S}, \mathcal{L}), \quad \mathcal{S} = \{S_i\}_{i=1}^n, n \geq 1, \quad \mathcal{L} \subseteq \{(S_i, S_j) \in \mathcal{S} \times \mathcal{S} | i \neq j\}, \tag{7}$$

Skill Distillation Pipeline

All distillation follows a four-stage process:

$$z \xrightarrow{\text{scope}} \mathcal{Q} \xrightarrow{\text{ground}} \mathcal{X} \xrightarrow{\text{construct}} \tilde{\mathcal{G}} \xrightarrow{\text{verify}} (\mathcal{G}, R), \tag{8}$$

where $z$ is the anchor, $\mathcal{Q}$ is the capability set, $\mathcal{X}$ is the evidence, $\tilde{\mathcal{G}}$ is the candidate graph, and $(\mathcal{G}, R)$ is the verified graph with construction record $R$.

Two forms of distillation:

Creator and Researcher Modes

• Creator mode: Carries out distillation, deposits verified graphs in the AREX-Skill Library (paid once per source, amortized across tasks)
• Researcher mode: Solves tasks with $\mathcal{K}$ drawn from the library, following progressive disclosure—reads entry points, follows only relevant links

---

Empirical Validation / Results

Setup

• Harness: Codex (fixed)
• Backbone: GPT-5.5 with xhigh reasoning effort (fixed)
• Only variable: Whether DisCo-distilled skills are provided

MLE-bench (Full 75 Competitions)

Table 1: Main results on MLE-bench (Any Medal %)

Key findings:

• Overall improvement: 31.11% → 72.89% (+134.3% relative)
• High-difficulty gains are largest: 13.33% → 62.22% (+366.8% relative, 4.67×)
• Surpasses strongest public baseline (64.44% → 72.89%) without a custom harness

PaperBench (Full 20 Papers)

Table 2: PaperBench replication scores (selected highlights)

Key findings:

• Average replication score: 29.45% → 39.59% (+34.4% relative)
• Improvements on 18 of 20 tasks; regressions only on sample-specific-masks (−5.07) and stay-on-topic (−4.52)
• Largest relative gains on low-baseline tasks (ftrl: 1.50 → 17.17, 11.4×)

FrontierCS (Agent Track, 188 Tasks)

Table 3: FrontierCS Agent Track results

Key findings:

• Score: 70.63 → 77.14 (+9.22% relative)
• 95% CI for mean improvement: [3.41, 9.83] via paired bootstrap
• Largest lift on tasks below 50: mean rises from 19.43 to 45.99 (+26.56)
• Additional usage (tokens, steps, tool calls) is uncorrelated with gains (Spearman's ρ ≈ 0.006–0.015)
• Pareto-dominates Claude Code configurations on Score, tokens, steps, and tool calls

PassNet (200 Samples)

Table 4: PassNet eval list results

Key findings:

• AS Score: 1.343 → 1.5313 (+14.0% relative)
• Failed samples reduced 64.3% (14 → 5)
• Correctness improved from 81.35% to 90.76%
• Surpasses TorchInductor's AS Score (1.5313 vs. 1.419)

---

Theoretical and Practical Implications

Theoretical Implications

1. Operational knowledge as a distinct layer: The paper formalizes a three-component view of research agents $\mathcal{A}_{\mathrm{res}} = (M_\theta, H, \mathcal{K})$, arguing that the two-component view leaves domain-specific knowledge unspecified and unaddressed.

2. Capability vs. policy distinction: The separation of capability (executable units) from usage policy (conditions and procedures) provides a principled framework for understanding what makes knowledge operational rather than merely declarative.

3. Scalability through cost asymmetry: Creator mode is paid once per source and amortized across tasks; researcher mode pays only for what a task opens. This asymmetry makes the knowledge layer scalable without crowding the context window.

4. Verification as distillation's differentiator: Verification is what separates distillation from summarization—no skill is admitted on source strength alone, and remaining gaps are recorded in the construction record $R$.

Practical Implications

1. Harness-agnostic knowledge: Distilled skills work with any compatible harness (demonstrated with Codex), separating knowledge content from control-loop design.

2. Gains grow with difficulty: The largest improvements occur on high-difficulty tasks (MLE-bench High: +366.8%), suggesting skills are most valuable when trial-and-error is most costly.

3. Recovery of low-performing tasks: On FrontierCS, tasks below 50 see mean gains of +26.56 points, with 30 tasks crossing the 50-point threshold—skills rescue tasks that would otherwise stall.

4. Cost-effectiveness: Codex + AREX-Skill Pareto-dominates Claude Code configurations using 3.29× more tokens, achieving higher scores with fewer resources.

---

Conclusion

Main Takeaways

1. Operational knowledge is the missing layer for autonomous ML research agents, complementing the model and harness. The harness governs how an agent researches; distilled skills determine what it knows when research begins.

2. DisCo demonstrates automatic, scalable distillation: The four-stage pipeline (scope → ground → construct → verify) converts declarative sources (repositories, papers) into verified, operational skills in both task-agnostic and task-oriented forms.

3. The AREX-Skill Library scales to the ecosystem: 5,000+ verified skills from 1,000 widely used ML repositories, organized into 20 areas and 178 capability families with a router for progressive disclosure.

4. Consistent, substantial benchmark gains: 134.3% (MLE-bench), 34.4% (PaperBench), 9.2% (FrontierCS), and 14.0% (PassNet) relative improvements under matched backbone, harness, and budget—demonstrating that adding operational knowledge, not just stronger control loops, improves autonomous research agents.

Future Directions

• Routing precision: Two PaperBench regressions suggest a precision-recall trade-off in skill retrieval; better routing or explicit fallback to unguided reasoning when skills are a poor match may reduce this failure mode.

• Extended verification: The paper-derived skill workflow can be extended to additional papers as verification budget permits.

• Broader source anchors: Beyond repositories and papers, tutorials and technical blogs could serve as additional task-agnostic sources.

• Harness compatibility: The same distilled skills can serve any compatible harness that exposes agent-readable skills, suggesting further evaluation across diverse agent architectures.