# POSEIDON: Scaling Training Environments for Scientific Agents from Code Repositories

> POSEIDON's automated pipeline generates 97,659 verifiable scientific-agent training environments from 7,110 repositories, and fine-tuning on them consistently boosts model performance across six benchmarks.

- **Source:** [arXiv](https://arxiv.org/abs/2610.32448)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/RH2mX6
- **Whiteboard:** https://picx.dev/p/RH2mX6/image

## Summary

## Summary (Overview)

- **POSEIDON** is an automated pipeline that generates diverse, verifiable training environments for scientific agents from code repositories at scale, achieving **9.3× higher yield** than prior approaches (D3-Gym) under matched repository budgets.
- The pipeline decomposes scientific workflows into **four complementary task types**: Reproduction, Subflow, Completion, and Correction, each targeting distinct scientific-agent capabilities.
- A **recursive improvement loop** recovers 49.4% of initially failed environment candidates through generator–validator feedback.
- The resulting dataset, **PEARL**, contains **97,659 executable and verifiable environments** from 7,110 repositories across 10 scientific disciplines—the largest multidisciplinary collection of scientific-agent training environments to date.
- Fine-tuning Qwen3.5 models (9B, 27B, 35B-A3B) on PEARL yields **consistent gains across six scientific-agent benchmarks**, narrowing the gap with frontier models.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Scientific research requires agents to execute **scientific workflows**—coordinated sequences of data processing and computational steps leading to evidence-supported conclusions. Unlike simple question-answering or isolated code generation, these workflows demand integration of:

- Scientific reasoning
- Coding proficiency
- Tool use capabilities

Despite the importance of these capabilities, even frontier models struggle. For instance, GPT-5.6 Sol achieves only **28.7% pass rate** on OpenAI's GeneBench-Pro at the highest reasoning level.

### The Scaling Challenge

Training environments for scientific agents remain severely limited. D3-Gym pioneered automated synthesis from code repositories but produced only **565 environments from 239 repositories** (~2.4 per repository), treating each workflow as a single end-to-end task.

### Central Research Question

> How can we efficiently exploit executable scientific workflows within code repositories to generate high-quality, diverse training environments at scale?

### Theoretical Foundation

The key insight is that **a scientific workflow provides richer supervision than a single end-to-end task** through structured modifications to its constituent steps. This aligns with prior work demonstrating that transforming existing code artifacts can substantially increase supervision quantity and diversity.

---

## Methodology

### Problem Formulation

A SciAgent training environment is formally defined as:

$$\mathbb{E} = \{(T_i, D_i, S_i, V_i)\}_{i=1}^{N}$$

where:
- $T_i$ = task description
- $D_i$ = input dataset
- $S_i$ = executable sandbox
- $V_i$ = evaluation script

An agent policy $\pi_\theta$ generates a trajectory $\tau = (r_1, a_1, o_1, \ldots, r_T, a_T, o_T)$ with interleaved reasoning ($r_t$) and actions ($a_t$).

### Environment Synthesis Objectives

Four properties are required: **(1) Data-Dependent**, **(2) Executable**, **(3) Verifiable**, **(4) Diverse**.

### Pipeline Stages

#### Stage 1: Repository Discovery
- **Keyword Pool Generation**: LLM-expanded domain keywords (e.g., GROMACS, MDAnalysis, PLUMED) filtered by GitHub search hit counts (30–5,000 range)
- **Repository Pool Collection**: Criteria = ≥20 stars, contains scientific data, backed by published paper, belongs to core scientific discipline
- **Executable Sandbox Construction**: Statically analyzed dependencies + parsed manifests
- **Result**: 7,110 repositories (47% biology, 11.8% pharmacology, 10.7% physics, etc.)

#### Stage 2: Task Synthesis
**Seed Program Extraction**: DeepSeek-V4-Flash groups related code files into cohesive sets, rewrites into single entry-point scripts, executes to validate.

**Four Task Types Derived from Workflow Graphs**:
| Task Type | Construction | Targeted Capability |
|-----------|-------------|---------------------|
| **Reproduction** | Full workflow inputs/outputs | End-to-end planning & execution |
| **Subflow** | Connected subgraph extraction | Local dependency understanding |
| **Completion** | Remove functional block | Dependency-aware workflow completion |
| **Correction** | Inject controlled error | Scientific error diagnosis & repair |

#### Stage 3: Verification
- Multi-stage pipeline assessing **environment reliability** (complete, executable, reproducible, gradable) and **scientific validity** (meaningful, correct, consistent)
- GPT-5.5 validator + DeepSeek-V4-Flash generator in **recursive improvement loop** (up to 3 rounds)
- Difficulty scoring (1–4) based on domain knowledge and analysis depth
- Manual review of 600 sampled tasks: **95.5% acceptance rate**

---

## Empirical Validation / Results

### Main Results (Table 2)

Training on PEARL consistently improves performance across model scales:

| Model | BioCoder | SciCode Main | MA-Bench | SA-Bench | BixBench MCQ | BixBench Open | BioMystery | Avg |
|-------|----------|--------------|----------|----------|--------------|---------------|------------|-----|
| **Qwen3.5-9B** | 46.8 → **51.9** (+5.1) | 3.1 → **9.2** (+6.1) | 30.2 → **34.0** (+3.8) | 59.0 → **63.0** (+4.0) | 26.5 → **33.3** (+6.8) | 48.3 → **50.7** (+2.4) | 35.1 → **37.6** (+2.5) | 33.2 → **37.5** (+4.3) |
| **Qwen3.5-35B-A3B** | 54.5 → **60.4** (+5.9) | 13.8 → **20.0** (+6.2) | 37.5 → **45.1** (+7.6) | 71.3 → **78.3** (+7.0) | 38.2 → **47.1** (+8.9) | 57.1 → **69.3** (+12.2) | 32.2 → **37.1** (+4.9) | 41.6 → **48.3** (+6.7) |
| **Qwen3.5-27B** | 57.1 → **67.5** (+10.4) | 12.3 → **16.9** (+4.6) | 42.7 → **50.3** (+7.6) | 75.7 → **79.0** (+3.3) | 34.3 → **50.0** (+15.7) | 61.5 → **69.8** (+8.3) | 34.1 → **39.5** (+5.4) | 44.3 → **51.5** (+7.2) |

**Key findings**:
- Qwen3.5-27B matches GLM-5.2 on BioCoder (67.5%)
- Qwen3.5-35B-A3B matches GPT-5.5 on SciCode main (20.0%)
- 27B model outperforms Hy4-Preview (79.0% vs 69.0% on MedAgentBench)

### Data-Scale-Controlled Comparison (Table 4)

With matched budgets (~2,000 trajectories):

| Training Data | Avg | Improvement Pattern |
|---------------|-----|---------------------|
| MedAgentGym | 43.3 | Improves 4/8 settings, hurts BixBench MCQ (−8.1) |
| D3-Gym | 41.9 | Improves 3/8 settings, hurts BixBench Open (−8.2) |
| **PEARL (2k)** | **46.4** | **Improves 7/8 settings** (+0.7 to +5.2) |

### Solvability by Difficulty (Table 3)

| Difficulty | Success Rate | Avg. Steps |
|------------|--------------|------------|
| D1 | 94.0% | 6.2 |
| D2 | 86.0% | 8.1 |
| D3 | 57.0% | 15.1 |
| D4 | 41.0% | 16.4 |

### Ablation Studies

**Repository Diversity** (top row of Figure 5): Increasing repository coverage from 20%→100% under fixed trajectory budget consistently improves performance across all benchmarks.

**Environment Difficulty** (bottom row of Figure 5): No consistent trend—difficulty correlates with solvability but doesn't affect training effectiveness within evaluated range (2.4–3.2).

**Task Type Removal (Table 5)**:

| Variant | BioCoder | MA-Bench | BixBench MCQ | SA-Bench | Avg |
|---------|----------|----------|--------------|----------|-----|
| Full PEARL | 60.4 | 78.3 | 69.3 | 47.1 | 63.8 |
| w/o Reproduction | 54.5 (−5.9) | 70.7 (−7.6) | 63.9 (−5.4) | 44.1 (−3.0) | 58.3 (−5.5) |
| w/o Subflow | 55.8 (−4.6) | 73.0 (−5.3) | 60.5 (−8.8) | 41.2 (−5.9) | 57.6 (−6.2) |
| w/o Completion | 59.7 (−0.7) | 71.7 (−6.6) | 62.0 (−7.3) | 43.1 (−4.0) | 59.1 (−4.7) |
| w/o Correction | 55.2 (−5.2) | 72.0 (−6.3) | 62.0 (−7.3) | 46.1 (−1.0) | 58.8 (−5.0) |

**Removing any single task type degrades performance on every benchmark**, confirming non-redundant contributions.

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Workflow Decomposition Principle**: Demonstrates that scientific workflows contain multiple complementary supervision signals beyond end-to-end execution, enabling more efficient exploitation of existing code repositories.

2. **Recursive Improvement Framework**: The generator–validator feedback loop (recovering 49.4% of initially failed candidates) provides a generalizable mechanism for scaling verifiable data synthesis.

3. **Diversity over Difficulty**: Ablations show that **repository diversity** drives training effectiveness, while environment difficulty alone doesn't—informing future data collection strategies.

### Practical Implications

1. **Data Efficiency**: PEARL's diverse composition yields gains even with <10% of full data, highlighting the importance of breadth over volume.

2. **Model Democratization**: Compact open models (9B–27B) can approach or exceed frontier model performance on specific benchmarks after PEARL training.

3. **Reproducibility Infrastructure**: Each environment includes task description, input data, executable sandbox, reference solution, and automatic evaluation script—enabling standardized agent training and evaluation.

4. **Disciplinary Breadth**: Coverage of 10 scientific disciplines (with intentional oversampling of biology) supports transfer across domains.

---

## Conclusion

POSEIDON introduces an automated pipeline that scales training environment generation from scientific code repositories by:
- Deriving **four complementary task types** from executable workflows
- Implementing **recursive improvement** to recover flawed but fixable candidates

PEARL, the resulting dataset of **97,659 environments from 7,110 repositories across 10 disciplines**, represents the largest multidisciplinary collection of scientific-agent training environments. Supervised fine-tuning on PEARL consistently improves Qwen3.5 models across six benchmarks, with ablations confirming that both repository diversity and complementary task types contribute to gains.

**Future directions**: The scale and diversity of PEARL open opportunities for training more capable scientific discovery agents, potentially extending to reinforcement learning paradigms and broader scientific domains beyond the current ten disciplines.

---

_Markdown view of https://picx.dev/p/RH2mX6, served by PicX — AI-generated visual whiteboard summaries of research papers._
