Full text not available for this paper

Summary (Overview)

  • POSEIDON is an automated pipeline that generates diverse, verifiable training environments for scientific agents from code repositories at scale, achieving 9.3× higher yield than prior approaches (D3-Gym) under matched repository budgets.
  • The pipeline decomposes scientific workflows into four complementary task types: Reproduction, Subflow, Completion, and Correction, each targeting distinct scientific-agent capabilities.
  • A recursive improvement loop recovers 49.4% of initially failed environment candidates through generator–validator feedback.
  • The resulting dataset, PEARL, contains 97,659 executable and verifiable environments from 7,110 repositories across 10 scientific disciplines—the largest multidisciplinary collection of scientific-agent training environments to date.
  • Fine-tuning Qwen3.5 models (9B, 27B, 35B-A3B) on PEARL yields consistent gains across six scientific-agent benchmarks, narrowing the gap with frontier models.

Introduction and Theoretical Foundation

Background and Motivation

Scientific research requires agents to execute scientific workflows—coordinated sequences of data processing and computational steps leading to evidence-supported conclusions. Unlike simple question-answering or isolated code generation, these workflows demand integration of:

  • Scientific reasoning
  • Coding proficiency
  • Tool use capabilities

Despite the importance of these capabilities, even frontier models struggle. For instance, GPT-5.6 Sol achieves only 28.7% pass rate on OpenAI's GeneBench-Pro at the highest reasoning level.

The Scaling Challenge

Training environments for scientific agents remain severely limited. D3-Gym pioneered automated synthesis from code repositories but produced only 565 environments from 239 repositories (~2.4 per repository), treating each workflow as a single end-to-end task.

Central Research Question

How can we efficiently exploit executable scientific workflows within code repositories to generate high-quality, diverse training environments at scale?

Theoretical Foundation

The key insight is that a scientific workflow provides richer supervision than a single end-to-end task through structured modifications to its constituent steps. This aligns with prior work demonstrating that transforming existing code artifacts can substantially increase supervision quantity and diversity.


Methodology

Problem Formulation

A SciAgent training environment is formally defined as:

E={(Ti,Di,Si,Vi)}i=1N\mathbb{E} = \{(T_i, D_i, S_i, V_i)\}_{i=1}^{N}

where:

  • TiT_i = task description
  • DiD_i = input dataset
  • SiS_i = executable sandbox
  • ViV_i = evaluation script

An agent policy πθ\pi_\theta generates a trajectory τ=(r1,a1,o1,…,rT,aT,oT)\tau = (r_1, a_1, o_1, \ldots, r_T, a_T, o_T) with interleaved reasoning (rtr_t) and actions (ata_t).

Environment Synthesis Objectives

Four properties are required: (1) Data-Dependent, (2) Executable, (3) Verifiable, (4) Diverse.

Pipeline Stages

Stage 1: Repository Discovery

  • Keyword Pool Generation: LLM-expanded domain keywords (e.g., GROMACS, MDAnalysis, PLUMED) filtered by GitHub search hit counts (30–5,000 range)
  • Repository Pool Collection: Criteria = ≥20 stars, contains scientific data, backed by published paper, belongs to core scientific discipline
  • Executable Sandbox Construction: Statically analyzed dependencies + parsed manifests
  • Result: 7,110 repositories (47% biology, 11.8% pharmacology, 10.7% physics, etc.)

Stage 2: Task Synthesis

Seed Program Extraction: DeepSeek-V4-Flash groups related code files into cohesive sets, rewrites into single entry-point scripts, executes to validate.

Four Task Types Derived from Workflow Graphs:

Task TypeConstructionTargeted Capability
ReproductionFull workflow inputs/outputsEnd-to-end planning & execution
SubflowConnected subgraph extractionLocal dependency understanding
CompletionRemove functional blockDependency-aware workflow completion
CorrectionInject controlled errorScientific error diagnosis & repair

Stage 3: Verification

  • Multi-stage pipeline assessing environment reliability (complete, executable, reproducible, gradable) and scientific validity (meaningful, correct, consistent)
  • GPT-5.5 validator + DeepSeek-V4-Flash generator in recursive improvement loop (up to 3 rounds)
  • Difficulty scoring (1–4) based on domain knowledge and analysis depth
  • Manual review of 600 sampled tasks: 95.5% acceptance rate

Empirical Validation / Results

Main Results (Table 2)

Training on PEARL consistently improves performance across model scales:

ModelBioCoderSciCode MainMA-BenchSA-BenchBixBench MCQBixBench OpenBioMysteryAvg
Qwen3.5-9B46.8 → 51.9 (+5.1)3.1 → 9.2 (+6.1)30.2 → 34.0 (+3.8)59.0 → 63.0 (+4.0)26.5 → 33.3 (+6.8)48.3 → 50.7 (+2.4)35.1 → 37.6 (+2.5)33.2 → 37.5 (+4.3)
Qwen3.5-35B-A3B54.5 → 60.4 (+5.9)13.8 → 20.0 (+6.2)37.5 → 45.1 (+7.6)71.3 → 78.3 (+7.0)38.2 → 47.1 (+8.9)57.1 → 69.3 (+12.2)32.2 → 37.1 (+4.9)41.6 → 48.3 (+6.7)
Qwen3.5-27B57.1 → 67.5 (+10.4)12.3 → 16.9 (+4.6)42.7 → 50.3 (+7.6)75.7 → 79.0 (+3.3)34.3 → 50.0 (+15.7)61.5 → 69.8 (+8.3)34.1 → 39.5 (+5.4)44.3 → 51.5 (+7.2)

Key findings:

  • Qwen3.5-27B matches GLM-5.2 on BioCoder (67.5%)
  • Qwen3.5-35B-A3B matches GPT-5.5 on SciCode main (20.0%)
  • 27B model outperforms Hy4-Preview (79.0% vs 69.0% on MedAgentBench)

Data-Scale-Controlled Comparison (Table 4)

With matched budgets (~2,000 trajectories):

Training DataAvgImprovement Pattern
MedAgentGym43.3Improves 4/8 settings, hurts BixBench MCQ (−8.1)
D3-Gym41.9Improves 3/8 settings, hurts BixBench Open (−8.2)
PEARL (2k)46.4Improves 7/8 settings (+0.7 to +5.2)

Solvability by Difficulty (Table 3)

DifficultySuccess RateAvg. Steps
D194.0%6.2
D286.0%8.1
D357.0%15.1
D441.0%16.4

Ablation Studies

Repository Diversity (top row of Figure 5): Increasing repository coverage from 20%→100% under fixed trajectory budget consistently improves performance across all benchmarks.

Environment Difficulty (bottom row of Figure 5): No consistent trend—difficulty correlates with solvability but doesn't affect training effectiveness within evaluated range (2.4–3.2).

Task Type Removal (Table 5):

VariantBioCoderMA-BenchBixBench MCQSA-BenchAvg
Full PEARL60.478.369.347.163.8
w/o Reproduction54.5 (−5.9)70.7 (−7.6)63.9 (−5.4)44.1 (−3.0)58.3 (−5.5)
w/o Subflow55.8 (−4.6)73.0 (−5.3)60.5 (−8.8)41.2 (−5.9)57.6 (−6.2)
w/o Completion59.7 (−0.7)71.7 (−6.6)62.0 (−7.3)43.1 (−4.0)59.1 (−4.7)
w/o Correction55.2 (−5.2)72.0 (−6.3)62.0 (−7.3)46.1 (−1.0)58.8 (−5.0)

Removing any single task type degrades performance on every benchmark, confirming non-redundant contributions.


Theoretical and Practical Implications

Theoretical Contributions

  1. Workflow Decomposition Principle: Demonstrates that scientific workflows contain multiple complementary supervision signals beyond end-to-end execution, enabling more efficient exploitation of existing code repositories.

  2. Recursive Improvement Framework: The generator–validator feedback loop (recovering 49.4% of initially failed candidates) provides a generalizable mechanism for scaling verifiable data synthesis.

  3. Diversity over Difficulty: Ablations show that repository diversity drives training effectiveness, while environment difficulty alone doesn't—informing future data collection strategies.

Practical Implications

  1. Data Efficiency: PEARL's diverse composition yields gains even with <10% of full data, highlighting the importance of breadth over volume.

  2. Model Democratization: Compact open models (9B–27B) can approach or exceed frontier model performance on specific benchmarks after PEARL training.

  3. Reproducibility Infrastructure: Each environment includes task description, input data, executable sandbox, reference solution, and automatic evaluation script—enabling standardized agent training and evaluation.

  4. Disciplinary Breadth: Coverage of 10 scientific disciplines (with intentional oversampling of biology) supports transfer across domains.


Conclusion

POSEIDON introduces an automated pipeline that scales training environment generation from scientific code repositories by:

  • Deriving four complementary task types from executable workflows
  • Implementing recursive improvement to recover flawed but fixable candidates

PEARL, the resulting dataset of 97,659 environments from 7,110 repositories across 10 disciplines, represents the largest multidisciplinary collection of scientific-agent training environments. Supervised fine-tuning on PEARL consistently improves Qwen3.5 models across six benchmarks, with ablations confirming that both repository diversity and complementary task types contribute to gains.

Future directions: The scale and diversity of PEARL open opportunities for training more capable scientific discovery agents, potentially extending to reinforcement learning paradigms and broader scientific domains beyond the current ten disciplines.

Related papers