Full text not available for this paper
Summary (Overview)
- POSEIDON is an automated pipeline that generates diverse, verifiable training environments for scientific agents from code repositories at scale, achieving 9.3× higher yield than prior approaches (D3-Gym) under matched repository budgets.
- The pipeline decomposes scientific workflows into four complementary task types: Reproduction, Subflow, Completion, and Correction, each targeting distinct scientific-agent capabilities.
- A recursive improvement loop recovers 49.4% of initially failed environment candidates through generator–validator feedback.
- The resulting dataset, PEARL, contains 97,659 executable and verifiable environments from 7,110 repositories across 10 scientific disciplines—the largest multidisciplinary collection of scientific-agent training environments to date.
- Fine-tuning Qwen3.5 models (9B, 27B, 35B-A3B) on PEARL yields consistent gains across six scientific-agent benchmarks, narrowing the gap with frontier models.
Introduction and Theoretical Foundation
Background and Motivation
Scientific research requires agents to execute scientific workflows—coordinated sequences of data processing and computational steps leading to evidence-supported conclusions. Unlike simple question-answering or isolated code generation, these workflows demand integration of:
- Scientific reasoning
- Coding proficiency
- Tool use capabilities
Despite the importance of these capabilities, even frontier models struggle. For instance, GPT-5.6 Sol achieves only 28.7% pass rate on OpenAI's GeneBench-Pro at the highest reasoning level.
The Scaling Challenge
Training environments for scientific agents remain severely limited. D3-Gym pioneered automated synthesis from code repositories but produced only 565 environments from 239 repositories (~2.4 per repository), treating each workflow as a single end-to-end task.
Central Research Question
How can we efficiently exploit executable scientific workflows within code repositories to generate high-quality, diverse training environments at scale?
Theoretical Foundation
The key insight is that a scientific workflow provides richer supervision than a single end-to-end task through structured modifications to its constituent steps. This aligns with prior work demonstrating that transforming existing code artifacts can substantially increase supervision quantity and diversity.
Methodology
Problem Formulation
A SciAgent training environment is formally defined as:
where:
- = task description
- = input dataset
- = executable sandbox
- = evaluation script
An agent policy generates a trajectory with interleaved reasoning () and actions ().
Environment Synthesis Objectives
Four properties are required: (1) Data-Dependent, (2) Executable, (3) Verifiable, (4) Diverse.
Pipeline Stages
Stage 1: Repository Discovery
- Keyword Pool Generation: LLM-expanded domain keywords (e.g., GROMACS, MDAnalysis, PLUMED) filtered by GitHub search hit counts (30–5,000 range)
- Repository Pool Collection: Criteria = ≥20 stars, contains scientific data, backed by published paper, belongs to core scientific discipline
- Executable Sandbox Construction: Statically analyzed dependencies + parsed manifests
- Result: 7,110 repositories (47% biology, 11.8% pharmacology, 10.7% physics, etc.)
Stage 2: Task Synthesis
Seed Program Extraction: DeepSeek-V4-Flash groups related code files into cohesive sets, rewrites into single entry-point scripts, executes to validate.
Four Task Types Derived from Workflow Graphs:
| Task Type | Construction | Targeted Capability |
|---|---|---|
| Reproduction | Full workflow inputs/outputs | End-to-end planning & execution |
| Subflow | Connected subgraph extraction | Local dependency understanding |
| Completion | Remove functional block | Dependency-aware workflow completion |
| Correction | Inject controlled error | Scientific error diagnosis & repair |
Stage 3: Verification
- Multi-stage pipeline assessing environment reliability (complete, executable, reproducible, gradable) and scientific validity (meaningful, correct, consistent)
- GPT-5.5 validator + DeepSeek-V4-Flash generator in recursive improvement loop (up to 3 rounds)
- Difficulty scoring (1–4) based on domain knowledge and analysis depth
- Manual review of 600 sampled tasks: 95.5% acceptance rate
Empirical Validation / Results
Main Results (Table 2)
Training on PEARL consistently improves performance across model scales:
| Model | BioCoder | SciCode Main | MA-Bench | SA-Bench | BixBench MCQ | BixBench Open | BioMystery | Avg |
|---|---|---|---|---|---|---|---|---|
| Qwen3.5-9B | 46.8 → 51.9 (+5.1) | 3.1 → 9.2 (+6.1) | 30.2 → 34.0 (+3.8) | 59.0 → 63.0 (+4.0) | 26.5 → 33.3 (+6.8) | 48.3 → 50.7 (+2.4) | 35.1 → 37.6 (+2.5) | 33.2 → 37.5 (+4.3) |
| Qwen3.5-35B-A3B | 54.5 → 60.4 (+5.9) | 13.8 → 20.0 (+6.2) | 37.5 → 45.1 (+7.6) | 71.3 → 78.3 (+7.0) | 38.2 → 47.1 (+8.9) | 57.1 → 69.3 (+12.2) | 32.2 → 37.1 (+4.9) | 41.6 → 48.3 (+6.7) |
| Qwen3.5-27B | 57.1 → 67.5 (+10.4) | 12.3 → 16.9 (+4.6) | 42.7 → 50.3 (+7.6) | 75.7 → 79.0 (+3.3) | 34.3 → 50.0 (+15.7) | 61.5 → 69.8 (+8.3) | 34.1 → 39.5 (+5.4) | 44.3 → 51.5 (+7.2) |
Key findings:
- Qwen3.5-27B matches GLM-5.2 on BioCoder (67.5%)
- Qwen3.5-35B-A3B matches GPT-5.5 on SciCode main (20.0%)
- 27B model outperforms Hy4-Preview (79.0% vs 69.0% on MedAgentBench)
Data-Scale-Controlled Comparison (Table 4)
With matched budgets (~2,000 trajectories):
| Training Data | Avg | Improvement Pattern |
|---|---|---|
| MedAgentGym | 43.3 | Improves 4/8 settings, hurts BixBench MCQ (−8.1) |
| D3-Gym | 41.9 | Improves 3/8 settings, hurts BixBench Open (−8.2) |
| PEARL (2k) | 46.4 | Improves 7/8 settings (+0.7 to +5.2) |
Solvability by Difficulty (Table 3)
| Difficulty | Success Rate | Avg. Steps |
|---|---|---|
| D1 | 94.0% | 6.2 |
| D2 | 86.0% | 8.1 |
| D3 | 57.0% | 15.1 |
| D4 | 41.0% | 16.4 |
Ablation Studies
Repository Diversity (top row of Figure 5): Increasing repository coverage from 20%→100% under fixed trajectory budget consistently improves performance across all benchmarks.
Environment Difficulty (bottom row of Figure 5): No consistent trend—difficulty correlates with solvability but doesn't affect training effectiveness within evaluated range (2.4–3.2).
Task Type Removal (Table 5):
| Variant | BioCoder | MA-Bench | BixBench MCQ | SA-Bench | Avg |
|---|---|---|---|---|---|
| Full PEARL | 60.4 | 78.3 | 69.3 | 47.1 | 63.8 |
| w/o Reproduction | 54.5 (−5.9) | 70.7 (−7.6) | 63.9 (−5.4) | 44.1 (−3.0) | 58.3 (−5.5) |
| w/o Subflow | 55.8 (−4.6) | 73.0 (−5.3) | 60.5 (−8.8) | 41.2 (−5.9) | 57.6 (−6.2) |
| w/o Completion | 59.7 (−0.7) | 71.7 (−6.6) | 62.0 (−7.3) | 43.1 (−4.0) | 59.1 (−4.7) |
| w/o Correction | 55.2 (−5.2) | 72.0 (−6.3) | 62.0 (−7.3) | 46.1 (−1.0) | 58.8 (−5.0) |
Removing any single task type degrades performance on every benchmark, confirming non-redundant contributions.
Theoretical and Practical Implications
Theoretical Contributions
-
Workflow Decomposition Principle: Demonstrates that scientific workflows contain multiple complementary supervision signals beyond end-to-end execution, enabling more efficient exploitation of existing code repositories.
-
Recursive Improvement Framework: The generator–validator feedback loop (recovering 49.4% of initially failed candidates) provides a generalizable mechanism for scaling verifiable data synthesis.
-
Diversity over Difficulty: Ablations show that repository diversity drives training effectiveness, while environment difficulty alone doesn't—informing future data collection strategies.
Practical Implications
-
Data Efficiency: PEARL's diverse composition yields gains even with <10% of full data, highlighting the importance of breadth over volume.
-
Model Democratization: Compact open models (9B–27B) can approach or exceed frontier model performance on specific benchmarks after PEARL training.
-
Reproducibility Infrastructure: Each environment includes task description, input data, executable sandbox, reference solution, and automatic evaluation script—enabling standardized agent training and evaluation.
-
Disciplinary Breadth: Coverage of 10 scientific disciplines (with intentional oversampling of biology) supports transfer across domains.
Conclusion
POSEIDON introduces an automated pipeline that scales training environment generation from scientific code repositories by:
- Deriving four complementary task types from executable workflows
- Implementing recursive improvement to recover flawed but fixable candidates
PEARL, the resulting dataset of 97,659 environments from 7,110 repositories across 10 disciplines, represents the largest multidisciplinary collection of scientific-agent training environments. Supervised fine-tuning on PEARL consistently improves Qwen3.5 models across six benchmarks, with ablations confirming that both repository diversity and complementary task types contribute to gains.
Future directions: The scale and diversity of PEARL open opportunities for training more capable scientific discovery agents, potentially extending to reinforcement learning paradigms and broader scientific domains beyond the current ten disciplines.
Related papers
- SWE-growth: Teacher-Guided Adaptive Self-Play for Software Engineering Agents
SWE-Growth uses a frozen teacher to guide self-play curricula, improving software agents to 31.7% on SWE-bench Pro and 17.2% on TerminalBench, surpassing student-only training.
- The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search
A naive LLM critic can drive AutoML search below the no-critic random-search floor on 7 of 8 tabular datasets, with losses up to 0.245 AUC.
- BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
BuggyMLE generates 6,600+ bug-injected ML repositories and a structural complexity metric (CCI) predicting solve rates with R²=0.951, enabling difficulty-graded benchmarks where a 9B model rivals frontier agents.