Scientist World: Grow a Research World for AI-Scientists

Summary (Overview)

• Scientist World converts research papers into executable environments where AI-Scientists can be evaluated on their ability to improve existing methods or innovate new ones from scratch.
• The system uses a fully autonomous multi-agent pipeline (no human in the loop) that reproduces a paper's method, designs a research task, packages it for portability, and verifies quality—releasing the first 1,000 verified environments.
• Frontier AI-Scientists surpass human researchers' methods 74% of the time in innovate mode and 84% in improve mode, as measured by the new Frontier-Normalized Improvement (FNI) metric.
• The project releases a corpus of 6,400 hours of research traces (dataset) and demonstrates that a small model (Qwen3-8B) can be trained on the environments to raise its research ideation by 158%, matching frontier models' performance.
• Scientist World serves as a benchmark, dataset, and training gym simultaneously—scalable, evolving with new literature, and grounded in real execution rather than rhetorical quality.

---

Introduction and Theoretical Foundation

The paper begins with a fundamental observation: AI-Scientists have arrived, but a world for them to grow in did not exist. Human scientists take the scientific literature for granted—centuries of open problems and shared ideas. To AI-Scientists, that literature is readable but not runnable.

What is Research?

The authors argue that research is not a single ability or task:

"Watch a scientist at work: they read until they find something no one has explained, guess at the reason, and build an experiment to test it. It fails. They eventually figure out what the failure is telling them, adjust, and run it again. It works."

What makes a scientist is judgment that carries from one problem to the next—the ability to discover again and again. The scientific literature provides two essential things:

1. Problem: Each paper defines a different research question no single answer can exhaust.
2. Idea: Each paper contributes to the field's shared memory of approaches, findings, successes, and failures.

Three Required Properties

Scientist World aims to hold three properties simultaneously:

Prior environments fail on at least one dimension: they recycle competition problems, confine to simulated worlds, are fixed at release (aging into training data), or judge proposed ideas as text rather than executing them.

---

Methodology

2.1 The Builders: Autonomous Paper-to-Environment Pipeline

Four builder roles convert papers into environments through iterative agent workflows on a shared workspace:

1. Reproducer: Re-implements the paper's method and original evaluation (from authors' code or from scratch), scaling experiments from quick checks to full runs until the paper's result is recovered.

2. Designer: Converts the reproduction into a task—writes the problem statement, fixes rules (which metric to improve, fixed settings, allowed solution types), and prepares visible and held-out evaluation sets.

3. Packager: Makes the environment portable—replaces local files with download scripts, declares dependencies, and verifies a fresh clone can rebuild and rerun.

4. Verifier: Independently audits the packaged environment (kept separate so the builder that grades never built the work).

Infrastructure: Builders submit jobs to a shared GPU pool, sleep, and wake with logs. Each paper has a capped GPU budget (up to 70 GPU-hours). The workflow uses nine iterations (3 reproducers, 3 designers, 1 packager, 2 verifier passes) with model families rotating across GPT, Claude, MiniMax, and GLM.

2.2 The Environment Structure

Each environment repository contains four parts:

1. Task briefing — problem, target, allowed direction, constraints
2. Data and evaluation files — define required artifacts and scoring
3. Reproduced method and baselines — starting solutions (optionally withheld)
4. Construction record — scripts, logs, commits

Design principles:

• Experiments may be shrunk to reduce compute cost without changing the scientific question
• Tasks must remain non-trivial (source method must beat simple baselines)
• The reproduced method is treated as one solution, not the task itself

Fairness rules across four files: problem, target, direction, constraints (which disallow changing the question, direct scale gains, etc.).

Frontier-Normalized Improvement (FNI)

The core evaluation metric is defined as:

$$\mathrm{FNI} = 100 \sum_c w_c \delta_c / (\sigma_c b)$$

where:

• $\delta_c$ = signed gain over the source method on metric $c$
• $w_c$ = weight for metric $c$
• $\sigma_c$ = frozen frontier cohort's spread on metric $c$
• $b$ = scaling factor setting the cohort's mean |FNI| to 100
• 0 matches the paper, negative is worse

Paper Selection

From 124,816 unique papers (arXiv AI categories, 14 months):

1. Fast screens (MiniMax M2.7): quality (top 25%) and type (empirical-method papers) → 21,354 papers
2. Detailed judgments: research quality and reproduction difficulty (preferring low compute)
3. Category-balanced top 1,000 selected via geometric-mean utility

---

Empirical Validation / Results

Build Costs and Quality

Table 1: Build costs for 1,000 papers.

Table 3: Quality and validity metrics.

Benchmark Results

Key finding: AI-Scientists beat the paper's reproduced method in:

• 84% of improve-mode attempts
• 74% of innovate-mode attempts

Leaderboard insights (Figure 12):

• Fable 5: strongest at improving existing methods
• Fable 5.1 and Fable 5: top two at innovating from scratch
• GLM 5.2: best open model, third at innovating

Two distinct abilities: The correlation between improve-rank and innovate-rank is loose (Figure 13)—some models excel at building on source code while others are better at designing from scratch.

Effort doesn't explain performance:

• Compute hours and agent turns explain almost none of the variance in scores ($R^2 \leq 0.02$)

Research quality drives outcomes:

• An LLM judge (blind to scores) rated attempts on creativity, engineering, rigor, significance, and depth
• These ratings track actual scores almost exactly: $\rho = 0.89$
• Winning models do better research, not more effort

Dataset Statistics

Table 4: Per-attempt dataset statistics (~6,400 total hours).

Gym: Training Results

Training Qwen3-8B with Tinker in two stages (SFT then GRPO), separating ideation from implementation (a fixed frontier implementer executes all ideas):

The trained scientist's ideas, faithfully implemented, reach the frontier cohort's anchor—a 158% improvement over the base model.

---

Theoretical and Practical Implications

For AI-Scientist Evaluation

• FNI provides a principled, execution-grounded metric that compares AI performance directly against human researchers' published results
• The separation of improve vs. innovate modes reveals that research ability is not monolithic—different models have different strengths
• Effort-based explanations of performance are rejected; research quality (creativity, rigor, depth) is what matters

For AI Training

• Scientist World demonstrates a complete training loop: environments provide data, rewards, and held-out exams
• The success of training a small model (8B) to match frontier performance shows that ideation can be learned from execution-grounded experience
• Separating ideation from implementation enables scalable training even for models that cannot yet carry the full research loop

For the Scientific Enterprise

• The world grows with the literature rather than with annotation effort—approximately 1,500 new feasible papers per month
• Every paper becomes a potential training ground, creating a self-reinforcing cycle where scientific progress accelerates
• The corpus of research traces provides an unprecedented record of how AI does science, including failures

---

Conclusion

Scientist World transforms the scientific literature from a static record into a living research universe. Its three-fold nature—benchmark, dataset, and gym—enables the full research loop: propose, execute, and update.

Key takeaways:

1. 1,000 papers have become runnable, scored environments, with the collection growing autonomously
2. Frontier AI-Scientists already surpass human results in the majority of attempts
3. The same environments train the next generation of scientists, with demonstrable gains even for small models
4. The world evolves with both models and literature, keeping problems as open as the research they came from

The authors frame this as the beginning: "each is a scaled-down paper and each score one contingent trajectory, but the loop they are built for... is just beginning." The chain reaction of AI-accelerated science now has a place to happen.