Summary (Overview)

• NatureBench is a cross-discipline benchmark of 90 tasks distilled from Nature-family publications (2022–2025), designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems, using the published SOTA as the scoring anchor.
• The benchmark is built via NatureGym, an automated pipeline that constructs standardized, containerized per-task environments from source papers, addressing the environment-fragmentation problem that has limited prior agent-on-research benchmarks.
• Under a strict web-search-disabled protocol, the strongest agent (Claude Opus 4.7) surpasses SOTA on only 17.8% of tasks ($g > 0.1$) and matches it on 47.8%.
• Success is driven primarily by methodological translation (45.5% of validated successes) — converting scientific tasks into familiar supervised-prediction problems — rather than genuine scientific invention.
• Failures are dominated by wrong method choice (45.1%) and insufficient compute budget (24.4%), not by task misunderstanding.

Introduction and Theoretical Foundation

Background and Motivation

AI coding agents are rapidly moving toward autonomous scientific research — from reproducing published implementations to conducting end-to-end research workflows. However, existing benchmarks for evaluating agent capabilities on scientific research have several limitations:

• Paper-based benchmarks (PaperBench, CORE-Bench, ReplicationBench) measure whether an agent can re-implement a published method, but stop short of asking whether an agent can discover a competitive method on its own.
• Engineering-optimization benchmarks (MLE-bench, PostTrainBench) target Kaggle competitions or post-training tasks, which do not require the domain reasoning, specialized tooling, or cross-discipline knowledge that characterize natural-science research, and suffer from environment fragmentation.

Theoretical Basis

The paper defines a Discovery-oriented evaluation protocol: rather than reproducing a known method, agents must independently solve the same scientific problem, using the source paper's reported SOTA as the scoring anchor to match or surpass.

The authors argue that existing AI-for-Science systems (AlphaFold, GNoME, etc.) share a structural limitation: humans specify the research programme, curate the data, and fix the success criterion, while AI acts as a more capable instrument inside that programme. NatureBench tests the missing horizontal capability: whether contemporary coding agents can solve tasks across six scientific domains using published SOTA as a unified scoring anchor.

Methodology

NatureGym Pipeline

NatureGym converts a published Nature-family paper into a containerized task package through three review-gated stages:

Stage 1: Paper Filtering

• Three-level cascade filter examining: task extractability, evaluation automatability, and data completeness
• Adversarial review to catch false positives

Stage 2: Dataset Acquisition & Verification

• Download data and determine the algorithm boundary — keep inputs to algorithm $A$, drop $A$'s outputs
• Verify decomposability (whether $D_{\text{dev}}$ separates from $D_{\text{eval}}$) and instance validity

Stage 3: Task Package Construction

• Imposes an information firewall that removes the source method from each package
• Builds containerized environments with:
  • Agent-visible: problem/ (README, data description, input data)
  • Hidden from agent: evaluation/ (evaluator, ground truth)
  • Infrastructure: Dockerfile, metadata

The pipeline refines a per-paper record $T = (A, D, M, S, B)$ representing algorithm, dataset, metric, SOTA score, and optional baseline.

SOTA-Normalized Relative Gap

To compare agents across tasks with heterogeneous metrics, each task is scored by a single normalized quantity:

$$g_i = \text{dir}_i \cdot \frac{m_i - m^{\text{sota}}_i}{|m^{\text{sota}}_i|}$$

where $m_i$ is the agent's primary metric value, $m^{\text{sota}}_i$ is the paper-reported SOTA, and $\text{dir}_i \in \{+1, -1\}$ encodes metric direction. $g_i \geq 0$ means the agent matches or surpasses the published result. The task-level score averages $g_i$ across instances, with $g^{\text{fail}}_i = -1.0$ for no valid submission.

Evaluation Protocol

• Agent operates in isolated Docker container with 4-hour wall-clock budget
• Web search disabled
• Iterative submission via three endpoints: /evaluate, /best_score, /time_remaining
• Post-hoc validity judge (Claude Sonnet 4.6) screens for shortcut behaviors
• 3 tasks CPU-only, 70 tasks on RTX 3090/4090, 17 compute-intensive tasks on A800

Benchmark Composition

The final 90 tasks span 6 scientific domains, sourced from 6 Nature-family journals (primarily Nature Machine Intelligence, Nature Methods, and Nature Computational Science). Tasks cover 8 ML task types (prediction/regression 29, classification 19, clustering 14, generation 9, and tail tasks), with 81 distinct primary metrics across 333 evaluation instances.

Empirical Validation / Results

Main Results

Table 4: Main results on NatureBench — sorted by overall Surpass-SOTA ($g > 0.1$) and Match-SOTA ($g \geq 0$), as percentages of tasks.

Score Distribution

Table 5: Gap summary and submission rates

Solution Mechanisms (from 900 runs × 10 agents)

Success modes (among Match-SOTA runs):

• Supervised proxy prediction: 45.5%
• Search/tuning: 17.6%
• Engineering pipeline: 11.0%
• Pretraining/scaling: 8.6%
• Domain-reasoned alternatives: 8.3%
• Method-aligned solutions: 9.0%

Failure modes (among below-SOTA/invalid runs):

• Method-layer failures: 61.1% (wrong method choice 45.1%)
• Execution-layer: 28.7% (insufficient budget/time 24.4%)
• Strategy: 7.0%
• Understanding: 3.1%

Domain Performance

A stable difficulty gradient emerges, shared across agents ($\rho \geq 0.71$):

• Easier tier: Relational Reasoning (60.0% Match-SOTA), Protein Biology (37.5%), Cellular Omics (35.5%)
• Harder tier: Physical Modeling (26.9%), Molecular Design (18.2%), Biomedical Modeling (17.9%)

Cross-discipline tasks (15 of 90) show wider gaps: median $\tilde{g}_{\text{all}}$ drops from −0.13 to −0.21.

Theoretical and Practical Implications

Theoretical Implications

1. Current coding agents excel at methodological translation, not scientific invention. The dominant success pathway is converting scientific tasks into familiar supervised-prediction problems, suggesting agents lack deep scientific reasoning capabilities.
2. Method selection and implementation depth are the primary bottlenecks, not code generation or task understanding. This suggests that future improvements in agent architecture should focus on method selection and resource management rather than language model capabilities alone.
3. Cross-discipline integration remains a distinct challenge, as evidenced by the wider performance gap on interdisciplinary tasks. This aligns with the observation that contemporary scientists face increasingly restrictive information cocoons.

Practical Implications

1. NatureGym provides a reusable pipeline for converting published papers into reproducible, containerized benchmark tasks, addressing the long-standing environment-fragmentation problem in AI-for-Science evaluation.
2. The benchmark establishes a credible Discovery-oriented evaluation protocol that separates genuine algorithmic progress from engineering optimization and shortcut-taking, providing a more rigorous standard for evaluating scientific coding agents.
3. The finding that agents succeed primarily through "supervised proxy prediction" suggests that current agent capabilities are best suited for tasks that can be reformulated as standard ML problems, and poorly suited for tasks requiring novel scientific insight.

Conclusion

NatureBench demonstrates that current frontier coding agents remain far from matching published SOTA on genuine scientific problems from Nature-family papers, with the strongest agent (Claude Opus 4.7) surpassing SOTA on only 17.8% of tasks. The dominant success pathway is methodological translation — converting scientific tasks into familiar supervised-prediction problems — rather than genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding.

The authors release:

• NatureBench: 90 Nature-sourced tasks across six scientific domains
• NatureGym: The automated pipeline for constructing task packages from papers
• A public leaderboard with maintainer-side reproduction

Future directions include turning the same benchmark substrate into training data for future scientific-discovery agents, enabling AI systems to learn directly from the task packages how to discover methods that advance the state of the art across disciplines.