Summary (Overview)

• Sci Π vot is a 248-case benchmark evaluating Principal Investigator (PI) Judgment—the scientific planning and decision-making ability—of LLM agents in real frontier research contexts.
• The benchmark is built from six months of research where senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly intervened to correct agents' research judgments.
• Each case pairs an original LLM plan (A) with a human-corrected plan (B), and evaluators must choose between them using only the project record available before the correction.
• Key finding: At medium reasoning effort with the project record, four of five frontier models perform below chance (50%), with the best model achieving 63.5% accuracy compared to 73.4% for fresh human researchers.
• Counterintuitive result: Every model performs worse with the project record than without it at medium effort, and further deliberation ("re-trapping") can lead models back to the same mistakes they had previously corrected.

---

Introduction and Theoretical Foundation

Motivation

The paper addresses a critical bottleneck in autonomous scientific research: scientific judgment. While LLM agents can keep experiments running for months with minimal supervision, they struggle with the open-ended decisions that determine whether research actually advances:

• Formulating useful questions
• Designing informative experiments
• Revising assumptions as evidence arrives
• Diagnosing why experiments fail

The authors frame this as "PI Judgment"—the scientific planning and decision-making typically performed by a Principal Investigator. The core question is: Can frontier models conduct high-quality scientific research autonomously?

Three Joint Requirements for AutoResearch

1. Autonomy: Work around the clock with minimal human intervention
2. Scale and scope: Run many projects in parallel across disciplines
3. Scientific quality: Pursue original, PhD-level frontier research worthy of publication

Why Rules and Checks Are Insufficient

The paper argues that adding more rules and automated checks has diminishing returns:

• A growing set of instructions becomes harder for models to follow reliably
• Extra instructions consume tokens and can impair task performance
• Rules leave scientific questions unresolved—an instruction to "check a measurement" doesn't specify which assumption is wrong or what evidence would expose it

Two Illustrative Examples of Human Judgment

Example A (Transformer project): A small model scores zero on LiveCodeBench groups. The LLM agent proposes scaling to a larger model; the senior researcher instead keeps the small model and checks the scorer. A later audit reveals the scorer skipped answer extraction—the zeros measured the grader, not the model.

Example B (Cloth simulation): Four of six parameters were fixed at zero in every run, and the best fits of the other two hit search limits. The agent proposes refining the two-parameter search; the researcher points out that missing fitted values don't justify setting parameters to zero and calls for fitting all six parameters.

"Both workflows could keep running without testing the mistaken assumptions that the corrections address."

---

Methodology

Benchmark Construction

The construction process is retrospective and grounded in real project histories:

1. Source data: Six months of research records from 26 projects where senior researchers ran LLM agents and intervened to correct judgments
2. Candidate decisions: Derived from human messages and changes to the project's human-instruction section
3. Case recovery: For each case, the writer recovers the original LLM plan (A) and reconstructs the human-corrected plan (B) from the intervention and subsequent investigation
4. Evaluation setup: Evaluators receive the pre-intervention project record and both plans—without correctness labels or later history—and must choose which plan to pursue

Case Selection

• 248 cases selected from 26 projects using 120 model responses per case (five models × twelve conditions × two option orders)
• Cases retained only if model responses vary across the two plans (model-in-the-loop selection, following challenge-set precedent)
• Four one-sided cases kept only after verification against documentation

Judgment Categories

Evaluation Conditions

• Project condition: Model receives two plans + full project record (research idea, proposal, progress notes, code, experimental logs)
• Options-only condition: Same plans without project record
• Seven effort settings: low, medium, high, xhigh, max, plus two review continuations at medium effort
• Two option orders per case to control for position bias

Models Evaluated

• GPT-6 Astra, Claude Opus 5, Grok 4.6, GPT-5.6 Sol, and GPT-5.6 Terra

Human Baseline

• 13 PhD students and postdocs with no prior exposure to the projects
• Used an interface for reading the project record with an agent assistant available
• Achieved 73.4% accuracy

---

Empirical Validation / Results

Main Results (Table 4)

Key Findings

1. Models systematically prefer the original LLM plan

• At medium effort with the project record, four of five models choose the rejected plan more often than the human-corrected reference plan
• Five-model average: 39.2% (well below chance)
• On 49 cases, all five models choose the rejected plan in both orders; only on 16 cases do they all choose the reference plan

2. The project record hurts performance

The accuracy change with project context relative to options-only is negative for all five models:

$$\text{Accuracy change} = \text{Accuracy}_{\text{project}} - \text{Accuracy}_{\text{options-only}}$$

Mean accuracy: 39.2% with project record vs. 55.8% without it. For four models, the paired 95% interval of this decrease excludes zero.

3. More thinking can lead back to the same mistake ("re-trapping")

• Across five models, the first review repairs 422 errors but introduces 302 new ones
• The second review repairs 264 but introduces 305—partially undoing the earlier gain
• Mean accuracy across stages: 39.2% → 44.0% → 42.3%
• In the Transformer case, Opus first chooses the larger model, then corrects to the reference plan, then reverts to the original mistake on the second review

4. Field and category variation (Figure 3)

• Astra reaches 72.2% on anomaly-awareness cases but only 56.1% on planning and 54.9% on focus cases
• Performance varies by scientific field, with models generally weakest on focus-related judgments

---

Theoretical and Practical Implications

For AI Research Evaluation

• Sci Π vot fills a gap in scientific research benchmarks: it tests judgment (choosing between plans) rather than execution (replicating experiments or optimizing defined objectives)
• Unlike benchmarks with objective rubrics (PaperBench, RE-Bench, MLE-bench), Sci Π vot's reference plans are grounded in senior researchers' corrections and experimental evidence from real frontier research

For NLP Training

• The case pairs provide preference supervision for scientific corrections rather than stylistic rewrites
• Each case matches the training format of Direct Preference Optimization (DPO) (Rafailov et al., 2023):

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w | x)}{\pi_{\text{ref}}(y_w | x)} - \beta \log \frac{\pi_\theta(y_l | x)}{\pi_{\text{ref}}(y_l | x)} \right) \right]$$

• A preference scorer trained on these comparisons could supply rewards for research actions in agentic reinforcement learning

For AutoResearch Systems

• The benchmark identifies recurring failure modes (planning, adapting, anomaly awareness, evidence sufficiency, focus) that agentic systems must address
• The "re-trapping" phenomenon suggests that self-correction without external feedback can be unreliable—consistent with prior work showing self-correction can reduce accuracy

Comparison with Existing Benchmarks

Sci Π vot is distinct in that it evaluates scientific planning and decision-making using:

• Real frontier scientific research (not synthetic tasks)
• Experimental evidence as the basis for correctness
• Senior researchers' corrections as reference standards

---

Conclusion

Main Takeaways

1. Scientific judgment is the bottleneck for autonomous research—not execution capability
2. Frontier models struggle to identify human-corrected decisions, even when given the full project record and both plans
3. More context and more reasoning don't reliably help—the project record can mislead, and further deliberation can undo correct choices
4. The benchmark provides a foundation for training: the case pairs are directly usable for preference optimization

Future Directions

• Train a preference scorer on Sci Π vot comparisons to supply rewards for agentic reinforcement learning
• Test whether such training improves decisions in new projects and reduces human intervention
• Extend evaluation to wet-lab execution and specialized scientific software (COMSOL, etc.)
• Explore how these judgments transfer across different tools and experimental procedures

Limitations

• Covers theoretical and computational research (simulation, data analysis, biological imaging) but not wet-lab execution
• The selected cases measure recognition of corrective decisions, not how often failures occur in general
• Effort settings are not matched token budgets; the experiments test available controls, not controlled compute scaling or verifier-guided search
• Several cases come from each project, so project-cluster intervals are reported alongside case-level intervals

---

"Sustained scientific research exposed decisions that frontier models and agentic systems still left to senior researchers. Sci Π vot turns these human corrections into tests of PI Judgment... Even with the project record and further deliberation, models often return to the decisions senior researchers corrected."

The paper's central contribution is demonstrating that scientific judgment—the ability to choose which experiment advances the research question—remains a critical weakness of current LLM agents, and providing a benchmark to measure and eventually train this capability.