# Sci$Π$vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

> Sci Pi vot shows frontier LLMs fail at scientific judgment, scoring below chance on choosing human-corrected plans, and more context or reasoning often makes them worse.

- **Source:** [arXiv](https://arxiv.org/abs/2610.21082)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/wNq1Yt
- **Whiteboard:** https://picx.dev/p/wNq1Yt/image

## Summary

## Summary (Overview)

- **Sci Π vot** is a 248-case benchmark evaluating Principal Investigator (PI) Judgment—the scientific planning and decision-making ability—of LLM agents in real frontier research contexts.
- The benchmark is built from **six months of research** where senior researchers ran LLM agents on **26 projects across six scientific fields** and repeatedly intervened to correct agents' research judgments.
- Each case pairs an **original LLM plan (A)** with a **human-corrected plan (B)**, and evaluators must choose between them using only the project record available before the correction.
- **Key finding**: At medium reasoning effort with the project record, **four of five frontier models perform below chance (50%)**, with the best model achieving 63.5% accuracy compared to 73.4% for fresh human researchers.
- **Counterintuitive result**: Every model performs *worse* with the project record than without it at medium effort, and further deliberation ("re-trapping") can lead models back to the same mistakes they had previously corrected.

---

## Introduction and Theoretical Foundation

### Motivation

The paper addresses a critical bottleneck in autonomous scientific research: **scientific judgment**. While LLM agents can keep experiments running for months with minimal supervision, they struggle with the *open-ended decisions* that determine whether research actually advances:

- Formulating useful questions
- Designing informative experiments
- Revising assumptions as evidence arrives
- Diagnosing why experiments fail

The authors frame this as **"PI Judgment"**—the scientific planning and decision-making typically performed by a Principal Investigator. The core question is: *Can frontier models conduct high-quality scientific research autonomously?*

### Three Joint Requirements for AutoResearch

1. **Autonomy**: Work around the clock with minimal human intervention
2. **Scale and scope**: Run many projects in parallel across disciplines
3. **Scientific quality**: Pursue original, PhD-level frontier research worthy of publication

### Why Rules and Checks Are Insufficient

The paper argues that adding more rules and automated checks has diminishing returns:
- A growing set of instructions becomes harder for models to follow reliably
- Extra instructions consume tokens and can impair task performance
- Rules leave scientific questions unresolved—an instruction to "check a measurement" doesn't specify *which assumption is wrong* or *what evidence would expose it*

### Two Illustrative Examples of Human Judgment

**Example A (Transformer project)**: A small model scores zero on LiveCodeBench groups. The LLM agent proposes scaling to a larger model; the senior researcher instead keeps the small model and checks the scorer. A later audit reveals the scorer skipped answer extraction—the zeros measured the grader, not the model.

**Example B (Cloth simulation)**: Four of six parameters were fixed at zero in every run, and the best fits of the other two hit search limits. The agent proposes refining the two-parameter search; the researcher points out that missing fitted values don't justify setting parameters to zero and calls for fitting all six parameters.

> *"Both workflows could keep running without testing the mistaken assumptions that the corrections address."*

---

## Methodology

### Benchmark Construction

The construction process is retrospective and grounded in real project histories:

1. **Source data**: Six months of research records from 26 projects where senior researchers ran LLM agents and intervened to correct judgments
2. **Candidate decisions**: Derived from human messages and changes to the project's human-instruction section
3. **Case recovery**: For each case, the writer recovers the original LLM plan (A) and reconstructs the human-corrected plan (B) from the intervention and subsequent investigation
4. **Evaluation setup**: Evaluators receive the pre-intervention project record and both plans—without correctness labels or later history—and must choose which plan to pursue

### Case Selection

- 248 cases selected from 26 projects using **120 model responses per case** (five models × twelve conditions × two option orders)
- Cases retained only if model responses *vary* across the two plans (model-in-the-loop selection, following challenge-set precedent)
- Four one-sided cases kept only after verification against documentation

### Judgment Categories

| Category | Cases | What the agent does wrong |
|----------|-------|---------------------------|
| **Planning** | 123 | Starts costly runs before basic checks, or runs experiments too small to answer the question |
| **Adapting** | 65 | Keeps the same hypothesis even when experiments show what should change |
| **Anomaly awareness** | 88 | Overlooks suspicious results that should trigger checks for bugs or confounding factors |
| **Evidence sufficiency** | 47 | Declares success/failure before experiments justify the conclusion |
| **Repetition** | 4 | Repeats an experiment whose answer is already available |
| **Focus** | 122 | Tests a different claim or optimizes a proxy, leaving the actual research question unanswered |

### Evaluation Conditions

- **Project condition**: Model receives two plans + full project record (research idea, proposal, progress notes, code, experimental logs)
- **Options-only condition**: Same plans without project record
- **Seven effort settings**: low, medium, high, xhigh, max, plus two review continuations at medium effort
- **Two option orders** per case to control for position bias

### Models Evaluated

- GPT-6 Astra, Claude Opus 5, Grok 4.6, GPT-5.6 Sol, and GPT-5.6 Terra

### Human Baseline

- **13 PhD students and postdocs** with no prior exposure to the projects
- Used an interface for reading the project record with an agent assistant available
- Achieved **73.4% accuracy**

---

## Empirical Validation / Results

### Main Results (Table 4)

| Model | Project (%) | Options only (%) | Best observed project (%) | Setting |
|-------|-------------|------------------|---------------------------|---------|
| Astra | 59.5 | 63.7 | 63.5 | xhigh |
| Opus | 38.9 | 58.9 | 46.8 | max |
| Grok | 33.1 | 54.0 | 38.1 | high |
| Sol | 35.1 | 46.8 | 37.7 | medium, after review 2 |
| Terra | 29.2 | 55.8 | 52.0 | medium, after review 1 |
| **Human baseline** | **73.4** | — | — | agent assistant available |

### Key Findings

**1. Models systematically prefer the original LLM plan**
- At medium effort with the project record, four of five models choose the rejected plan more often than the human-corrected reference plan
- Five-model average: **39.2%** (well below chance)
- On 49 cases, *all five models* choose the rejected plan in both orders; only on 16 cases do they all choose the reference plan

**2. The project record hurts performance**

The accuracy change with project context relative to options-only is **negative for all five models**:

$$ \text{Accuracy change} = \text{Accuracy}_{\text{project}} - \text{Accuracy}_{\text{options-only}} $$

Mean accuracy: 39.2% with project record vs. **55.8% without it**. For four models, the paired 95% interval of this decrease excludes zero.

**3. More thinking can lead back to the same mistake ("re-trapping")**

- Across five models, the first review repairs 422 errors but introduces 302 new ones
- The second review repairs 264 but introduces 305—partially undoing the earlier gain
- Mean accuracy across stages: **39.2% → 44.0% → 42.3%**
- In the Transformer case, Opus first chooses the larger model, then corrects to the reference plan, then *reverts* to the original mistake on the second review

**4. Field and category variation (Figure 3)**

- Astra reaches **72.2%** on anomaly-awareness cases but only **56.1%** on planning and **54.9%** on focus cases
- Performance varies by scientific field, with models generally weakest on focus-related judgments

---

## Theoretical and Practical Implications

### For AI Research Evaluation

- Sci Π vot fills a gap in scientific research benchmarks: it tests **judgment** (choosing between plans) rather than execution (replicating experiments or optimizing defined objectives)
- Unlike benchmarks with objective rubrics (PaperBench, RE-Bench, MLE-bench), Sci Π vot's reference plans are grounded in **senior researchers' corrections and experimental evidence** from real frontier research

### For NLP Training

- The case pairs provide **preference supervision for scientific corrections** rather than stylistic rewrites
- Each case matches the training format of **Direct Preference Optimization (DPO)** (Rafailov et al., 2023):

$$ \mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w | x)}{\pi_{\text{ref}}(y_w | x)} - \beta \log \frac{\pi_\theta(y_l | x)}{\pi_{\text{ref}}(y_l | x)} \right) \right] $$

- A preference scorer trained on these comparisons could supply **rewards for research actions in agentic reinforcement learning**

### For AutoResearch Systems

- The benchmark identifies **recurring failure modes** (planning, adapting, anomaly awareness, evidence sufficiency, focus) that agentic systems must address
- The "re-trapping" phenomenon suggests that self-correction without external feedback can be unreliable—consistent with prior work showing self-correction can reduce accuracy

### Comparison with Existing Benchmarks

Sci Π vot is distinct in that it evaluates **scientific planning and decision-making** using:
- Real frontier scientific research (not synthetic tasks)
- Experimental evidence as the basis for correctness
- Senior researchers' corrections as reference standards

---

## Conclusion

### Main Takeaways

1. **Scientific judgment is the bottleneck** for autonomous research—not execution capability
2. **Frontier models struggle to identify human-corrected decisions**, even when given the full project record and both plans
3. **More context and more reasoning don't reliably help**—the project record can mislead, and further deliberation can undo correct choices
4. **The benchmark provides a foundation for training**: the case pairs are directly usable for preference optimization

### Future Directions

- Train a preference scorer on Sci Π vot comparisons to supply rewards for agentic reinforcement learning
- Test whether such training improves decisions in **new projects** and reduces human intervention
- Extend evaluation to wet-lab execution and specialized scientific software (COMSOL, etc.)
- Explore how these judgments transfer across different tools and experimental procedures

### Limitations

- Covers theoretical and computational research (simulation, data analysis, biological imaging) but **not wet-lab execution**
- The selected cases measure *recognition* of corrective decisions, not how often failures occur in general
- Effort settings are not matched token budgets; the experiments test available controls, not controlled compute scaling or verifier-guided search
- Several cases come from each project, so project-cluster intervals are reported alongside case-level intervals

---

> *"Sustained scientific research exposed decisions that frontier models and agentic systems still left to senior researchers. Sci Π vot turns these human corrections into tests of PI Judgment... Even with the project record and further deliberation, models often return to the decisions senior researchers corrected."*

The paper's central contribution is demonstrating that **scientific judgment—the ability to choose which experiment advances the research question—remains a critical weakness** of current LLM agents, and providing a benchmark to measure and eventually train this capability.

---

_Markdown view of https://picx.dev/p/wNq1Yt, served by PicX — AI-generated visual whiteboard summaries of research papers._