# One Run Is Not an Idea: The Implementation Lottery in Automated Research

> Automated research systems misattribute single-implementation scores to research ideas, with 26-44% of winner decisions reversing across implementations, making multi-implementation audits essential.

- **Source:** [arXiv](https://arxiv.org/abs/2607.26587)
- **Published:** 2026-08-15
- **Permalink:** https://picx.dev/p/f5hlW9
- **Whiteboard:** https://picx.dev/p/f5hlW9/image

## Summary

## Summary (Overview)

- **Core phenomenon identified**: Automated research systems conflate single-run scores of one implementation with evidence about the parent research idea, creating an "implementation lottery" where idea-level conclusions depend on which plausible implementation was sampled.
- **Key quantitative findings**: Across 312 assignments on 13 tabular tasks, implementation variance exceeded same-artifact rerun variance by more than 5× (Bounded setup) and 10× (Agentic setup). Leave-one-implementation-out (LOO) winner reversal occurred in 25.6% and 43.6% of decisions respectively.
- **Novel measurement framework**: The paper introduces the Idea Reliability Audit, which formally separates idea-level quality Q from best-of-N artifact utility $B_N$, and reports idea ICC (intraclass correlation coefficient) as the share of within-task variance attributable to ideas.
- **Robustness across conditions**: The implementation lottery persists across two different coding-agent setups, survives outcome-blind card-level filtering, and appears in an exploratory materials-science diagnostic with deterministic evaluation.
- **Practical implication**: Before a score guides idea-level branching, transfer, or research memory, evidence should cover multiple implementations—search delivers artifacts, but the audit supports cumulative science.

## Introduction and Theoretical Foundation

The paper addresses a fundamental unit-mismatch problem in automated research systems. Systems like The AI Scientist, Dolphin, and Agent Laboratory (Lu et al. 2024; Yuan et al. 2025; Schmidgall et al. 2025) automate the loop of proposing, implementing, testing, and selecting research directions. While exploring several realizations and retaining the best is appropriate when the deliverable is the artifact itself, the same loop also uses results to decide what research to pursue—and here a critical distinction emerges.

**The core problem**: A research idea (e.g., "early stopping for a booster") fixes a mechanism but leaves reasonable implementation choices open (validation fraction, patience, monitored metric, library interface). One run scores one implementation of that idea. Crediting that realization-level score as evidence about the parent mechanism creates the *implementation lottery*.

**Illustrative example**: Three valid, blind-faithful implementations of one frozen card span +2.31% to −1.85% relative utility, while all same-artifact reruns are exact. Against the same competitor, every one-draw winner reverses under the other-two mean.

The paper formalizes this using reliability and generalizability theory (Shrout and Fleiss 1979; Brennan 2001). The measured object is a research idea, and the repeated measurements are its implementations. The key distinction is between:

- **Operational idea quality Q**: The average ITT utility over independent implementation attempts
- **Best-of-N artifact utility $B_N$**: What a budgeted search can return

Both are legitimate estimands but answer different questions. Crediting a realization-level score Z as evidence about Q creates the implementation lottery.

## Methodology

### Mathematical Framework

For a fixed setup $e$, observed utility is decomposed as:

$$
Y_{siejk} = \mu_e + A_{se} + I_{sie} + M_{siej} + R_{siejk}, \tag{1}
$$

where $A$ is the task contribution, $I$ is the idea contribution within a task, $M$ is the implementation contribution within an idea, and $R$ is variation observed when the same implementation is rerun.

Operational idea quality is:

$$
Q(s,i,e) = \mathbb{E}_{j,k}[Y_{siejk}]. \tag{2}
$$

The **idea ICC** measures the share of within-task ITT variation that separates ideas:

$$
\mathrm{ICC}_{\mathrm{idea}} = \frac{\sigma_I^2}{\sigma_I^2 + \sigma_M^2 + \sigma_R^2}. \tag{3}
$$

**LOO winner reversal**: Three one-draw worlds are formed, each containing one implementation per idea. For each alignment $h \in \{1,2,3\}$:

$$
\hat{i}_{s,e,h} = \arg \max_i Z_{sieh}, \tag{4}
$$

compared with $\arg\max_i \bar{Z}_{sie,-h}$ (the other-two mean).

### Experimental Design

**Tabular tasks**: 13 public OpenML classification tasks (522–6,430 examples, 4–57 features, binary to ten-class). Four accepted cards per task, three implementations per card, two coding setups → 312 assignments.

**Implementation setups**:
- **Bounded**: ≤4 model calls, limited code inspections, 2 repairs, fixed token allowance
- **Agentic**: One fresh Read/Edit SDK session, `max_turns=20`, no external repair pass

**Key audit components**:
1. **Mechanism-level idea cards**: Fix the scientific intervention while leaving implementation choices open; exclude recipes (exact constructors, hyperparameters, seeds)
2. **Independent implementations**: Three draws per card in fresh sessions with outcome-blind fidelity review
3. **Same-artifact reruns**: Byte-identical code replayed with three frozen seeds to separate implementation choice from rerun noise
4. **ITT retention**: Failed and drifting runs remain in analysis (penalties: −.10 for failure/invalidity, −.02 for drift)

## Empirical Validation / Results

### Implementation Variance Exceeds Rerun Noise

| Metric | Bounded | Agentic |
|--------|---------|---------|
| Implementation variance share | 33% | 45% |
| Rerun variance share | 6% | 4% |
| Implementation σ | .0202 | .0295 |
| Rerun σ | .00871 | .00917 |
| Idea ICC | .612 [.477, .745] | .511 [.248, .747] |
| LOO reversal | 25.6% [7.7%, 46.2%] | 43.6% [20.5%, 66.7%] |

Implementation variance is more than 5× rerun variance (Bounded) and more than 10× (Agentic).

### Robustness Across Processes

- Phase 1 ICC: .661 (Bounded) vs .260 (Agentic), 22.2% vs 66.7% reversal
- Fresh extension reverses ordering: .588 vs .711 ICC, 28.6% vs 23.8% reversal (paired $p = 1.00$)
- Pooled Agentic-minus-Bounded intervals include zero for ICC (−.101 [−.337, .173]) and reversal (+17.9% [0.0%, 38.5%], $p = .1875$)

### Card Filtering Sensitivity

**Table 1: LOO winner reversal (%) before and after outcome-blind card filtering**

| Candidate set | Cards | Bounded | Agentic |
|---------------|-------|---------|---------|
| All frozen | 52 | 25.6 | 43.6 |
| Conservative rule | 38 | 43.6 | 35.9 |
| Construct rule | 43 | 33.3 | 38.5 |

Reversal persists at 33.3–43.6% with task-cluster intervals spanning 12.8–66.7%.

### Materials Diagnostic (Exploratory)

- Winner reversal: 33.3% (Bounded), 55.6% (Agentic)
- Frozen-card ICC: .290 [.000, .824] and .177 [.040, .777]
- Implementation shares: 71% and 82%; rerun variance = 0 (deterministic evaluator)
- Portfolio validity failures exposed (baseline-equivalent, bundled, duplicate candidates)

### Execution Validity

- 147/156 valid assignments per setup; zero code-invalid patches
- Fidelity: faithful/partial for 150/156 (Bounded), 152/156 (Agentic)
- Reviewer agreement on double-labeled subset: 47/49 and 43/49
- Constructed control detection: 50–52/52 drifts, 52/52 invalids

## Theoretical and Practical Implications

### Distinguishing Idea Reliability from Artifact Utility

The paper's central theoretical contribution is separating two estimands:
- **$B_N(s,i,e) = \mathbb{E}[\max_{j \leq N} Z_{siej}]$**: What a budgeted search can return (artifact utility)
- **$Q(s,i,e)$**: How an idea performs across realizations (idea quality)

Best-of-N search targets an upper tail and can favor ideas with greater implementation variance or search effort. This is useful for delivery but consequential when the score is credited to the idea and steers the next hypothesis.

### Practical Guidance for Automated Research Systems

The paper provides a concrete checklist for the Idea Reliability Audit:

> **Idea Reliability Audit checklist**
> *Unit*: mechanism, invariants/open choices, portfolio gate.
> *Process*: executor/model, budget, independence, repeats.
> *Adherence*: blind validity/fidelity; failures retained in ITT.
> *Noise*: byte-identical reruns and evaluator policy.
> *Evidence*: idea ICC/uncertainty, LOO reversal, sensitivities.

**Promotion policies**: Before a score prunes a branch, justifies transfer, enters research memory, or supports a mechanism-level claim, the candidate must denote a coherent intervention whose ranking survives another realization. The three-draw design triples executor assignments, but rerun calibration and fidelity review use only evaluator calls.

**Diagnostic guidance**: Large implementation variation → request new implementations; rerun variation → seek more stable evaluator; low fidelity → tighten control.

## Conclusion

One run is not an idea. The paper demonstrates that automated research systems face a structural attribution problem: realization-level scores cannot serve as idea-level evidence without multiple implementations. Key takeaways:

1. **The implementation lottery is real and substantial**: 25.6–43.6% of one-draw winner decisions reverse under other implementations, with implementation variance dominating rerun noise by 5–10×.

2. **The phenomenon is structural, not incidental**: It persists across implementation processes, survives outcome-blind card filtering, and appears in deterministic-evaluation materials workflows.

3. **The measurement framework is generalizable**: Idea ICC and LOO reversal provide concrete tools for any automated research system to audit its own evidence quality.

4. **Future directions**: The paper calls for measuring reversal rates in larger deep-learning workflows (which add choices in data, architecture, optimization, checkpoints, and systems), evaluating best-of-N stability, end-to-end tree search, and downstream audit gains. The attribution problem grows with workflow complexity.

The measured rates vary by setting, but the attribution problem is structural whenever a realization updates beliefs about an idea. **Search delivers artifacts; the audit supports cumulative science.**

---

_Markdown view of https://picx.dev/p/f5hlW9, served by PicX — AI-generated visual whiteboard summaries of research papers._