# Rollout Cards: A Reproducibility Standard for Agent Research

> Rollout cards—preserving full interaction records with explicit reporting rules—make agent evaluation scores reproducible, revealing that reporting choices alone can flip model rankings by up to 20.9 points.

- **Source:** [arXiv](https://arxiv.org/abs/2605.12131)
- **Published:** 2026-08-22
- **Permalink:** https://picx.dev/p/hdg20S
- **Whiteboard:** https://picx.dev/p/hdg20S/image

## Summary

# Rollout Cards: A Reproducibility Standard for Agent Research

## Summary (Overview)

- **Core contribution**: Introduces "rollout cards" as a new publication standard for agent research, treating complete rollout records (not just reported scores) as the fundamental unit of reproducibility.

- **Empirical audit findings**: A structured audit of 50 popular training and evaluation repositories found that **none** report how many runs failed, errored, or were skipped alongside headline scores, and identified **37 reporting-rule discrepancies** that can change task-success rates, cost/token accounting, or timing measurements.

- **Key experimental results**: Re-grading preserved benchmark outputs shows that changing only the reporting rule can change reported scores by up to **20.9 absolute percentage points** and can **invert rankings of frontier models** (e.g., GPT-4o vs. Claude 3.5 Sonnet on τ-bench).

- **Reanalysis findings**: Four partial public releases (GAP, MAESTRO, COPRA miniF2F, Tree-of-Thought) yielded analyses their original reports did not include—e.g., **20.6% of text-safe GAP responses made forbidden tool calls**, and longer proof search in COPRA correlates *negatively* with success.

- **Practical deliverable**: A reference implementation integrated into ERGON (open-source RL gym) with 21 publicly released rollout-card exports spanning tool use, software engineering, web interaction, multi-agent coordination, safety, and search.

## Introduction and Theoretical Foundation

The paper addresses a reproducibility crisis emerging in agent research, building on historical precedents in machine learning and reinforcement learning. The authors cite prior work showing that identical LLaMA-65B weights score 63.7 under Berkeley evaluation code but only 48.8 under lm-evaluation-harness on MMLU—a 14.9-point gap attributable solely to evaluation-code choices about prompt templates and answer extraction.

The theoretical foundation rests on three key definitions:

- **Rollout**: The record of one episode of agent–environment interaction, including environment configuration, observations, actions (model outputs and tool calls), resulting environment states, per-step timing, and terminal status.

- **Rollout batch**: The collection of episodes used to compute a reported score.

- **View**: The projection of each rollout onto the fields a particular analysis reads.

- **Reporting rule**: The procedure that turns a viewed rollout batch into a reported score (e.g., graders, failure-handling heuristics, per-episode score combination rules).

The paper identifies two failure modes:

1. **Recording failure**: Rollouts are scored once and discarded, preventing later re-analysis or alternative views.
2. **Reporting failure**: Reporting rules vary without disclosure, making it impossible to distinguish score gaps caused by agent behavior from those caused by evaluation conventions.

The authors argue that backfilling missing rollouts is impractical because LLM-agent scaffolds couple models, prompts, tool layers, retry logic, and environments, creating implementation sensitivity, and because frontier evaluation costs have risen 3–18× per year as task horizons double every seven months.

## Methodology

### Audit of Repositories

The authors audited 50 popular repositories across agent harnesses, evaluation libraries, benchmarks, and RL stacks at pinned commits. They inspected how each handles rule-sensitive events: generation failures, reward/judge failures, unparseable outputs, environment failures, mid-run termination, denominator choices, and training-data filters.

### Rollout Card Specification

The rollout card is a publication bundle containing:

1. **Rollout record**: A self-describing archive with run metadata, agent/workder nodes, dependency edges, task/environment state, actions, artefacts, and environment records—readable without the original runtime.

2. **Reporting-rule registry**: Names every view and reporting rule applied to the card, with implementation, configuration, or version.

3. **Drops manifest**: A typed record of which streams, fields, and rows the analysis read; which filters/exclusions it applied; which structures it collapsed; and which semantic loss classes it declares.

4. **Release-scope metadata**: Declares redaction, access, licensing, or redistribution limits (full trace, redacted trace, gated trace, derived view, or metadata-only).

### Experimental Design

**RQ1** (recording): Test whether public releases preserve enough rollout evidence for another community to compute a different view without rerunning. Four cases were analyzed:

- **GAP** (tool safety): Text-safety labels vs. tool-call safety
- **MAESTRO** (multi-agent): Run outcome vs. coordination overhead
- **COPRA miniF2F** (theorem proving): Proof outcome vs. realized proof-search cost
- **Tree-of-Thought** (search): Final reward vs. action-budget and return profile

**RQ2** (reporting): Hold rollout evidence fixed and change only the reporting rule, measuring impact on scores and conclusions across HumanEval, GPQA, BrowseComp, SWE-bench Verified, τ-bench, and MLE-Bench.

## Empirical Validation / Results

### Audit Results

**None of the 50 audited repositories** report failed, errored, or skipped rollouts alongside headline accuracy. Specifically:
- 11 scored entries silently absorb catastrophic failures into reported scores
- 31 drop failures with no visible counter
- 9 log failures somewhere but do not surface them beside the reported score

### Table 1: Flagship Reporting-Rule Discrepancies

| Family | Setup | Convention A | Convention B | Δ |
|--------|-------|--------------|--------------|---|
| Task success | LLaMA-65B MMLU 5-shot | Berkeley (63.7) | lm-eval-harness (48.8) | 14.9pp |
| Task success | Mistral/Mixtral MMLU | best prompt template | worst prompt template | up to 24.6pp |
| Cost/tokens | Anthropic cached API call | OpenTelemetry input-inclusive | Anthropic separated | 2.0× |
| Cost/tokens | Sonnet 3.5 on Aider | Edit benchmark ($0) | Refactor+Polyglot ($14.41) | $14.41 |
| Latency/timing | SWE-bench Verified | Docker backend | Modal backend | "systematic" |
| Latency/timing | GSM8K-GRPO epoch | optimized TRL | OpenRLHF | 3.1× |

### RQ1 Results: Preserved Rollouts Support Cross-Community Reanalysis

**GAP (tool safety)**: Of 4,855 text-safe samples, **1,002 (20.64%) contained unsafe tool calls**. Of those 1,002, 997 visibly refuse the harmful request in text—so the unsafe behavior appears *only* in the tool channel. A response-only safety score certifies visible refusal while missing forbidden actions.

**MAESTRO (multi-agent)**: Failures involve substantially more coordination work than successes:
- Failed runs: median 48 spans, 78,523 tokens
- Successful runs: median 10 spans, 11,586 tokens

The median failed run uses roughly **5× more spans and 7× more tokens** than the median successful run, contradicting test-time-scaling interpretations.

**COPRA miniF2F (theorem proving)**: Among 908 theorem-result records, all 44 one-step attempts succeed, but only 74 of 864 longer attempts do. Success falls from 27.4% at 2–3 steps to 7.6% at 4–7 steps. A logistic model finds each doubling of realized steps is associated with lower odds of success (odds ratio 0.62, $p < 10^{-6}$).

**Tree-of-Thought (search)**: Pruning raises mean final word reward from 0.320 to 0.440 while reducing mean unique atomic actions from 48.65 to 28.65. In five pairs, both runs finish with the same reward despite large differences in action coverage, depth, and backtracking.

### RQ2 Results: Reporting Rules Can Change Scientific Conclusions

### Table 2: Reporting-Rule Effects on Fixed Public Artefacts

| Benchmark | Held fixed | Reporting choice | Score gap | Consequence |
|-----------|-----------|------------------|-----------|-------------|
| HumanEval/GPQA | Generated outputs | Driver/extraction variants | 0.6pp/1.0pp | Minor changes: independent rules mostly agree |
| BrowseComp | Gold/predicted answers | Rule-based vs LLM judge | 4.17pp | Judge convention changes 274 answer labels |
| SWE-bench Verified | GPT-4o submissions | Missing patches counted as failures vs excluded | 2.3pp of 15.6pp gap | Affects reported capability gap |
| τ-bench | Tool-call trajectories | DB-state vs sequence/set graders | 16.9pp | 12 changes in ordering of frontier models |
| MLE-Bench | Kaggle submissions | Medal/pass definition | 20.9pp | Gold-only reporting cuts pass rate from 34.2% to 13.3% |

On SWE-bench Verified, both submissions use GPT-4o-2024-05-13, but no-submission rates differ (50/500 for SWE-agent vs. 4/500 for Agentless). Counting no-submissions as unresolved gives a 15.6pp gap; scoring only submitted patches gives 13.3pp—so the reporting rule accounts for 2.3pp of the reported gap.

## Theoretical and Practical Implications

### Theoretical Implications

The paper establishes that **reported scores are not a sufficient statistic for agent evaluation**. Two scores can differ due to three separable factors: different rollout batches, different views of the same batch, or different reporting rules over the same view. Without preserving the rollout record and declaring the reporting rule, these cannot be distinguished.

The finding that reporting-rule effects **grow with evaluation complexity** has significant implications: as evaluations move toward longer-horizon tasks and richer environments, harnesses make more local choices about what to count, drop, combine, or treat as success, increasing the risk of reporting-rule score gaps.

### Practical Implications

- **For researchers**: Rollout cards enable post-publication re-analysis without rerunning expensive frontier evaluations (which cost 3–18× more per year as horizons grow).

- **For benchmark maintainers**: The drops manifest makes reporting rules inspectable, allowing disagreements to be traced to preserved evidence, reporting choices, or agent behavior.

- **For the community**: Shared rollout-card repositories would support reuse, reporting-rule comparison, and meta-analyses without new frontier rollout budgets.

- **For safety research**: The GAP finding—20.6% of text-safe responses making forbidden tool calls—demonstrates that safety evaluations must inspect tool channels, not just visible responses.

## Conclusion

The paper's main takeaways are:

1. **The problem is prevalent**: Across 50 popular repositories and 37 reporting-rule discrepancies, failures and reporting choices are routinely hidden behind headline scores.

2. **The solution is feasible**: Rollout cards preserve the rollout record behind a reported score and pair it with explicit views, reporting rules, and drops manifests.

3. **The evidence supports the approach**: Preserved traces reveal safety, coordination, proof-search, and search-efficiency findings that original reports did not include, while fixed benchmark artefacts can be re-graded to change reported scores and flip model orderings.

**Limitations**: Rollout cards do not prevent selective metric choice, privacy constraints, redaction, or selective reporting. Their role is narrower: to make the record, rule, and omissions inspectable so disagreements can be traced to preserved evidence, reporting choices, or agent behavior.

**Future directions**: At scale, shared rollout-card repositories would support reuse, reporting-rule comparison, and meta-analyses without new frontier rollout budgets. The authors frame rollout cards as an extension of the Datasheets and Model Cards lineage, positioning them as the prerequisite record layer that makes existing reporting-rule standardization efforts (HELM, OLMES, Unitxt) replayable after publication.

---

_Markdown view of https://picx.dev/p/hdg20S, served by PicX — AI-generated visual whiteboard summaries of research papers._