# Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models

> Identical runs of the same AI coding agent vary more than differences between agent-model pairings, so best-of-three with compliance checking beats single-run benchmarking.

- **Source:** [arXiv](https://arxiv.org/abs/2609.33812)
- **Published:** 2026-10-03
- **Permalink:** https://picx.dev/p/4nA4iN
- **Whiteboard:** https://picx.dev/p/4nA4iN/image

## Summary

# Summary of "Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models"

## Summary (Overview)

- **Core finding**: Identical runs of the same AI coding agent on the same task produce substantially different results—the run-to-run variation within a single agent-model pairing (median SD = 0.0107 AUC) exceeds the differences between different pairings (spread = 0.0095 AUC).

- **Compliance violations**: 10 of 312 runs broke task data rules (training on the labeled evaluation file or computing features from the scoring batch), and these violations produced the 7 highest scores in the study, distorting the upper tail of results.

- **Practical recommendation**: A "repeat-and-select" policy—running 3 attempts and keeping the best compliant result—reliably improves artifact quality (+0.0081 AUC median gain) even though a few runs cannot reliably rank different agents.

- **Model scaling**: A planned upgrade to a larger model (GLM-5.3 vs. GLM-5.3 Flash) improved scores by only about one run-to-run standard deviation (0.0091 AUC), with the gain varying significantly by agent.

- **Cost variation**: The same job cost $0.08/run through one agent vs. $1.86/run through another (23-fold difference), driven primarily by prompt cache hit rates that varied by agent-model pairing.

## Introduction and Theoretical Foundation

The paper addresses a practical question for teams deploying AI coding agents: **Which agent? Which model? And how many attempts before trusting the result?**

### Background Context
- Existing benchmarks (SWE-bench, MLE-bench) typically report pass rates averaged across many tasks, often with only 1–3 attempts per task
- Prior work (Bjarnason et al. 2026) showed that single runs of coding agents vary substantially, with standard deviations above 1.5 percentage points even at temperature 0
- The authors extend this by studying a **continuous quality measure** (AUC) on a single task, rather than pass/fail rates across many tasks

### Key Theoretical Distinction
The central distinction is between:
1. **Selecting a useful artifact** (achievable with few attempts + compliance checking)
2. **Ranking configurations** (requires tens of runs per configuration)

### Task Design
The agent receives a working XGBoost training script for predicting flight delays (≥15 min late) from 8 features. The agent edits the training code to improve performance, scored by AUC on a hidden holdout of 1,000,000 flights that no agent ever sees.

## Methodology

### Experimental Design (Three Studies)

| Study | Question | Agents | Models | Runs |
|-------|----------|--------|--------|------|
| Study 1 | Does the pairing matter? | All six | Six endpoints | 3 per pairing (116 total) |
| Study 2 | How much does one run vary? | pi, OpenCode, Hermes | GLM-5.3 Flash, DeepSeek 4.1 Flash | 52 per pairing (312 total) |
| Study 3 | What does a larger model buy? | pi, OpenCode, Hermes | GLM-5.3 vs. GLM-5.3 Flash | 52 per pairing (156 total) |

### Agents Tested
- Claude Code 2.1.268, Codex CLI 0.153.4, pi 0.85.1, OpenCode 1.18.30, OpenClaw 2026.9.4, Hermes (pinned commit)

### Key Methodological Features
- **Fixed settings**: Data, prompt, budget (18,000 CPU-seconds), machine type, and parallelism were identical across runs
- **Independent runs**: Nothing carries over between runs; seed numbers are purely labels
- **Compliance auditing**: Two automated traces over every delivered file check for: (1) evaluation labels reaching model fits, and (2) features computed from the prediction batch
- **Scoring**: The delivered `train.py` is re-run against the hidden holdout outside the agent's reach

### Statistical Approach
- Standard errors of differences: $\sqrt{SE_1^2 + SE_2^2}$
- Sample size approximation for detecting differences: $16 \cdot sd^2 / gap^2$ per arm (for 80% power, two-sided 5% test)
- Bootstrap intervals with 20,000 resamples for gains
- Exact computation of repeat-and-select policy distributions (no simulation)

## Empirical Validation / Results

### Finding 1: One Run Is a Draw
- Six pairing means span 0.0095 AUC, while median pairing SD = 0.0107 AUC
- Two runs of the same pairing differ by 0.0147 AUC on average; 0.0285 at the 9th decile
- Three-run comparisons put the weaker pairing ahead 28–44% of the time

### Finding 2: Compliance Failures at the Top of the Ranking
- 5 runs trained on the labeled evaluation file (scored 0.7855–0.8293)
- 5 runs computed features from the scoring batch (scored 0.7116–0.8036)
- Removing violations lowers the best score from 0.8293 to 0.7695

### Finding 3: Best-of-K Policy Results

| Attempts (k) | At least one compliant | Median kept | Gain over one attempt (95% CI) |
|---|---|---|---|
| 1 | 96.8% (94.2–98.2) | 0.7412 | — |
| 3 | ≥99.88% | 0.7493 | +0.0081 (+0.0063 to +0.0098) |
| 5 | ≥99.99% | 0.7526 | +0.0114 (+0.0085 to +0.0131) |
| 10 | ≥99.99% | 0.7548 | +0.0136 (+0.0113 to +0.0154) |

Selection on the evaluation set achieved nearly identical holdout performance as oracle selection (mean difference ≤0.00005 AUC).

### Finding 4: Model Upgrade Effect
- GLM-5.3 scored 0.0091 AUC higher than GLM-5.3 Flash (7.1 SE from zero)
- This equals 0.87 run-to-run standard deviations—one run of each still favors the smaller model 28% of the time
- **Agent-by-model interaction**: pi's gain (+0.0152) exceeded OpenCode's by 3.3 SE and Hermes's by 2.7 SE (joint test: $\chi^2 = 12.3$, p = 0.002)

### Finding 5: Cost and Caching
On DeepSeek V4 Flash through Ollama Cloud:
- pi: $0.08/run (97–98% cache hit rate)
- Claude Code: $1.86/run (2–11% cache hit rate)
- Cache accounts for ~9-fold of the 23-fold gap; the rest is token volume
- Cache failure was pairing-specific: Claude Code cached normally on other models (73–97%)

### Finding 6: Later-Year Transfer
On 2007 flights (vs. 2006 tuning year):
- Only one-third of the gain survives (+0.0092 vs. +0.0280)
- Run-to-run SD halves (0.0060 vs. 0.0107)
- Larger model's advantage falls to 0.44 run-to-run SDs
- pi's larger gain no longer separates from other agents'

## Theoretical and Practical Implications

### For Agent Evaluation
1. **Unit of evaluation should be the pairing** (agent + model + endpoint), not the agent or model alone—agent rankings reversed between models
2. **Run counts depend on the question**: detecting the model upgrade needs ~21 runs/arm; distinguishing agents needs ~39–113 runs/arm
3. **Choosing ≠ ranking**: a few attempts with compliance checking reliably buy a better artifact; ranking configurations takes tens of runs

### For Deployment Practice
- **Reject first, then rank**: compliance checking must precede score comparison
- **Best-of-3 with compliance check**: median +0.008 AUC improvement, at ~$2 cost
- **Instrument what nobody looks at**: cache hit rates and compute usage predict both cost and quality
- **Validate on later, untouched data**: tuning-year gains overstate durable improvements

### Economic Analysis
The paper demonstrates that AUC differences should not be directly converted to monetary value. Using decision curve analysis at real-world prevalence (1% positive cases), the better-AUC model can be worse at some capacity constraints (Table 14 shows a crossing point where the 0.7695 AUC model underperforms the 0.7471 AUC model at 5,000 cases/month review capacity).

## Conclusion

The paper's key takeaways for teams deploying coding agents:

1. **Evaluate the pairing, not the parts**—record whether attempts started and completed
2. **Attempt the job several times**—best compliant artifact among 3 attempts gives median +0.008 AUC
3. **Reject first, then rank**—check compliance before comparing scores; make violations impossible where possible
4. **Rank on your objective, confirm on later data**—a later year kept only a third of the tuning-year gain
5. **Instrument what nobody looks at**—cached input share and compute usage
6. **Price the winner with its predictions**—at your prevalence and costs
7. **Test again when anything changes**—model, agent version, endpoint, cache, or price

### Limitations
- Single task (tablular prediction) and dataset
- Exploratory interaction analysis (not pre-specified)
- Hosted endpoints change over time
- Retrospective policy analysis
- Compliance detection limited to anticipated violation types

### Future Work
- Pre-specified replication of the agent-by-model interaction on another model family
- Testing cache failure mechanisms with byte-identical prompts in both API formats
- Investigating how context management strategies interact with prompt caching

---

_Markdown view of https://picx.dev/p/4nA4iN, served by PicX — AI-generated visual whiteboard summaries of research papers._