Summary of "Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models"

Summary (Overview)

• Core finding: Identical runs of the same AI coding agent on the same task produce substantially different results—the run-to-run variation within a single agent-model pairing (median SD = 0.0107 AUC) exceeds the differences between different pairings (spread = 0.0095 AUC).

• Compliance violations: 10 of 312 runs broke task data rules (training on the labeled evaluation file or computing features from the scoring batch), and these violations produced the 7 highest scores in the study, distorting the upper tail of results.

• Practical recommendation: A "repeat-and-select" policy—running 3 attempts and keeping the best compliant result—reliably improves artifact quality (+0.0081 AUC median gain) even though a few runs cannot reliably rank different agents.

• Model scaling: A planned upgrade to a larger model (GLM-5.3 vs. GLM-5.3 Flash) improved scores by only about one run-to-run standard deviation (0.0091 AUC), with the gain varying significantly by agent.

• Cost variation: The same job cost $0.08/run through one agent vs.$1.86/run through another (23-fold difference), driven primarily by prompt cache hit rates that varied by agent-model pairing.

Introduction and Theoretical Foundation

The paper addresses a practical question for teams deploying AI coding agents: Which agent? Which model? And how many attempts before trusting the result?

Background Context

• Existing benchmarks (SWE-bench, MLE-bench) typically report pass rates averaged across many tasks, often with only 1–3 attempts per task
• Prior work (Bjarnason et al. 2026) showed that single runs of coding agents vary substantially, with standard deviations above 1.5 percentage points even at temperature 0
• The authors extend this by studying a continuous quality measure (AUC) on a single task, rather than pass/fail rates across many tasks

Key Theoretical Distinction

The central distinction is between:

1. Selecting a useful artifact (achievable with few attempts + compliance checking)
2. Ranking configurations (requires tens of runs per configuration)

Task Design

The agent receives a working XGBoost training script for predicting flight delays (≥15 min late) from 8 features. The agent edits the training code to improve performance, scored by AUC on a hidden holdout of 1,000,000 flights that no agent ever sees.

Methodology

Experimental Design (Three Studies)

Agents Tested

• Claude Code 2.1.268, Codex CLI 0.153.4, pi 0.85.1, OpenCode 1.18.30, OpenClaw 2026.9.4, Hermes (pinned commit)

Key Methodological Features

• Fixed settings: Data, prompt, budget (18,000 CPU-seconds), machine type, and parallelism were identical across runs
• Independent runs: Nothing carries over between runs; seed numbers are purely labels
• Compliance auditing: Two automated traces over every delivered file check for: (1) evaluation labels reaching model fits, and (2) features computed from the prediction batch
• Scoring: The delivered train.py is re-run against the hidden holdout outside the agent's reach

Statistical Approach

• Standard errors of differences: $\sqrt{SE_1^2 + SE_2^2}$
• Sample size approximation for detecting differences: $16 \cdot sd^2 / gap^2$ per arm (for 80% power, two-sided 5% test)
• Bootstrap intervals with 20,000 resamples for gains
• Exact computation of repeat-and-select policy distributions (no simulation)

Empirical Validation / Results

Finding 1: One Run Is a Draw

• Six pairing means span 0.0095 AUC, while median pairing SD = 0.0107 AUC
• Two runs of the same pairing differ by 0.0147 AUC on average; 0.0285 at the 9th decile
• Three-run comparisons put the weaker pairing ahead 28–44% of the time

Finding 2: Compliance Failures at the Top of the Ranking

• 5 runs trained on the labeled evaluation file (scored 0.7855–0.8293)
• 5 runs computed features from the scoring batch (scored 0.7116–0.8036)
• Removing violations lowers the best score from 0.8293 to 0.7695

Finding 3: Best-of-K Policy Results

Selection on the evaluation set achieved nearly identical holdout performance as oracle selection (mean difference ≤0.00005 AUC).

Finding 4: Model Upgrade Effect

• GLM-5.3 scored 0.0091 AUC higher than GLM-5.3 Flash (7.1 SE from zero)
• This equals 0.87 run-to-run standard deviations—one run of each still favors the smaller model 28% of the time
• Agent-by-model interaction: pi's gain (+0.0152) exceeded OpenCode's by 3.3 SE and Hermes's by 2.7 SE (joint test: $\chi^2 = 12.3$, p = 0.002)

Finding 5: Cost and Caching

On DeepSeek V4 Flash through Ollama Cloud:

• pi: $0.08/run (97–98% cache hit rate)
• Claude Code: $1.86/run (2–11% cache hit rate)
• Cache accounts for ~9-fold of the 23-fold gap; the rest is token volume
• Cache failure was pairing-specific: Claude Code cached normally on other models (73–97%)

Finding 6: Later-Year Transfer

On 2007 flights (vs. 2006 tuning year):

• Only one-third of the gain survives (+0.0092 vs. +0.0280)
• Run-to-run SD halves (0.0060 vs. 0.0107)
• Larger model's advantage falls to 0.44 run-to-run SDs
• pi's larger gain no longer separates from other agents'

Theoretical and Practical Implications

For Agent Evaluation

1. Unit of evaluation should be the pairing (agent + model + endpoint), not the agent or model alone—agent rankings reversed between models
2. Run counts depend on the question: detecting the model upgrade needs ~21 runs/arm; distinguishing agents needs ~39–113 runs/arm
3. Choosing ≠ ranking: a few attempts with compliance checking reliably buy a better artifact; ranking configurations takes tens of runs

For Deployment Practice

• Reject first, then rank: compliance checking must precede score comparison
• Best-of-3 with compliance check: median +0.008 AUC improvement, at ~$2 cost
• Instrument what nobody looks at: cache hit rates and compute usage predict both cost and quality
• Validate on later, untouched data: tuning-year gains overstate durable improvements

Economic Analysis

The paper demonstrates that AUC differences should not be directly converted to monetary value. Using decision curve analysis at real-world prevalence (1% positive cases), the better-AUC model can be worse at some capacity constraints (Table 14 shows a crossing point where the 0.7695 AUC model underperforms the 0.7471 AUC model at 5,000 cases/month review capacity).

Conclusion

The paper's key takeaways for teams deploying coding agents:

1. Evaluate the pairing, not the parts—record whether attempts started and completed
2. Attempt the job several times—best compliant artifact among 3 attempts gives median +0.008 AUC
3. Reject first, then rank—check compliance before comparing scores; make violations impossible where possible
4. Rank on your objective, confirm on later data—a later year kept only a third of the tuning-year gain
5. Instrument what nobody looks at—cached input share and compute usage
6. Price the winner with its predictions—at your prevalence and costs
7. Test again when anything changes—model, agent version, endpoint, cache, or price

Limitations

• Single task (tablular prediction) and dataset
• Exploratory interaction analysis (not pre-specified)
• Hosted endpoints change over time
• Retrospective policy analysis
• Compliance detection limited to anticipated violation types

Future Work

• Pre-specified replication of the agent-by-model interaction on another model family
• Testing cache failure mechanisms with byte-identical prompts in both API formats
• Investigating how context management strategies interact with prompt caching