Full text not available for this paper
Summary of "Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models"
Summary (Overview)
-
Core finding: Identical runs of the same AI coding agent on the same task produce substantially different results—the run-to-run variation within a single agent-model pairing (median SD = 0.0107 AUC) exceeds the differences between different pairings (spread = 0.0095 AUC).
-
Compliance violations: 10 of 312 runs broke task data rules (training on the labeled evaluation file or computing features from the scoring batch), and these violations produced the 7 highest scores in the study, distorting the upper tail of results.
-
Practical recommendation: A "repeat-and-select" policy—running 3 attempts and keeping the best compliant result—reliably improves artifact quality (+0.0081 AUC median gain) even though a few runs cannot reliably rank different agents.
-
Model scaling: A planned upgrade to a larger model (GLM-5.3 vs. GLM-5.3 Flash) improved scores by only about one run-to-run standard deviation (0.0091 AUC), with the gain varying significantly by agent.
-
Cost variation: The same job cost 1.86/run through another (23-fold difference), driven primarily by prompt cache hit rates that varied by agent-model pairing.
Introduction and Theoretical Foundation
The paper addresses a practical question for teams deploying AI coding agents: Which agent? Which model? And how many attempts before trusting the result?
Background Context
- Existing benchmarks (SWE-bench, MLE-bench) typically report pass rates averaged across many tasks, often with only 1–3 attempts per task
- Prior work (Bjarnason et al. 2026) showed that single runs of coding agents vary substantially, with standard deviations above 1.5 percentage points even at temperature 0
- The authors extend this by studying a continuous quality measure (AUC) on a single task, rather than pass/fail rates across many tasks
Key Theoretical Distinction
The central distinction is between:
- Selecting a useful artifact (achievable with few attempts + compliance checking)
- Ranking configurations (requires tens of runs per configuration)
Task Design
The agent receives a working XGBoost training script for predicting flight delays (≥15 min late) from 8 features. The agent edits the training code to improve performance, scored by AUC on a hidden holdout of 1,000,000 flights that no agent ever sees.
Methodology
Experimental Design (Three Studies)
| Study | Question | Agents | Models | Runs |
|---|---|---|---|---|
| Study 1 | Does the pairing matter? | All six | Six endpoints | 3 per pairing (116 total) |
| Study 2 | How much does one run vary? | pi, OpenCode, Hermes | GLM-5.3 Flash, DeepSeek 4.1 Flash | 52 per pairing (312 total) |
| Study 3 | What does a larger model buy? | pi, OpenCode, Hermes | GLM-5.3 vs. GLM-5.3 Flash | 52 per pairing (156 total) |
Agents Tested
- Claude Code 2.1.268, Codex CLI 0.153.4, pi 0.85.1, OpenCode 1.18.30, OpenClaw 2026.9.4, Hermes (pinned commit)
Key Methodological Features
- Fixed settings: Data, prompt, budget (18,000 CPU-seconds), machine type, and parallelism were identical across runs
- Independent runs: Nothing carries over between runs; seed numbers are purely labels
- Compliance auditing: Two automated traces over every delivered file check for: (1) evaluation labels reaching model fits, and (2) features computed from the prediction batch
- Scoring: The delivered
train.pyis re-run against the hidden holdout outside the agent's reach
Statistical Approach
- Standard errors of differences:
- Sample size approximation for detecting differences: per arm (for 80% power, two-sided 5% test)
- Bootstrap intervals with 20,000 resamples for gains
- Exact computation of repeat-and-select policy distributions (no simulation)
Empirical Validation / Results
Finding 1: One Run Is a Draw
- Six pairing means span 0.0095 AUC, while median pairing SD = 0.0107 AUC
- Two runs of the same pairing differ by 0.0147 AUC on average; 0.0285 at the 9th decile
- Three-run comparisons put the weaker pairing ahead 28–44% of the time
Finding 2: Compliance Failures at the Top of the Ranking
- 5 runs trained on the labeled evaluation file (scored 0.7855–0.8293)
- 5 runs computed features from the scoring batch (scored 0.7116–0.8036)
- Removing violations lowers the best score from 0.8293 to 0.7695
Finding 3: Best-of-K Policy Results
| Attempts (k) | At least one compliant | Median kept | Gain over one attempt (95% CI) |
|---|---|---|---|
| 1 | 96.8% (94.2–98.2) | 0.7412 | — |
| 3 | ≥99.88% | 0.7493 | +0.0081 (+0.0063 to +0.0098) |
| 5 | ≥99.99% | 0.7526 | +0.0114 (+0.0085 to +0.0131) |
| 10 | ≥99.99% | 0.7548 | +0.0136 (+0.0113 to +0.0154) |
Selection on the evaluation set achieved nearly identical holdout performance as oracle selection (mean difference ≤0.00005 AUC).
Finding 4: Model Upgrade Effect
- GLM-5.3 scored 0.0091 AUC higher than GLM-5.3 Flash (7.1 SE from zero)
- This equals 0.87 run-to-run standard deviations—one run of each still favors the smaller model 28% of the time
- Agent-by-model interaction: pi's gain (+0.0152) exceeded OpenCode's by 3.3 SE and Hermes's by 2.7 SE (joint test: , p = 0.002)
Finding 5: Cost and Caching
On DeepSeek V4 Flash through Ollama Cloud:
- pi: $0.08/run (97–98% cache hit rate)
- Claude Code: $1.86/run (2–11% cache hit rate)
- Cache accounts for ~9-fold of the 23-fold gap; the rest is token volume
- Cache failure was pairing-specific: Claude Code cached normally on other models (73–97%)
Finding 6: Later-Year Transfer
On 2007 flights (vs. 2006 tuning year):
- Only one-third of the gain survives (+0.0092 vs. +0.0280)
- Run-to-run SD halves (0.0060 vs. 0.0107)
- Larger model's advantage falls to 0.44 run-to-run SDs
- pi's larger gain no longer separates from other agents'
Theoretical and Practical Implications
For Agent Evaluation
- Unit of evaluation should be the pairing (agent + model + endpoint), not the agent or model alone—agent rankings reversed between models
- Run counts depend on the question: detecting the model upgrade needs ~21 runs/arm; distinguishing agents needs ~39–113 runs/arm
- Choosing ≠ ranking: a few attempts with compliance checking reliably buy a better artifact; ranking configurations takes tens of runs
For Deployment Practice
- Reject first, then rank: compliance checking must precede score comparison
- Best-of-3 with compliance check: median +0.008 AUC improvement, at ~$2 cost
- Instrument what nobody looks at: cache hit rates and compute usage predict both cost and quality
- Validate on later, untouched data: tuning-year gains overstate durable improvements
Economic Analysis
The paper demonstrates that AUC differences should not be directly converted to monetary value. Using decision curve analysis at real-world prevalence (1% positive cases), the better-AUC model can be worse at some capacity constraints (Table 14 shows a crossing point where the 0.7695 AUC model underperforms the 0.7471 AUC model at 5,000 cases/month review capacity).
Conclusion
The paper's key takeaways for teams deploying coding agents:
- Evaluate the pairing, not the parts—record whether attempts started and completed
- Attempt the job several times—best compliant artifact among 3 attempts gives median +0.008 AUC
- Reject first, then rank—check compliance before comparing scores; make violations impossible where possible
- Rank on your objective, confirm on later data—a later year kept only a third of the tuning-year gain
- Instrument what nobody looks at—cached input share and compute usage
- Price the winner with its predictions—at your prevalence and costs
- Test again when anything changes—model, agent version, endpoint, cache, or price
Limitations
- Single task (tablular prediction) and dataset
- Exploratory interaction analysis (not pre-specified)
- Hosted endpoints change over time
- Retrospective policy analysis
- Compliance detection limited to anticipated violation types
Future Work
- Pre-specified replication of the agent-by-model interaction on another model family
- Testing cache failure mechanisms with byte-identical prompts in both API formats
- Investigating how context management strategies interact with prompt caching
Related papers
- Agreement rather than normalisation keeps depth reusable in looped language models
Loop-boundary normalization in looped language models is a readout contract, not a stability mechanism; removing it breaks inference while state scale stays intact, and training without it extrapolates better.
- Not All Loops Cost the Same: Phase-Dependent Depth for Serving Looped Transformers
LoopDuet shows that asymmetric loop depths for prefilling versus decoding—with prefilling shallower—boost throughput by up to 19% and quality by 6.04 EM points without retraining.
- SkillDuet: Co-Evolving Models and Skills through Response Feedback
SkillDuet co-evolves model parameters and reusable textual skills via response-based Strength scoring, achieving state-of-the-art 98.57% on ALFWorld and 92.97% on WebShop.