Full text not available for this paper
Revisiting the Survival of Agent-Attributed Code: How Measurement Choices Decide the Verdict
Summary (Overview)
- This paper systematically investigates how nine measurement choices in post-merge studies of AI coding agent code can change research conclusions, building on a corpus of 854,139 lines from 2,802 merged pull requests (PRs) across 215 repositories in the AIDev dataset.
- The authors construct a specification curve of 88 survival analyses, varying choices related to line tracking, unit of analysis, modification definition, intent classification, comparison scope, variance estimation, observation window, and maintainer identification.
- Key finding: For human-supervised agents (mostly GitHub Copilot), measurement choices change only the effect size but not the direction—agent code is consistently retained longer than human code. For the fully-autonomous agent Devin, measurement choices can reverse the verdict entirely.
- The variance estimator is decisive for Devin: with model-based standard errors, 15 of 22 specifications report Devin code modified sooner; with errors clustered by repository, only 3 do—no more than permuted labels produce.
- LLM-based intent classifiers agree with human labels (–) nearly as well as two human raters agree with each other (), whereas keyword rules perform poorly () and produce different verdicts on why agent code is modified.
Introduction and Theoretical Foundation
Background and Motivation
Coding agents such as GitHub Copilot, OpenAI Codex, Cursor, Claude Code, and Devin now contribute pull requests to open-source projects. Once merged, this code must be maintained—a cost that accounts for approximately 70% of total software cost (Boehm's analysis). A growing body of research studies the post-merge behavior of agent code, particularly its survival or retention—how long code stays unmodified after being merged.
However, existing studies reach conflicting verdicts:
| Study | Finding |
|---|---|
| Xia and Miller [109] | Agentic code terminated at similar rate to human code (HR = 1.11, not significant) |
| Popescu et al. [80] | Human code survives consistently longer than code from every agent |
| Kraishan [62] | Some agents' code churns less than human code; others at human level |
| Takerngsaksiri et al. [100] | Most follow-up fixes of agent PRs come from the same agent |
| Sawada et al. [89] | Human developers do most maintenance of agent-created files |
Theoretical Foundation
The paper builds on established survival analysis methods:
Kaplan-Meier estimation estimates the retention function , the probability that a code unit remains unmodified beyond time :
where is the number of modifications at time and is the number of unmodified units just prior to .
Cox proportional hazards regression estimates a hazard ratio (HR) comparing how likely agent and human lines are to be modified at any moment after merge. An HR below 1 means agent lines are modified later (retained longer); an HR above 1 means modified sooner.
The Nine Measurement Choices
A post-merge study must decide:
- Line tracking: by location (line number) or identity (following the line through moves/renames)
- Files kept: whether vendored/generated code is removed
- Unit of analysis: line, hunk (block of consecutive added lines), file, or PR; and how much each unit counts
- Modification definition: whether whitespace-only or rename edits count
- Intent classification: how the intent of a modification is inferred
- Comparison scope: across repositories or within them
- Uncertainty estimation: whether lines from the same repository are treated as independent
- Observation window: how long code is followed after merge
- Maintainer identification: how the developer/agent who modifies code is identified
Methodology
Dataset and Cohort
The study uses the AIDev dataset [64], which attributes PRs to coding agents via metadata signals (branch prefixes, bot usernames, Co-Authored-By strings). The cohort comprises:
- 2,802 merged PRs in 215 repositories
- 1,470 agent PRs and 1,332 human PRs
- 854,139 added lines total; 644,177 lines after removing vendored/generated code
- Agents grouped into two tiers:
- Human-supervised: Copilot, Cursor, Claude Code, Codex (1,055 PRs; 177,787 lines)
- Fully-autonomous: Devin (412 PRs; 87,634 lines)
Agent Taxonomy
| Tier | Agents | Collaboration Mode |
|---|---|---|
| Human-supervised | Copilot, Cursor, Claude Code, Codex | Variably human-supervised; degree of oversight unverifiable from PR metadata |
| Fully-autonomous | Devin | Whole tasks delegated to the agent, which submits PRs as an independent contributor |
Line Tracking
Three trackers were compared:
- Identity tracking (primary):
git blame --reversewith rename/copy detection () along the first-parent chain - Earlier study's tracker:
git blame --reversewithout rename/copy detection, first later commit touching the file - Line-number tracking: follows lines by position, with file deletion treated as deletion of all lines
Manual validation of 300 units showed:
- Identity tracking: 98.7% precision on "modified" claims, 92.8% recall
- Earlier tracker: 93.3% precision, 96.1% recall
- Line-number tracking: only 35.9% precision, 84.0% recall
Modification Definitions
Definition 1 (Syntactic): A line is modified at its first verified change—content change, line deletion, or file deletion. Includes whitespace edits, renames, and literal changes.
Definition 2 (AST-structural): A line is modified only when the edit changes the structure of the enclosing function, method, or class (the shape of its abstract syntax tree). Leaf-only changes (names, values, formatting) are not counted.
The AST classification uses tree-sitter with language-specific grammars covering 17 languages, validated against human labels with 93.7% agreement ().
Intent Classification
Seven approaches applied to 5,303 modification commits:
- Keyword rules: Conventional Commits prefixes, bracket prefixes, 86 regular expressions
- Three LLMs: DeepSeek-V4.1-Flash, gpt-oss-120b, MiniMax-M3.1-Flash (temperature 0)
- Three combinations: Majority vote, weighted vote, Dawid-Skene aggregation
Validation against 618 human-labeled commits with four categories: corrective, perfective, adaptive, preventive.
Statistical Analysis
- Cox models across and within repositories (stratified by repository)
- Standard errors: model-based (treating lines as independent), clustered by PR, clustered by repository (primary)
- Cluster bootstrap: 500 resamples of whole repositories for headline within-repository estimates
- Permutation tests: 500 shuffles of tier labels among PRs
- Bonferroni correction: for two tier comparisons
- Specification curve: 88 specifications (44 per tier)
Empirical Validation / Results
RQ1: Line Tracking and Unit of Analysis
The measurement ladder shows how choices move Devin's hazard ratio:
| Step | Human-supervised HR | Devin HR |
|---|---|---|
| Start: earlier tracker, all lines incl. vendored, per line | 0.39 | 0.39 |
| Step 1: identity tracking, all lines incl. vendored | 0.38–0.72 | 0.39–0.84 |
| Step 2: vendored code removed | 0.72–0.90 | 0.83–1.19 |
| Step 3: sampled units | 0.67–1.29 | 0.72–1.18 |
| Step 4: unit variation | 0.72–0.89 | 0.90–1.56 |
Key findings:
- Line-number tracking records about twice as many modifications as identity tracking (64.5% vs. 29.0% of human lines modified), moving Devin's HR from 1.294 to 0.992
- Vendored code removal changes HR from 0.38–0.39 to ~0.9 for both tiers
- Unit choice matters greatly for Devin: per line HR = 0.896, per PR = 1.187, per hunk = 1.56
- One PR (pdfme/pdfme#1059) holds 41.7% of all Devin lines with only 4.8% modified
RQ2: Modification Definition
| Group | Syntactic modification rate | AST-structural rate |
|---|---|---|
| Human-supervised | 20.1% | 15.7% |
| Devin | 21.5% | 15.3% |
| Human baseline | 25.0% | 18.9% |
The AST-structural definition changes effect sizes but not directions. Within repositories, it makes the human-supervised difference significant (HR = 0.717, 95% CI [0.477, 0.921]). Devin receives the most leaf-only changes (14.2% of modifications vs. 7.0% for human-supervised and 5.8% for human code).
RQ3: Comparison Scope and Variance Estimator
Repository coverage: The tiers come from largely different repositories—only 10 of 215 repositories contain both agent tiers (Jaccard overlap = 0.058).
Verdict counts under three variance estimators:
| Tier | Comparison | Clustered by repository | Clustered by PR | Model-based |
|---|---|---|---|---|
| Human-supervised | Across | 13/9/0 | 13/9/0 | 21/0/1 |
| Human-supervised | Within | 11/10/1 | 12/9/1 | 21/0/1 |
| Devin | Across | 1/18/3 | 0/13/9 | 5/2/15 |
| Devin | Within | 0/22/0 | 1/20/1 | 11/4/7 |
Format: longer/none/sooner
Per-agent within-repository results:
| Agent | HR [95% CI] | Repositories |
|---|---|---|
| GitHub Copilot | 0.653 [0.459, 0.954] | 53 |
| OpenAI Codex | 0.831 [0.468, 1.218] | 63 |
| Cursor | 0.379 [0.150, 3.695] | 16 |
| Claude Code | 0.934 [0.136, 1.950] | 13 |
| Devin | 1.059 [0.409, 2.076] | 31 |
RQ4: Intent Classification
Agreement with human labels:
| Approach | Cohen's |
|---|---|
| Keyword rules | 0.38 [0.33, 0.42] |
| DeepSeek | 0.70 |
| gpt-oss | 0.69 |
| MiniMax | 0.71 |
| Majority vote | 0.71 |
| Weighted vote | 0.71 |
| Dawid-Skene | 0.72 |
Key findings:
- LLMs agree with human labels about as well as two human raters agree with each other ()
- Only keyword rules find that Devin code is modified for different reasons than human code (, ); no LLM approach confirms this ()
- After correcting for classification error, corrective shares overlap across groups (e.g., majority vote: 24.0% human-supervised, 24.4% Devin, 23.6% human)
RQ5: Maintainer Identification
| Code written by | Human modifiers | Agent modifiers | Most frequent agent |
|---|---|---|---|
| Human-supervised agents | 88.6% | 11.4% | Copilot (151 of 166) |
| Devin | 85.7% | 14.3% | Devin (85 of 88) |
| Human developers | 96.9% | 3.1% | Copilot and Devin (40 each of 100) |
Overall: humans make 87.7% of modifications to agent code.
Agent share varies by unit and window (e.g., 29.9% for Devin follow-up PRs within 7 days) but never exceeds ~30%—humans remain the primary maintainers under every choice.
Specification Curve (88 Specifications)
With repository-clustered standard errors:
- Human-supervised: 24 "retained longer", 19 "no difference", 1 "modified sooner"
- Devin: 1 "retained longer", 40 "no difference", 3 "modified sooner"
Permutation tests show human-supervised verdicts exceed chance (), while Devin's do not ().
Published Designs Re-run
| Study | On our data | Single change that moves it |
|---|---|---|
| Xia and Miller [109] | HR 0.92, (agree) | Intent classifier: corrective HR 0.68 (keyword) vs. 1.37 (LLM) |
| Sawada et al. [89] | Humans make 88.2% (agree) | Unit and window |
| Popescu et al. [80] | Agree for Claude Code and Devin | Within-repository comparison |
| Kraishan [62] | Agree on Copilot | Repository model |
| Takerngsaksiri et al. [100] | Not comparable (different outcome) | Within-repository comparison |
Theoretical and Practical Implications
Operationalization Checklist
The paper derives a 10-item checklist for post-merge studies:
- Line tracking: Follow line identity (reverse blame with rename/copy detection); verify modifications in diffs
- Files, sampling, weighting: Remove vendored/generated files; report line- and PR-weighted estimates side by side
- Modification definition: Report modification events by type; use AST-structural definition as sensitivity check
- Intent classifier: Prefer validated LLM classifiers; report with CI; correct shares for classification error
- Comparison: Compare within repositories; report number of contributing repositories and minimum detectable effect
- Standard errors: Cluster by repository in every model; test specification curve against permuted labels
- Observation window: Report follow-up window and the cohort it implies
- Maintainer: State unit, window, and modifier rule; report self-modification share separately
- Time: Check proportional hazards; report piecewise hazard ratios with clustered errors
- Reporting: Publish a specification curve rather than a single pipeline
Implications for Researchers
- Disagreements between post-merge studies should be read through their measurement choices, not just data differences
- The variance estimator should be treated as a measurement choice—clustering by repository is essential
- Collaboration modes should be separated; findings about Devin are findings about Devin, not about all autonomous agents
- LLM classifiers can replace keyword rules for commit intent if validated against human labels
- Power should be reported with null results (minimum detectable effect)
Implications for Practitioners
- Retention/churn figures for AI code depend heavily on measurement; compare only within the same project, measured the same way
- Teams should plan for human maintenance of agent code—agents don't reduce maintenance burden
- Autonomous agents' self-modifications (often within days of merge) deserve the same review as their initial contributions
Conclusion
The paper demonstrates that measurement choices decide the verdict for post-merge studies of agent code, but not uniformly:
- Human-supervised code (mostly Copilot) is consistently retained longer than human code—the effect is robust across specifications, though its size varies with measurement
- Devin code shows no difference that survives the variance estimator; with repository-clustered errors, its "modified sooner" verdicts are no more frequent than chance
- Humans maintain agent code—87.7% of modifications to agent code come from humans, regardless of unit, window, or identification rule
- Intent conclusions depend on the classifier—keyword rules create a Devin difference that LLM classifiers and human labels do not confirm
The authors recommend that future post-merge studies follow the operationalization checklist, cluster standard errors by repository, and report specification curves rather than single pipelines. Future work includes replicating agent effects on other datasets (including newer autonomous agents like Jules and Kiro), following code over longer windows, and using controlled studies to separate generation quality, review, and task selection effects.
Related papers
- Pro$^2$ Agent: Scientific Problem Probing for Autonomous AI R&D
Pro²Agent's explicit Problem State—tracking diagnosed bottlenecks, candidate solutions, and evidence—achieves a record 76% medal rate on MLE-Bench, outperforming code-centric agents.
- RIPPLE: Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit for Long-Horizon Agentic Coding
RIPPLE internalizes prefix-only recovery guidance into a single policy via stage-specific credit, improving coding agents by 4.8 points across four benchmarks without deployment-time critics.
- GUARD: Stabilizing Recursive Self-Improvement with Regret Guidance
GUARD achieves non-regressive agent harness self-improvement without external graders by separating edit-induced regressions from capability gaps via paired regret, guaranteeing monotonic accuracy gains.