Full text not available for this paper

Revisiting the Survival of Agent-Attributed Code: How Measurement Choices Decide the Verdict

Summary (Overview)

  • This paper systematically investigates how nine measurement choices in post-merge studies of AI coding agent code can change research conclusions, building on a corpus of 854,139 lines from 2,802 merged pull requests (PRs) across 215 repositories in the AIDev dataset.
  • The authors construct a specification curve of 88 survival analyses, varying choices related to line tracking, unit of analysis, modification definition, intent classification, comparison scope, variance estimation, observation window, and maintainer identification.
  • Key finding: For human-supervised agents (mostly GitHub Copilot), measurement choices change only the effect size but not the direction—agent code is consistently retained longer than human code. For the fully-autonomous agent Devin, measurement choices can reverse the verdict entirely.
  • The variance estimator is decisive for Devin: with model-based standard errors, 15 of 22 specifications report Devin code modified sooner; with errors clustered by repository, only 3 do—no more than permuted labels produce.
  • LLM-based intent classifiers agree with human labels (κ=0.69\kappa = 0.69–0.720.72) nearly as well as two human raters agree with each other (κ=0.68\kappa = 0.68), whereas keyword rules perform poorly (κ=0.38\kappa = 0.38) and produce different verdicts on why agent code is modified.

Introduction and Theoretical Foundation

Background and Motivation

Coding agents such as GitHub Copilot, OpenAI Codex, Cursor, Claude Code, and Devin now contribute pull requests to open-source projects. Once merged, this code must be maintained—a cost that accounts for approximately 70% of total software cost (Boehm's analysis). A growing body of research studies the post-merge behavior of agent code, particularly its survival or retention—how long code stays unmodified after being merged.

However, existing studies reach conflicting verdicts:

StudyFinding
Xia and Miller [109]Agentic code terminated at similar rate to human code (HR = 1.11, not significant)
Popescu et al. [80]Human code survives consistently longer than code from every agent
Kraishan [62]Some agents' code churns less than human code; others at human level
Takerngsaksiri et al. [100]Most follow-up fixes of agent PRs come from the same agent
Sawada et al. [89]Human developers do most maintenance of agent-created files

Theoretical Foundation

The paper builds on established survival analysis methods:

Kaplan-Meier estimation estimates the retention function S(t)S(t), the probability that a code unit remains unmodified beyond time tt:

S^(t)=∏ti≤t(1−dini)\hat{S}(t) = \prod_{t_i \leq t} \left(1 - \frac{d_i}{n_i}\right)

where did_i is the number of modifications at time tit_i and nin_i is the number of unmodified units just prior to tit_i.

Cox proportional hazards regression estimates a hazard ratio (HR) comparing how likely agent and human lines are to be modified at any moment after merge. An HR below 1 means agent lines are modified later (retained longer); an HR above 1 means modified sooner.

The Nine Measurement Choices

A post-merge study must decide:

  1. Line tracking: by location (line number) or identity (following the line through moves/renames)
  2. Files kept: whether vendored/generated code is removed
  3. Unit of analysis: line, hunk (block of consecutive added lines), file, or PR; and how much each unit counts
  4. Modification definition: whether whitespace-only or rename edits count
  5. Intent classification: how the intent of a modification is inferred
  6. Comparison scope: across repositories or within them
  7. Uncertainty estimation: whether lines from the same repository are treated as independent
  8. Observation window: how long code is followed after merge
  9. Maintainer identification: how the developer/agent who modifies code is identified

Methodology

Dataset and Cohort

The study uses the AIDev dataset [64], which attributes PRs to coding agents via metadata signals (branch prefixes, bot usernames, Co-Authored-By strings). The cohort comprises:

  • 2,802 merged PRs in 215 repositories
  • 1,470 agent PRs and 1,332 human PRs
  • 854,139 added lines total; 644,177 lines after removing vendored/generated code
  • Agents grouped into two tiers:
    • Human-supervised: Copilot, Cursor, Claude Code, Codex (1,055 PRs; 177,787 lines)
    • Fully-autonomous: Devin (412 PRs; 87,634 lines)

Agent Taxonomy

TierAgentsCollaboration Mode
Human-supervisedCopilot, Cursor, Claude Code, CodexVariably human-supervised; degree of oversight unverifiable from PR metadata
Fully-autonomousDevinWhole tasks delegated to the agent, which submits PRs as an independent contributor

Line Tracking

Three trackers were compared:

  1. Identity tracking (primary): git blame --reverse with rename/copy detection (−M−C-M -C) along the first-parent chain
  2. Earlier study's tracker: git blame --reverse without rename/copy detection, first later commit touching the file
  3. Line-number tracking: follows lines by position, with file deletion treated as deletion of all lines

Manual validation of 300 units showed:

  • Identity tracking: 98.7% precision on "modified" claims, 92.8% recall
  • Earlier tracker: 93.3% precision, 96.1% recall
  • Line-number tracking: only 35.9% precision, 84.0% recall

Modification Definitions

Definition 1 (Syntactic): A line is modified at its first verified change—content change, line deletion, or file deletion. Includes whitespace edits, renames, and literal changes.

Definition 2 (AST-structural): A line is modified only when the edit changes the structure of the enclosing function, method, or class (the shape of its abstract syntax tree). Leaf-only changes (names, values, formatting) are not counted.

The AST classification uses tree-sitter with language-specific grammars covering 17 languages, validated against human labels with 93.7% agreement (κ=0.60\kappa = 0.60).

Intent Classification

Seven approaches applied to 5,303 modification commits:

  • Keyword rules: Conventional Commits prefixes, bracket prefixes, 86 regular expressions
  • Three LLMs: DeepSeek-V4.1-Flash, gpt-oss-120b, MiniMax-M3.1-Flash (temperature 0)
  • Three combinations: Majority vote, weighted vote, Dawid-Skene aggregation

Validation against 618 human-labeled commits with four categories: corrective, perfective, adaptive, preventive.

Statistical Analysis

  • Cox models across and within repositories (stratified by repository)
  • Standard errors: model-based (treating lines as independent), clustered by PR, clustered by repository (primary)
  • Cluster bootstrap: 500 resamples of whole repositories for headline within-repository estimates
  • Permutation tests: 500 shuffles of tier labels among PRs
  • Bonferroni correction: α=0.05/2=0.025\alpha = 0.05/2 = 0.025 for two tier comparisons
  • Specification curve: 88 specifications (44 per tier)

Empirical Validation / Results

RQ1: Line Tracking and Unit of Analysis

The measurement ladder shows how choices move Devin's hazard ratio:

StepHuman-supervised HRDevin HR
Start: earlier tracker, all lines incl. vendored, per line0.390.39
Step 1: identity tracking, all lines incl. vendored0.38–0.720.39–0.84
Step 2: vendored code removed0.72–0.900.83–1.19
Step 3: sampled units0.67–1.290.72–1.18
Step 4: unit variation0.72–0.890.90–1.56

Key findings:

  • Line-number tracking records about twice as many modifications as identity tracking (64.5% vs. 29.0% of human lines modified), moving Devin's HR from 1.294 to 0.992
  • Vendored code removal changes HR from 0.38–0.39 to ~0.9 for both tiers
  • Unit choice matters greatly for Devin: per line HR = 0.896, per PR = 1.187, per hunk = 1.56
  • One PR (pdfme/pdfme#1059) holds 41.7% of all Devin lines with only 4.8% modified

RQ2: Modification Definition

GroupSyntactic modification rateAST-structural rate
Human-supervised20.1%15.7%
Devin21.5%15.3%
Human baseline25.0%18.9%

The AST-structural definition changes effect sizes but not directions. Within repositories, it makes the human-supervised difference significant (HR = 0.717, 95% CI [0.477, 0.921]). Devin receives the most leaf-only changes (14.2% of modifications vs. 7.0% for human-supervised and 5.8% for human code).

RQ3: Comparison Scope and Variance Estimator

Repository coverage: The tiers come from largely different repositories—only 10 of 215 repositories contain both agent tiers (Jaccard overlap = 0.058).

Verdict counts under three variance estimators:

TierComparisonClustered by repositoryClustered by PRModel-based
Human-supervisedAcross13/9/013/9/021/0/1
Human-supervisedWithin11/10/112/9/121/0/1
DevinAcross1/18/30/13/95/2/15
DevinWithin0/22/01/20/111/4/7

Format: longer/none/sooner

Per-agent within-repository results:

AgentHR [95% CI]Repositories
GitHub Copilot0.653 [0.459, 0.954]53
OpenAI Codex0.831 [0.468, 1.218]63
Cursor0.379 [0.150, 3.695]16
Claude Code0.934 [0.136, 1.950]13
Devin1.059 [0.409, 2.076]31

RQ4: Intent Classification

Agreement with human labels:

ApproachCohen's κ\kappa
Keyword rules0.38 [0.33, 0.42]
DeepSeek0.70
gpt-oss0.69
MiniMax0.71
Majority vote0.71
Weighted vote0.71
Dawid-Skene0.72

Key findings:

  • LLMs agree with human labels about as well as two human raters agree with each other (κ=0.68\kappa = 0.68)
  • Only keyword rules find that Devin code is modified for different reasons than human code (p<0.001p < 0.001, V=0.11V = 0.11); no LLM approach confirms this (p≥0.37p \geq 0.37)
  • After correcting for classification error, corrective shares overlap across groups (e.g., majority vote: 24.0% human-supervised, 24.4% Devin, 23.6% human)

RQ5: Maintainer Identification

Code written byHuman modifiersAgent modifiersMost frequent agent
Human-supervised agents88.6%11.4%Copilot (151 of 166)
Devin85.7%14.3%Devin (85 of 88)
Human developers96.9%3.1%Copilot and Devin (40 each of 100)

Overall: humans make 87.7% of modifications to agent code.

Agent share varies by unit and window (e.g., 29.9% for Devin follow-up PRs within 7 days) but never exceeds ~30%—humans remain the primary maintainers under every choice.

Specification Curve (88 Specifications)

With repository-clustered standard errors:

  • Human-supervised: 24 "retained longer", 19 "no difference", 1 "modified sooner"
  • Devin: 1 "retained longer", 40 "no difference", 3 "modified sooner"

Permutation tests show human-supervised verdicts exceed chance (p=0.008p = 0.008), while Devin's do not (p=0.347p = 0.347).

Published Designs Re-run

StudyOn our dataSingle change that moves it
Xia and Miller [109]HR 0.92, p=0.68p = 0.68 (agree)Intent classifier: corrective HR 0.68 (keyword) vs. 1.37 (LLM)
Sawada et al. [89]Humans make 88.2% (agree)Unit and window
Popescu et al. [80]Agree for Claude Code and DevinWithin-repository comparison
Kraishan [62]Agree on CopilotRepository model
Takerngsaksiri et al. [100]Not comparable (different outcome)Within-repository comparison

Theoretical and Practical Implications

Operationalization Checklist

The paper derives a 10-item checklist for post-merge studies:

  1. Line tracking: Follow line identity (reverse blame with rename/copy detection); verify modifications in diffs
  2. Files, sampling, weighting: Remove vendored/generated files; report line- and PR-weighted estimates side by side
  3. Modification definition: Report modification events by type; use AST-structural definition as sensitivity check
  4. Intent classifier: Prefer validated LLM classifiers; report κ\kappa with CI; correct shares for classification error
  5. Comparison: Compare within repositories; report number of contributing repositories and minimum detectable effect
  6. Standard errors: Cluster by repository in every model; test specification curve against permuted labels
  7. Observation window: Report follow-up window and the cohort it implies
  8. Maintainer: State unit, window, and modifier rule; report self-modification share separately
  9. Time: Check proportional hazards; report piecewise hazard ratios with clustered errors
  10. Reporting: Publish a specification curve rather than a single pipeline

Implications for Researchers

  • Disagreements between post-merge studies should be read through their measurement choices, not just data differences
  • The variance estimator should be treated as a measurement choice—clustering by repository is essential
  • Collaboration modes should be separated; findings about Devin are findings about Devin, not about all autonomous agents
  • LLM classifiers can replace keyword rules for commit intent if validated against human labels
  • Power should be reported with null results (minimum detectable effect)

Implications for Practitioners

  • Retention/churn figures for AI code depend heavily on measurement; compare only within the same project, measured the same way
  • Teams should plan for human maintenance of agent code—agents don't reduce maintenance burden
  • Autonomous agents' self-modifications (often within days of merge) deserve the same review as their initial contributions

Conclusion

The paper demonstrates that measurement choices decide the verdict for post-merge studies of agent code, but not uniformly:

  • Human-supervised code (mostly Copilot) is consistently retained longer than human code—the effect is robust across specifications, though its size varies with measurement
  • Devin code shows no difference that survives the variance estimator; with repository-clustered errors, its "modified sooner" verdicts are no more frequent than chance
  • Humans maintain agent code—87.7% of modifications to agent code come from humans, regardless of unit, window, or identification rule
  • Intent conclusions depend on the classifier—keyword rules create a Devin difference that LLM classifiers and human labels do not confirm

The authors recommend that future post-merge studies follow the operationalization checklist, cluster standard errors by repository, and report specification curves rather than single pipelines. Future work includes replicating agent effects on other datasets (including newer autonomous agents like Jules and Kiro), following code over longer windows, and using controlled studies to separate generation quality, review, and task selection effects.

Related papers