Update from Hell: Can Coding Agents Survive Dependency Upgrades?

Paper Summary

1. Introduction & Motivation

This paper introduces DEPBENCH, the first executable, repository-level benchmark designed to evaluate whether coding agents can handle dependency-upgrade repair—a realistic and challenging software maintenance task. The core problem is that when a dependency is upgraded, downstream code may break in ways that are not immediately obvious, requiring agents to adapt to hidden, upgrade-induced behavioral changes.

---

2. Key Concepts: Explicit vs. Implicit Breakage

Explicit breakage occurs when signatures, types, or build constraints change, and compilers or type checkers surface the error.

Implicit breakage occurs when internal semantics change; downstream code still compiles but misbehaves.

A benchmark must therefore collect real client-level, upgrade-caused failures rather than library-level API diffs.

---

3. The DEPBENCH Benchmark

DEPBENCH is the first executable, repository-level benchmark that isolates dependency-upgrade repair through:

• Patch decomposition — splitting PR diffs into disjoint manifest, repair, and test patches
• Held-out test patches — the oracle keeps test changes hidden from the agent
• A four-state oracle (verification protocol) — validating each task across four states
• Cross-ecosystem coverage — including npm/yarn, Maven, Go, Cargo, and Python

---

4. Construction Pipeline

The pipeline combines:

• Mining of dependency-update pull requests (e.g., Dependabot)
• Path-based hunk classification (manifest vs. repair vs. test patches)
• Manual + LLM-assisted audit for ambiguous cases
• Leakage controls to prevent the agent from seeing test patches
• Four-state oracle validation to filter tasks where the repair patch is sufficient

---

5. Patch Decomposition

Given a merged PR bumping a dependency, the diff is split into three disjoint patches:

All remaining diff hunks after manifest/test removal are the repair patch.

---

6. Four-State Oracle Validation

State 0 (before repair): manifest + test, no repair → should fail
State 1 (full PR): manifest + repair + test → should pass  
State 2 (repair only): manifest + repair + original tests → should pass
State 3 (no repair): manifest + updated tests without repair → should fail

7. Results

Dependency-upgrade repair remains difficult: the best configuration correctly repairs less than half of all tasks.

Pass rates vary strongly by ecosystem (Table III). The best config solves 46/65 Maven tasks but only 6/20 Cargo tasks under Gemini. Python has few tasks but similar patterns.

---

7. Failures Modes

Incomplete migration is the dominant failure category:

• Agents locate the changed API but fail to propagate it through wrappers, helpers, generated state, fixtures, or call sites.
• Example: in glob v10 upgrade, agent fixes the main call site but misses other occurrences in test helpers or build scripts.
• This mode is the most frequent in every configuration, excluding runtime exceptions.

The paper introduces the pass-overlap regime distinction:

• Consensus-hard tasks — unsolved by all model/harness configurations (e.g., 77/203 for GPT-5.5)
• Configuration-asymmetric tasks — solvable by some configurations but not others; these are most diagnostic, revealing incomplete propagation as the key failure mode.

---

6. Key Findings (RQ1–RQ4 Summary)

Main Findings:

1. Dependency-upgrade repair remains largely unsolved. Best configuration (Copilot CLI + GPT-5.5) solves 104/203 tasks, meaning it still fails ~50% of the time.

2. LLM choice dominates. GPT-5.5 > Claude Opus 4.8 > Gemini 3.5 Flash across all matched harnesses. Variation between harnesses is also significant (up to 15 points for the same model).

3. Ecosystem difficulty varies sharply. Maven/Java tasks are the most tractable (44/68 solved); Cargo/Rust tasks are the hardest (6/20 solved by best config). npm/yarn tasks sit in the middle.

4. Primary failure mode: incomplete migration. Agents often identify the right dependency surface but fail to propagate changes to all wrappers, call sites, helpers, fixtures, generated code, or output assertions.

5. Consensus-hard tasks (e.g., 77/203 for GPT-5.5) remain unsolved by all configurations—often requiring multi-file or cross-cutting changes.

---

6. Key Results & Statistics

Ecosystem Variation: Maven/Java is the most solvable (≈44–46 passes across top configs), while Cargo is the hardest (≤6 passes across all configs). npm/yarn is moderately hard (28/68). Python is small but reasonably solved (5–6/10).

Consensus-hard tasks: 77/203 for GPT-5.5, 114/203 for Gemini, 102/203 for Claude Opus 4.8 — i.e., many tasks remain unsolved by all configurations.

---

6. Key Findings

• Dependency-upgrade repair is still largely unsolved. The best configuration solves only ~51% of tasks.
• Incomplete propagation is the dominant failure mode. Even when agents identify the new API surface, they fail to propagate changes through wrappers, helpers, fixtures, or all call sites.
• Test-first workflows help in some ecosystems (e.g., Maven) but hurt in others
• LLM choice dominates harness choice. GPT-5.5 > Claude Opus 4.8 > Gemini 3.5 Flash on matched harnesses.
• Ecosystem heterogeneity: Maven is the easiest for most configs; Cargo and Python are hardest.
• Diagnostic signal: configuration-asymmetric tasks (partially solvable) are more useful than consensus-hard or consensus-easy tasks for benchmarking progress.

---

6. Methodology & Experimental Setup

Experimental protocol:

• Each task run: 10-minute containerized execution with verifier.
• Code repair completed via code editing tool (no internet).
• Verifier applies test patch, executes npm test/equivalent, checks exit code 0.
• Each model–harness config is evaluated with up to 10 minutes and 20 attempts per task.
• Targeted failure re-runs classify non-pass trials.

7. Related Discussions

While commercial coding agents can trigger library upgrades and scaffold migrations that compile, they struggle when updates are silent, invisible, or occur outside compilation units (e.g., ESM-style exports, removed transitive APIs, generated files, behavioral changes in test fixtures). This motivates constructing benchmarks around reproducibly verifiable hidden test expectations.

8. Conclusion

DEPEND-UPGRADE remains a substantial challenge for current coding agents. Even the best-performing configuration (Copilot CLI with GPT-5.5) fails on approximately half of the 203 tasks stomach. Cargo/Rust tasks are almost uniformly unsolved. The most common failure mode is incomplete migration—agents correctly identify the dependency surface change but fail to propagate it across all wrappers/call sites. Consensus-hard tasks are those that require broad, multi-file changes or hidden behavioral invariantsable.

---

9. Future Work

• Expanding the benchmark to cover more ecosystems (e.g., Rust, Go) and more complex upgrades
• Testing newer agentic (e.g., MCP-based) and non-agentic approaches
• Studying how task formulation affects outcome; whether natural-language summaries of failures would help
• Automated construction of validators from oracle PRs to reduce cost
• Adding insight into how model choice, agent scaffolding, and context loading affect upgrade repair performance
• Ultimately, building agents that can audit their own migration subtasks and be more reliable when autonomy is required.

---

10. Limitations

• The verifier runs npm test, cargo test, mvn test etc., which may be flaky or environment-dependent
• The benchmark is static and version-pinned; once oracle commits are encoded in the validator, passing is not equal to producing the oracle solution
• Base-image pinning prevents reproducibility from being harmed by package manager drift; it also fixes the environment at the time of release construction
• The dependency-upgrade paradigm biases toward explicit errors and often masks the full extent of implicit breakage
• Patch decomposition may occasionally assign a chunk that pertains to a manifest-plus-repair combination to the wrong partition, though audit mitigates this risk
• Need for a generalizable, reliable methodology to construct real-world studies of language-model-based software agents and to cover both regression and security-relevant breakage

---

7. Conclusion

Current coding agents succeed on many tasks but fail on a consistent remainder. The agents that provide rich context plus tool-mediated actions, such as Codex CLI or Copilot CLI, perform bestley in this settingiments are less performant. Style matters: the same backend LLM produces qualitatively different results across harnesses mapped to different modelsство; harness choices can shift scores by more than ten points even at constant model, and models with strong benchmark results in other coding regimes (Claude Opus 4.8) do not automatically generalize to this task. Second, architectural choices and scaffolding strongly affect performance. Pattern-matching backends, one-shot evaluation modes, and conservative edit policies appear to harm DEP-UPGRADE performance.

D. A RQ3: When Do Researchers Fail?

Configuration-asymmetric tasks (solved by some configurations, not others) are the most informative: they help us isolate which agent or environment behaviors make the task succeed or fail.

"The configuration-asymmetric per-LLM records are not evenly distributed across ecosystems. For GPT-5.5, 14/18 Cargo tasks and 38/48 npm tasks are configuration asymmetric, meaning that tooling and harness design make an otherwise solvable task succeed or fail." — from paper

• Maven/Java tasks are more symmetric—once an agent knows the correct API, it converges.
• Cargo and npm tasks show large variation between configurations. This is likely because JS and Rust both require propagating changes through multiple wrappers, fixtures, and generated lockfiles.

---

7. Failure Modes

The following table shows the primary failure modes for non-pass trials after adjudication (excluding runtime exceptions):

Configuration-asymmetric non-passes often reveal incomplete migrations: an agent finds the changed dependency surface but fails to propagate it through every wrapper, type, helper, fixture, or call site.

---

8. Key Findings & Takeaways

1. Dependency-upgrade repair is hard even for the strongest models:

   • Best config (Codex + GPT-5.5) solves only 104/203 tasks (51%).
   • Many tasks remain unsolved by any configuration (>50% consensus-hard).

2. Best configuration setup matters significantly.
   The best configuration passes 104/203 benchmarks, versus as low as 51 for weaker configurations. The delta between models is large: GPT-5.5 vs Gemini-3.5-Flash is ~50 points; GPT-5.5 > Claude Opus 4.8 in matched harnesses.

3. Ecosystem matters. Maven/Java tasks have the highest pass rates; Cargo/Rust and front-end JS tasks are the hardest. Failure to propagate dependency-surface changes through the repo is the dominant error mode.

---

Table of Contents

• 1. Introduction
• 2. Background
• 3. Related Work
• 4. The DEPUPGRADE-BENCH benchmark
• 5. Experiments
• 6. Discussion
• 7. Conclusion

---

1. Introduction

As software projects grow, dependency upgrades are frequent but notoriously risky. Even a version bump that exposes no upstream breaking change can break downstream code, because the upgrade to a new version may silently change package behavior or structure. Debugging such failures — dubbed dependency hell / activation hell by practitioners — can easily consume days of engineering effort.

Coding agents are now commonly asked to “update dependency X to version Y” in open-source maintenance. Yet the hidden breakage that appears after dependency updates is a particularly difficult repair mode for agents, because the repository can be in an intermediate state where the changes introduced by the update break existing tests, but the agent must determine what exactly broke and whether to fix the code or the test. Prior agent benchmarks largely focus on issue-to-code generation or end-to-end GitHub-issue resolution; few have focused on dependency-upgrade repair specifically.

---

8. Conclusions

We introduced DEPEND-REPAIR, a benchmark constructed from real Dependabot pull requests that separates dependency-upgrade repair into manifest, repair, and test patches vesting the restoration of tests after dependency bumps in coding agents. Our evaluation shows that this task remains largely unsolved: even the best configuration only solves about half of the 203 upgrades.

The primary failure mode is incomplete migration: agents identify the right upgrade surface but fail to propagate changes through all wrappers, types, helpers, and generated files. This is especially pronounced in ecosystems with weaker static checks (JavaScript, Python) and in Cargo, where absent lockfile awareness causes repeated build failures.

DEPEND-REPAIR contributes the first executable, repository-level benchmark for dependency-upgrade repair, brings agent-based evaluation to this challenging maintenance task, and provides a reproducible construction and diagnosis pipeline for future work.
```# Update from Hell: Can Coding Agents Survive Hidden Breakage in Dependency Upgrades?

**Authors:** Zijian Luo, Runzhi He, Pengfei Gao, Yu Kang, Zeqi Lin, Minghua Ma, Qingwei Lin

## 1. Core Research Problem

The paper addresses a critical gap in evaluating coding agents: **dependency-upgrade repair**. When a dependency is bumped (e.g., via Dependabot), the codebase may break silently or explicitly, and agents must repair the code so tests pass again.

> **Key claim**: Prior benchmarks focus on issue-to-code generation or repository-level bugs but neglect the specific challenge of *dependency-upgrade repair*, where the agent must propagate a changed API surface across the entire repository.

---

### 2. Benchmark Construction: DEPEND-REPAIR

**Contributions:**
1. **DEPEND-REPAIR** — first executable, repository-level benchmark isolating dependency-upgrade repair across five ecosystems
2. A construction pipeline combining:
   - Mining of real Dependabot PRs
   - Path-based hunk classification
   - Manual + LLM-assisted audit
   - Leakage controls
   - **Four-state oracle validation**
3. Evaluation of coding agents under containerized, reproducible verification

---

### 3. Patch Decomposition

Each merged PR is split into three disjoint patches:

| Patch | Content | Routing |
|-------|---------|---------|
| **Manifest patch** | Dependency declarations & lockfiles (`package.json`, `go.mod`, `go.sum`, `Cargo.lock`) | By path |
| **Repair patch** | Production source code changes | All non-test, non-manifest files |
| **Test patch** | Tests, fixtures, mocks, expected outputs | Conventional test directories & names |

**Example (glob 8.1.0 → 10.3.3):**

```diff
// manifest
- "glob": "^8.1.0",
+ "glob": "^10.3.3",

// repair  
- import glob from 'glob'
+ import { globSync } from 'glob'
- for (const file of glob.sync(path)) {
+ for (const file of globSync(path)) {

// test patch
- import glob from 'glob'
+ import { globSync } from 'glob'
- assert.equal(warcCount, glob.sync(FIXTURE_INPUT).length)
+ assert.equal(warcCount, globSync(FIXTURE_INPUT).length)

The test patch is held out during evaluation — this is what makes the task a hidden breakage problem.

---

4. Oracle Design: Four-State Validation

To ensure the task is solvable and the test is meaningful, each candidate task is validated under four states:

Only tasks passing S1 and S2–S3 (i.e., the repair patch is necessary and sufficient) are retained.

---

5. Results: How Hard Is Dependency-Upgrade Repair?

The headline result: even the best configuration solves only ~51% of the benchmark (104/203).

Pass counts by configuration:

---

6. Results by Ecosystem

Pass counts by ecosystem for each configuration (total per ecosystem: npm/yarn 68, Maven 65, Go 20?, Cargo 20, Python 10):

Key findings:

• Maven/Java tasks are solved most consistently (≥32/65 for GPT-5.5)
• Cargo/Rust tasks are hardest (≤6/20 solved)
• JavaScript tasks are moderate (23–28/68)
• Python and Go are mid-difficulty

---

7. Failure Modes & Effects

---

7. Experimental Results

Overall pass rates (of 203 tasks):

Key Results:

• Best configuration: Codex + GPT-5.5 with 104/203 solved (51.2%)
• Consensus-hard tasks: 77/203 (GPT-5.5), 114/203 (Gemini), 102/203 (Claude Opus 4.8)
• Gemini 3.5 Flash performs notably worse in every matched harness.

---

9. Ecosystem Difficulty

Pass counts by ecosystem (GPT-5.5 + Codex):

Key observations:

• Maven/Java tasks are the most solvable (best: 46/65)
• Cargo/Rust tasks are hardest (best: 6/20) — likely due to strict type/macro systems and fewer agents having strong Rust support
• JavaScript, Go, and Python sit in the middle

---

7. Root-Cause Analysis

We manually adjudicated non-passing runs into nine categories. The dominant failure mode across all configurations is incomplete migration: agents find the right dependency surface but fail to propagate changes through all wrappers, types, helpers, fixtures, and call sites. Example:

"Agents often identify the right dependency-upgrade surface but do not propagate the change through every wrapper, type, helper, or generated file."

Table: Adjudicated non-pass categories (each config column sums to total non-passes)

Consensus-hard tasks: unsolved by all configurations of a given model — 77/203 for GPT-5.5, 114/203 for Gemini, 102/203 for Claude Opus 4.8. The remaining tasks are configuration-asymmetric, meaning some setups solve them and others don't — these are useful for diagnosing agent differences.

---

5. Main Findings

Best configuration: Copilot CLI + GPT-5.5 solves 99/203 tasks (48.8%); still fails more than half.

Biggest failure mode: incomplete migration — the agent finds the right API change but fails to propagate it through wrappers, fixtures, generated files, and call sites.

Ecosystem difficulty varies significantly:

Cargo (Rust) tasks are consistently the hardest; JavaScript/Python tasks also show low pass rates. Maven/Java tasks are comparatively more tractable, likely due to stronger static checking.

---

7. Table of Pass Counts by Configuration (from paper)

Key result: The best configuration solves only 104/203 tasks (≈51%), and the majority of failures come from incomplete migration — agents identify the right dependency surface but fail to propagate changes through wrappers, generated files, fixtures, and call sites.

---

9. Failure Analysis

Consensus-hard vs. asymmetric tasks:

• Consensus-hard (unsolved by all configs with a given LLM): 77/203 for GPT-5.5, 114/203 for Gemini, 102/203 for Claude Opus 4.8.
• Configuration-asymmetric: solvable by some configurations but not others — these are useful for diagnosing which harnesses fail and why.

Primary failure category (excluding runtime exceptions): incomplete migration — agents detect the changed dependency surface but fail to propagate changes through all wrappers, types, helpers, and generated files.

---

9. Conclusions

Dependency-upgrade repair remains largely unsolved for current coding agents. Key findings:

1. Best configuration solves only ~50% of the benchmark (99/203 tasks with Copilot CLI + GPT-5.5 per Table II; the paper reports 104 from the Codex/GPT-5.5 combination).
2. Model scale matters: GPT-5.5 substantially outperforms Gemini 3.5 Flash on matched harnesses.
3. Ecosystem difficulty varies: Maven (Java) tasks show highest pass rates; Cargo (Rust) tasks are hardest.
4. Primary failure mode: incomplete migration — agents find the right surface but fail to propagate changes through wrappers, types, helpers, fixtures, and generated files.
5. No configuration solves consensus-hard tasks; these require deeper semantic reasoning about implicit API breakage.

Held-out Test Patch

The test patch, which contains updated test expectations, is hidden from the agent during evaluationcher; the agent must infer the breakage from build/test output after the update, without seeing the actual fix to the tests.

---

10. Evaluation Protocol

Each task is evaluated by:

1. Applying the manifest patch (dependency version bump) to a clean repository state.
2. Running the agent with a natural-language instruction: “Update the code so tests pass after the dependency bump.”
3. Applying the agent’s generated final patch to the repository.
4. Running the held-out test patch in a containerized environment.
5. Verifying that the agent’s patch solves the task (not just the visible pre-update tests).

---

11. Results (Summary)

Best configuration: Copilot CLI + GPT-5.5: 99/203 tasks solved (48.8%)

---

10. Key Findings

1. Dependency-upgrade repair is largely unsolved. The best configuration solves 99/203 tasks, leaving over half unsolved. Gemini 3.5 Flash performs worse in every matched harness.

2. Incomplete migration is the dominant failure mode. Agents identify the right dependency API but fail to propagate changes through:

   • Wrappers and helper functions
   • Type definitions
   • Fixtures, mocks, and generated files
   • Behavioral/expected-value updates

3. Some tasks are "consensus-hard" (e.g., 104/203 unsolved by all GPT-5.5 configs), while others are configuration-asymmetric, making them useful diagnostics for model/harness debugging.

4. Ecosystem asymmetry:

   • Maven/Java: highest pass rates (up to 46/65), likely because compilers surface explicit breakage.
   • npm/yarn: moderate (up to 28/68), with mixed explicit/implicit breakage.
   • Cargo/Rust: very hard (2–6/20), often due to more complex trait/type system interactions.

---

10. Related Work

• Agent benchmarks (e.g., SWE-bench, etc.): focused on issue resolution and repository understanding; DEPEND-REPAIR focuses specifically on dependency upgrades and hidden breakage.
• Dependency-management tools: Dependabot etc. only address supply-chain issues; they do not repair source code.
• Library migration studies: prior work focuses on API evolution, but not on agent-driven repair with hidden tests.
• Static analysis / ML repair: existing repair tools do not target the unique dependency-bump-after-failure mode captured here.

---

Table: Adjudicated Failure Categories (Table V)

(Selection of rows; for space, the most representative row is shown.)

---

10. Key Takeaways & Conclusions

1. Dependency-upgrade repair is still largely unsolved for coding agents: the best configuration (Copilot CLI + GPT-5.5) achieves only 51.2% (104/203) task success.
2. Ecosystems differ sharply: Maven/Java tasks are easier; Cargo and npm/yarn are harder, due to weaker tooling guidance and more dynamic semantics.
3. The main failure mode is incomplete migration — agents apply the right change locally but fail to propagate it through wrappers, fixtures, types, and call sites.
4. Model choice matters more than harness for these tasks; GPT-5.5 > Claude Opus 4.8 > Gemini Flash on similar harnesses.
5. Consensus-hard tasks (unsolved by all configurations) indicate areas where current agents most lack capability.

---

References (Selected)

• Dependabot: https://github.com/dependabot
• SWE-bench: https://github.com/SWE-bench/SWE-bench
• Terminal-Bench, RepoBench, Commit0, Aider, etc.
• Nix, Bootstrap, Docker containerization for reproducible verifier environments.
• Dependabot version-bump PRs on GitHub as the primary seed data.

---

Key takeaway: Current coding agents solve at most ~50% of real-world dependency-upgrade repair tasks; the main failure mode is incomplete migration — identifying the right upgrade surface but failing to propagate changes through wrappers, types, fixtures, and generated files. Cargo/Rust tasks are especially hard (≤6/20 solved), while Maven/Java tasks are comparatively tractable. Benchmark remains open: http://github.com/xxx/benchmark (anonymized for review).

Abstracted and summarized from the paper "Update from Hell: Can Coding Agents Survive Hidden Breakage in Dependency Upgrades?" by Zijian Luo et al.