# SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

> SWE Refactor Bench shows frontier agents fail 95% of whole-repository migrations, with even perfect test passes undermined by hidden behavioral differences.

- **Source:** [arXiv](https://arxiv.org/abs/2608.23564)
- **Published:** 2026-08-29
- **Permalink:** https://picx.dev/p/VEmtrD
- **Whiteboard:** https://picx.dev/p/VEmtrD/image

## Summary

# SWE Refactor Bench: Can Coding Agents Complete Long-Horizon, Whole-Repository Stack Migrations?

## Summary

- **Introduces SWE Refactor Bench**, a benchmark of 20 whole-repository migration tasks drawn from real open-source infrastructure (e.g., SQLite, zlib, libsodium, GraphHopper), covering four debt classes: language, framework, platform, and build toolchain rewrites.
- **Identifies "Blindness"** in existing benchmarks: behaviour-only evaluation awards full credit to submissions that return the repository unchanged, because the original repository already passes all behavioural tests.
- **Proposes a three-stage evaluation protocol**: Migration Audit (veto on whether migration occurred), Behavioural Tests (130,118 fixed checks recorded from the original), and Agentic Verification (six independent coding agents search for hidden behavioural differences).
- **Reports results across 520 runs** of 8 frontier models and 26 configurations: only 28 runs (5.4%) pass all three stages; 13 of 20 tasks are never solved; the best model scores 47.0/100.
- **Demonstrates that migration completeness and behavioural correctness are distinct abilities**, often satisfied by different runs, and that even perfect fixed-test performance is fragile under agentic verification.

---

## 1. Introduction

### 1.1 The Problem: Evaluating Behaviour-Preserving Migration

Modern software accumulates technical debt over decades. Migrating a repository onto a new technology stack (new language, framework, platform, or build toolchain) is expensive and largely manual. As coding agents become more capable, a natural question arises: *can they autonomously perform such migrations?*

Existing benchmarks (e.g., SWE-bench) evaluate agents on "red to green" tasks: a test fails before the patch and passes after. This signal is unavailable for behaviour-preserving migration, because the starting state is already green. A repository handed back unchanged passes any behavioural test suite by construction.

### 1.2 The "Blindness" Failure Mode

The paper defines **Blindness** as follows:

> A behaviour-only evaluator awards full credit without establishing that the migration occurred. Since the preservation condition is defined relative to the original repository, setting the submission equal to the original (the empty diff) earns full marks on any behavioural suite while satisfying not one clause of the migration condition.

This is not reward hacking in the usual sense (no rule is circumvented); rather, the evaluation instrument itself cannot see whether the migration happened.

### 1.3 Contributions

1. **Blindness identified**: behaviour-only evaluation cannot distinguish "nothing was broken" from "something was changed".
2. **SWE Refactor Bench introduced**: 20 tasks, each requiring a real open-source repository to be migrated in its entirety onto another stack.
3. **Three-stage adversarial evaluation protocol**:
   - Stage I: Migration Audit (hard veto)
   - Stage II: Behavioural Tests (all-or-nothing)
   - Stage III: Agentic Verification (per-verifier scoring)
4. **Empirical findings**: current frontier models rarely deliver migrations that are complete, behaviour-preserving, and robust to agentic verification.

---

## 2. Benchmark Design

### 2.1 Task Formulation

A behaviour-preserving migration task is a tuple:

$$\tau = (R_A, \Sigma_A \to \Sigma_B, \mathcal{O}, \mathcal{I}, E, B) \tag{1}$$

- **$R_A$** (State A): a real open-source repository at one commit, in buildable and working condition.
- **$\Sigma_A \to \Sigma_B$**: the stack change (language, framework, platform, or build toolchain).
- **$\mathcal{O}$**: the observable interface of the artifact (process output, exit status, exported symbols, installation tree, HTTP responses).
- **$\mathcal{I}$**: the instruction given to the agent.
- **$E$**: an offline image (no network).
- **$B$**: the time budget.

The agent's input is $(R_A, \mathcal{I}, E, B)$; its output is the working tree $R_S$. $R_S$ solves $\tau$ exactly when:

1. **Migration condition**: $\Sigma_B$ builds the delivered artifact, and $\Sigma_A$ is absent from the repository and the build closure.
2. **Preservation condition**: $\mathcal{O}(R_S) = \mathcal{O}(R_A)$.

### 2.2 Why Behavioural Tests Fail

A behavioural test suite $T$ is a finite set of observations; the score it gives is the fraction passed. Since $T \subseteq \mathcal{O}$ and $\mathcal{O}(R_A)$ passes all of $T$ by construction, setting $R_S = R_A$ earns full marks while satisfying no clause of the migration condition. Enlarging $T$ does not help, because every case the migrated repository must pass is a case the original already passes.

### 2.3 Task Selection: Debt First, Repository Second

Three admission requirements for each task:

1. **The old stack must be load-bearing**: removing it reaches the design (e.g., cmark's hand-written C parser state machine moved to Rust requires redesigning ownership, not adding an FFI layer).
2. **The observable interface must have real downstream consumers**: a C ABI, an HTTP API, an installation tree.
3. **There must be a runnable reference**: State A can be built and executed repeatedly, enabling differential testing.

### 2.4 Four Kinds of Technical Debt

| Debt Class | Representative Migrations | Tasks | Source LoC | Budget (h) | Criteria (Stage I) | Modules (Stage II) | Checks (Stage II) |
|---|---|---|---|---|---|---|---|
| Language | C → Rust, C → Java, Go → Zig | 7 | 4.4k–39.8k | 12–30 | 60 | 117 | 59,771 |
| Framework | Flask → Starlette, Gin → chi, Vue → React | 7 | 0.8k–94.8k | 6–16 | 39 | 79 | 55,852 |
| Platform | POSIX → wasm32-wasi, Node → V8 realm | 3 | 16.0k–358.0k | 6–10 | 19 | 31 | 6,725 |
| Build toolchain | Autotools → CMake, Maven → Gradle | 3 | 19.0k–78.8k | 6 | 18 | 37 | 7,770 |
| **Total** | 20 upstream projects | **20** | **867,062** | **262** | **136** | **264** | **130,118** |

*Table 1: Composition of the SWE Refactor Bench task set.*

### 2.5 The Three-Stage Protocol

The agent's workspace is isolated from the evaluation environment. The three stages are applied in order; each can end the run (except that the fixed suite is run on every submission, vetoed or not, to count blindness).

#### Stage I: Migration Audit

- **Question**: Did the migration actually happen? Has the old stack disappeared from the repository and the build?
- **Mechanism**: Criteria written as prompt questions about this repository, answered by a model reading both source trees. Every failing verdict must cite re-checkable evidence. Each criterion is judged three times independently; majority taken.
- **Rationale**: A behavioural suite gives full marks to an untouched repository, so "was the work done at all" must be checked separately.

#### Stage II: Behavioural Tests

- **Question**: Did behaviour survive unchanged?
- **Mechanism**: Tests built from State A (same calls run against the original; recorded output becomes expected answer). 130,118 checks across the suite. A single wrong check scores zero; only a clean sweep opens Stage III.
- **Rationale**: A drop-in replacement must be correct every time; a library wrong once in a thousand calls is not a drop-in replacement.

#### Stage III: Agentic Verification

- **Question**: What did we not think of in advance?
- **Mechanism**: Six coding agents, one hour each, each holding both source trees, looking for something the original does that the submission does not. Five probe assigned directions (C ABI, routes, installation tree, etc.); the sixth is unrestricted. They must hand in an executable test case that passes on State A and fails on the submission; it must first go green against the reference and then reproduce three times.
- **Rationale**: This is differential testing in agentic form. Only an executable counterexample turns "a problem nobody thought of" into a re-checkable fact.

### 2.6 Scoring

Let $g$ be the Migration Audit verdict, $r_i$ the pass rate of module $i$ in Behavioural Tests, and $s \in \{0, \ldots, 6\}$ the number of verifiers that failed to produce a counterexample. The score is:

$$S(\tau) = \underbrace{\mathbf{1}[g = \text{pass}]}_{\text{Stage I: veto}} \cdot \underbrace{\mathbf{1}[r_i = 1 \forall i]}_{\text{Stage II: all or nothing}} \cdot \left(0.4 + \underbrace{0.6 \cdot \frac{s}{6}}_{\text{Stage III: per verifier}}\right) \in \{0\} \cup [0.4, 1] \tag{4}$$

- **Stage I multiplies** because without the migration the task was not done at all.
- **Stage II is all or nothing** because a drop-in replacement must be correct every time.
- **Stage III carries 0.6** because it looks at a residue no fixed suite can see; it is scored linearly because "not broken" is a matter of evidence strength, not a demonstration of equivalence.

---

## 3. Experiments

### 3.1 Setup

- **Models evaluated**: claude-opus-5, claude-sonnet-5, gpt-5.6-luna, gpt-5.6-sol, kimik3, qwen3.8-max, dsv4-flash, glm-5.2.
- **Harnesses**: GPT-series use Codex; all others use Claude Code.
- **Scale**: 8 models, 26 configurations, 520 scored runs (each model runs all 20 tasks).
- **Metrics**: Migrated (Stage I passed), All tests pass (Stage II perfect), Accepted (all three stages), Broken (share of runs that passed Stages I–II but were defeated by at least one verifier).

### 3.2 The Funnel: Where Runs Stop

From Table 2 (aggregated across all 520 runs):

- **340 runs (65.4%)** passed Stage I.
- **118 runs (22.7%)** passed every fixed check.
- **88 runs** did both and reached Stage III.
- **28 runs (5.4%)** survived all six verifiers.
- **13 of 20 tasks** were never solved by any model.
- Mean score over all runs: **13.44/100**; over the 88 that reached Stage III: **79.43/100**.

### 3.3 Analysis of Agent Behaviour

#### 3.3.1 "Getting the Code Right" and "Getting the Migration Done" Are Two Different Abilities

- **30 runs** never migrated and kept every fixed check (Blindness), spread over 7 of 8 models.
- **252 runs** completed the migration and broke behaviour doing it.
- Neither stage can stand in for the other: Stage II alone would reward doing nothing; Stage I alone would reward doing damage.

**Per-task example**: On lang04 (acorn, JavaScript → Rust), 20/26 runs passed Stage I, yet not one passed every fixed check. On lang01 (cmark, C → Rust), only 6 runs passed Stage I, yet 5 passed every fixed check—and all five are Blindness: the Rust is a transliteration that reproduces the original's control flow statement for statement, never redesigning ownership.

#### 3.3.2 The Last 1%: Agents Cannot Deliver a Perfect Migration

Among the 340 runs that passed Stage I:
- 91% get the fixed suite past half.
- 58% reach 99%.
- 36% reach 99.9%.
- **Only 26% make no error at all**.

The final step alone eliminates 35 of the 123 runs that reached 99.9%. 140 runs land in [99%, 100%), missing a median of 12.5 checks; 18 miss exactly one.

**Concrete examples of single-check failures**:
- **fw03 (conduit, Vue → React)**: four different models all ended at 21768/21769, failing the same check: the original uses hash routing ($/$#), while the React version stays at $/$, breaking every bookmark and shared link.
- **build03 (PyCryptodome, setuptools → Meson)**: five different models all ended at 380/381: the METADATA long description in the built wheel is 0 characters, so the PyPI project page is blank.

**Even perfect fixed-test performance is not enough**: Among the 88 submissions that passed every fixed check, only 28 survived all six verifiers; the other 60 (68.2%) had a counterexample found within the hour. The median time to a counterexample is 17.0 minutes; the median survival time is 32.8 minutes.

#### 3.3.3 Capability Profiles Differ Across Migration Categories

| Category | Runs | Stage I pass rate | Stage II pass rate (given Stage I) | Stage III survival (given Stage II) | Final acceptance | Mean score |
|---|---|---|---|---|---|---|
| Build toolchain | 78 | 80.8% | 54.0% | 17.6% | 6 | 31.4 |
| Platform port | 78 | 57.7% | 37.8% | 23.5% | 4 | 17.2 |
| Framework | 182 | 72.5% | 18.9% | 56.0% | 14 | 12.0 |
| Language | 182 | 54.9% | 12.0% | 33.3% | 4 | 5.6 |
| **All** | **520** | **65.4%** | **25.9%** | **31.8%** | **28** | **13.4** |

*Table 5: Conditional pass rates by category.*

Agents do not perform best on one category throughout the pipeline. They achieve their highest Stage I and Stage II pass rates on build toolchain rewrites but their lowest Stage III survival rate on the same category. On framework rewrites, they achieve only 18.9% at Stage II but the highest Stage III survival rate (56.0%).

---

## 4. Analysis of Benchmark Validity

### 4.1 Stage I Decisions Agree Across Models and Humans

**Judge stability**: The judge (gpt-5.6-sol) is sampled three times per criterion. Of 3,536 criterion verdicts, 96.3% had all three samples agree, and only 131 split 2:1. However, because Stage I requires every criterion to pass, a single 2:1 disagreement can decide a run: 35 of 340 runs passed with at least one criterion passed only 2:1. Without majority voting, the number passing Stage I would be 305 rather than 340.

**Human agreement**: Two independent software-engineering researchers judged 156 runs of 6 tasks. Judge and human agree 89.7% of the time (κ = 0.795). Of the 16 disagreements, 14 are the judge being too strict and only 2 too lenient. Following all 16 through the later stages changes almost nothing: 12 too-strict cases would have failed Stage II anyway; 1 too-lenient case reached Stage III where four of six verifiers constructed counterexamples. The judge's error has a direction, and that direction makes points harder to earn, not easier.

### 4.2 Agentic Verification Finds Failures Beyond Fixed Tests

**Example from Stage III**: On fw04 (ChartMuseum, Gin → chi), POST /api/charts dispatches between multipart-form and raw-body upload handlers. The original truncates Content-Type at the first space or semicolon; the migrated version truncates only at the semicolon. A request with one extra space before the boundary reaches a different handler on each side—an implementation detail of Gin unknown when the suite was written.

**Other examples**: lang05 (go-yaml, Go → Zig) where go-yaml accepts a comma as decimal separator (2001-12-14T21:59:43,10Z is a !!timestamp on the original, !!str in the Zig version); pf01 (SQLite, POSIX → WASI) where after checkpointing a WAL database and switching back to DELETE mode, native SQLite removes both -wal and -shm files while the WASI version removes only -wal.

### 4.3 Six Verifiers: Strength and Diversity Both Matter

**Finding 1: Strong verifier model matters more than configuration.** The two claude-opus-5 verifiers have break rates of 55.7% and 53.4%; the other four between 21.6% and 26.1%. Changing prompt and effort within one model moves the break rate by ~2 points; changing the model moves it by ~30 points. If the two strongest verifiers were retired, the remaining four would accept 46 submissions instead of 28.

**Finding 2: Verifiers are complementary.** Counting exclusive breaks (where one verifier alone found a counterexample), the two opus verifiers have 5 and 2, and the four weaker ones have 4 between them. Even the lowest break-rate verifier rejected a submission the other five let through.

**Finding 3: Models do not spare their own family's work.** Verifiers broke 33.9% of submissions from their own family vs. 34.3% of everyone else's. The apparent counter-evidence (opus verifiers break 40.8% of opus-authored submissions vs. 65.0% of others) fails its own control: non-opus verifiers on the same submissions are at 17.1% vs. 29.1%, a drop by the same factor. What separates the two is submission quality, not collusion.

---

## 5. Related Work

### 5.1 Repository-Level Coding Benchmarks: "Red to Green" Signal

SWE-bench established the paradigm: a real repository, a real issue, and a test that fails before the patch and passes after. Extensions include other languages, continuously refreshed task streams, building a whole library from scratch, and writing tests rather than patches. The criterion fits every such task type because the starting state is red.

**SWE Refactor Bench's tasks have no such jump available** because the starting state is already green—a property of behaviour-preserving evolution, not a defect.

### 5.2 Code Migration and Reward Hacking

Behaviour-preserving rewriting has a long history (TransCoder at function level; later whole-repository work; C-to-Rust benchmarks; language-version and API moves). The scoring is always the same: run a test suite, count what passes. For whole-repository stack changes, this is insufficient: a repository handed back as-is still turns any fixed suite green.

This is **not reward hacking** in the usual sense: a repository handed back as-is circumvents no check; it earns full marks by the rules, and the rules simply cannot see whether the migration happened. Since the hole is not in how tight the tests are, no amount of extra tests will close it; the only remedy is a second check outside behaviour, holding a veto.

### 5.3 Model-Based Judging and Differential Testing

- **Stage I** uses a model as judge (standard practice), with known failure modes (position and verbosity bias, self-preference) mitigated by narrow questions, cited evidence, and three independent samples.
- **Stage III** is differential testing: with the reference as the other side, this is how compilers have been checked at scale. What a verifier must submit is not an opinion but evidence that runs.
- **Formal verification** is not available because cross-language whole-repository rewrites do not offer two sides whose semantics can be related.

---

## 6. Conclusion

**SWE Refactor Bench** provides 20 long-horizon whole-repository migrations with a three-stage evaluation protocol that does not rely on behaviour alone:

1. **Migration Audit** decides whether the migration actually happened and holds a veto.
2. **Behavioural Tests** decide whether behaviour is unchanged down to the last observation (130,118 fixed checks recorded from the original).
3. **Agentic Verification** sends six coding agents, one hour each, to look for differences the fixed tests may have missed, accepting nothing but an executable counterexample.

**Results**: Across 520 evaluations on 8 frontier models and 26 configurations, only 28 (5.4%) passed all three stages; 13 of 20 tasks were solved by nobody.

**Key findings**:
- "Getting the migration done" and "not breaking anything" are two different abilities, missed in opposite directions.
- Even when the migration was genuinely done, only 26% of runs passed every fixed check, and two thirds of those still had a counterexample found against them within the hour.

**Conclusion**: There is a long way to go before an agent can complete a whole-repository migration that is genuinely deliverable. Progress will be measured not by how much code the agent wrote, but by whether, once it is done, the system is still the same system.

---

_Markdown view of https://picx.dev/p/VEmtrD, served by PicX — AI-generated visual whiteboard summaries of research papers._