SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

Summary (Overview)

• Novel benchmark: SWE-Shift evaluates coding agents under mid-execution changes to tasks and environments, addressing a gap in static repository-level benchmarks like SWE-bench.
• Matched continuation protocol: From the same saved execution state, agents resume in two matched continuations—Clean (no intervention) and Shifted (task-relevant change)—enabling controlled comparison of success under static vs. changing conditions.
• Comprehensive coverage: 120 scenarios across 12 change families, each evaluated at 9 intervention positions (T10–T90), yielding 1,080 scheduled evaluation cells per system.
• Key finding: All 17 evaluated coding-agent systems show lower Shifted than Clean success. GPT-6 Astra drops from 67.35% to 24.79%, with only 20.34% paired success.
• Shared failure points: Tool/session changes (F06) and stale evidence (F11) yield zero paired success across all 17 systems, revealing systematic weaknesses in state recovery and evidence freshness.

Introduction and Theoretical Foundation

Background and Motivation

Repository-level benchmarks such as SWE-bench have enabled evaluation of coding agents on realistic software engineering tasks—requiring agents to understand issues, navigate repositories, modify executable code, and satisfy tests. However, these benchmarks assume that once execution begins, the task and environment remain static. Real software development is dynamic: code may be edited, dependencies may change, runtime state may evolve, and requirements may be updated while an agent is still working.

The central research question is:

Can a coding agent continue to succeed when task-relevant changes occur during execution?

This question cannot be answered by evaluating agents on changed tasks alone, because failure after a change could stem from either an inability to solve the underlying task or disruption of an otherwise successful execution. SWE-Shift addresses this through matched Clean and Shifted continuations that share the same prior actions, workspace, and observable history.

Theoretical Foundation

The benchmark builds on the insight that robustness to mid-execution change captures behavior that static task success alone does not reveal. A change must be both relevant (ignoring it can invalidate an otherwise successful continuation) and recoverable (a valid way to complete the task remains after the change). Success is judged against requirements that apply after the update, including preservation and evidence requirements that original task tests may not cover.

Related Work Comparison

Methodology

Designing Task-Relevant Changes

SWE-Shift builds 120 change scenarios on 23 task-root snapshots from 17 repositories, drawn from SWE-bench Verified, Multi-SWE-bench, and SWE-bench Multilingual. Each scenario combines an existing repository task with a task-relevant change introduced during execution.

Scenario structure—four elements are specified:

1. The initial repository task
2. An authorized mid-execution change
3. A way for the agent to encounter/discover the change
4. Executable checks for a correct continuation

12 change families organized into three groups:

• World Shift (F01–F07): Changes to executable state or dependencies (user/collaborator edits, interface changes, runtime/session state, external effects)
• Goal Interaction (F08–F10): Newly disclosed requirements, corrective feedback, revised goals
• Observation Shift (F11–F12): Stale or misbound evidence

Matched Continuation Protocol

For each system, scenario, and intervention position:

1. The agent executes a single shared prefix
2. At the scheduled checkpoint, the runner saves agent-visible history, workspace, controller state, tools, system settings, and execution counters
3. The saved state is restored twice:
   • Clean continuation: proceeds without intervention
   • Shifted continuation: receives the authorized update at the fork

Intervention timing: Nine progressively later positions (T10–T90) constructed from an eligible prefix of a reference solution, ending before the first action whose correctness depends on the update.

Success Criteria and Metrics

A continuation succeeds only if it satisfies every check applicable under its ending conditions. Checks include: task, preservation, recovery, authority, effect, evidence, budget, and terminal-integrity requirements.

Core metrics (for model $m$, scenario $i$, intervention position $t$):

Let $C_{mit} \in \{0,1\}$ and $S_{mit} \in \{0,1\}$ denote verified success in Clean and Shifted continuations.

Ordinary paired credit requires success in both continuations:

$$R_{mit} = C_{mit} S_{mit}. \tag{1}$$

Cell score with early-completion credit $P_{mit}$:

$$K_{mit} = \max\{R_{mit}, P_{mit}\}. \tag{2}$$

Aggregated metrics (with normalized scenario weights $w_i$):

$$\text{Clean}_{m} = \frac{1}{9}\sum_{t\in\mathcal{T}}\sum_{i}w_{i}C_{mit}, \tag{3}$$ $$\text{Shifted}_{m} = \frac{1}{9}\sum_{t\in\mathcal{T}}\sum_{i}w_{i}S_{mit}, \tag{4}$$ $$\mathrm{PSR}_{m} = \frac{1}{9}\sum_{t\in\mathcal{T}}\sum_{i}w_{i}K_{mit}. \tag{5}$$

Conditional Retention Rate (CRR):

$$\mathrm{CRR}_{m} = \frac{\sum_{t\in\mathcal{T}}\sum_{i}w_{i}R_{mit}}{\sum_{t\in\mathcal{T}}\sum_{i}w_{i}C_{mit}}. \tag{6}$$

Empirical Validation / Results

Experimental Setup

17 coding-agent systems evaluated on the same 120 scenarios and 9 intervention positions, with post-fork token budget $B = 1{,}000{,}000$ tokens per continuation. Provider-hosted systems use native agent applications (Codex app for GPT models, Claude Code app for Claude models) at xhigh reasoning. Open-weight systems served locally on 32 NVIDIA B300 GPUs.

RQ1: Does coding capability survive a workspace change?

All 17 systems show reduced success after changes. Clean success ranges from 10.71% to 67.35%, compared to 1.16–24.79% for Shifted and 0.90–20.34% for PSR.

Higher Clean success does not imply higher retention. Kimi-K2.6 achieves 34.63% CRR despite only 20.50% Clean success.

RQ2: Which changes are persistent failure points?

Shared weaknesses (zero PSR across all/most systems):

• F06 (Tool/session changes): 0.00% PSR for 17/17 systems
• F11 (Stale evidence): 0.00% PSR for 17/17 systems
• F05 (Runtime/resource changes): 16/17 systems score zero
• F08 (Disclosure changes): 13/17 systems score zero

Selective strengths:

• Kimi-K3: 63.50% on F01 (User implementation)
• GPT-6 Astra: 57.30% on F04 (Indirect collaborator), 52.40% on F09 (Corrective feedback)
• Claude Fable 5.1: 60.80% on F01, 16.20% on F08

RQ3: Does timing matter?

Late-position PSR can be lower or higher depending on system:

• Decreasing: GPT-6 Astra (−20.05 pts), Claude Fable 5.1 (−15.43), GPT-5.6 Sol (−12.53), Claude Opus 5 (−11.93)
• Increasing: GLM-5.3-Flash (+2.97), Kimi-K2.6 (+2.20), DeepSeek-V4-Flash (+1.60)

RQ4: What do task tests miss after an update?

Three qualitative cases from the audit set:

1. Django: Tests pass after collaborator's change is removed—preservation check fails
2. Astropy: Compensation succeeds but workspace-preservation check fails
3. scikit-learn: Original task checks pass but fresh evidence observation is missing

Diagnostic Controls

State refresh and retained history: Adding history reset yields the largest incremental gain in 4/5 systems. Complete intervention adds 5.98–8.71 PSR points, but final PSR remains only 10.89–17.54%.

Post-fork token budget: Each doubling up to 4B adds 0.79–2.67 points; 4B→8B adds only 0.12–0.28 points; 8B→16B changes PSR by −0.09 to +0.15 points. PSR at 16B remains 7.15–11.98%.

Theoretical and Practical Implications

Implications for Evaluation Design

1. Static success is insufficient: Success on unchanged tasks can coexist with failures to preserve work, recover state, or refresh evidence after an update. Evaluation must jointly assess task completion, preservation, recovery, and evidence validity.

2. Matched comparisons are essential: The Clean continuation provides a matched baseline that separates failures on the underlying task from losses of successful continuations after a change.

3. Multiple intervention positions matter: Timing profiles differ across systems, suggesting that robustness at one point does not predict robustness at another.

Implications for Agent Development

1. State recovery is a critical weakness: Tool/session changes and stale evidence are universal failure points, suggesting agents need better mechanisms for detecting and adapting to state changes.

2. Preservation awareness is lacking: Agents can pass original tests while removing valid collaborator work, indicating a need for explicit preservation objectives.

3. Evidence freshness checking is underdeveloped: Agents often rely on outdated observations without re-querying current state.

Conclusion

SWE-Shift addresses the gap in evaluating coding agents' ability to handle mid-execution changes through task-relevant, recoverable interventions and matched Clean and Shifted continuations from the same saved execution state. The benchmark's success criteria jointly assess task completion, preservation of valid work, recovery, and evidence validity.

Evaluation across 17 coding-agent systems reveals that success under unchanged conditions often does not survive an update. Family-level and trajectory analyses identify recurring failures in state recovery, preservation, and evidence use that original task tests can miss. SWE-Shift provides a common testbed for measuring robustness to change, diagnosing failures, and assessing whether new methods improve agents' ability to work reliably in evolving software environments.

Future directions: Evaluating repeated changes in natural software-development workflows, expanding diagnostic controls to more systems, and exploring adaptive intervention timing based on execution stage rather than fixed positions.