Summary (Overview)

• This paper presents the first systematic robustness evaluation of repository-level code agents against semantics-preserving transformations (SPTs), extending prior work that focused on single-turn, non-agentic settings.
• The authors introduce a random variant sampler that applies 14 SPTs (control-flow rewrites, dead-code injection, identifier renaming) to produce semantically equivalent perturbed repositories, without feedback from model outcomes (non-adversarial by design).
• Evaluating 2 agent scaffolds × 4 frontier models × 2 benchmarks (16 configurations) on 54 instances from SWE-bench Verified and SWE-bench Pro, they find mean resolve-rate degradations up to 6.7 percentage points, with statistically significant degradations in 6 of 16 configurations.
• Key finding: robustness is a "jagged frontier" — a joint property of model, scaffold, and workload, not of the model alone. Model robustness rankings do not transfer across scaffolds or benchmarks.
• Perturbations also inflate agent effort — increasing step counts and token costs by up to 9.9% and 22.9% respectively, even when resolve rates are largely unchanged.

---

Introduction and Theoretical Foundation

Background and Motivation

AI coding tools have moved from research prototypes to mainstream use, with developer surveys reporting majority adoption. Their progress is measured on benchmarks like SWE-bench Verified and SWE-bench Pro. However, these benchmarks do not account for the known tendency of neural models to learn shortcuts (Geirhos et al. 2020) and be susceptible to small input perturbations.

Two concerns motivate the work:

1. If agent behavior shifts with superficial changes to surrounding code, benchmark numbers may overstate deployment reliability.
2. Such shifts would suggest models rely on shallow syntactic patterns rather than program semantic understanding.

Theoretical Foundation

The paper builds on prior work showing code LLMs are brittle to semantics-preserving perturbations in single-turn settings (Yefet et al. 2020; Ramakrishnan et al. 2022; Wang et al. 2023). However, repository-level code agents — which interact with the LLM over many turns, localize faults, navigate entire repositories, and synthesize patches — have received little systematic study.

The authors frame their contribution as computing lower bounds for perturbation impact:

"A feedback-guided adversary can only do more damage. These bounds can inform developer choices about whether and which code agents to use in deployment."

---

Methodology

Definition of Semantics-Preserving Transformations

A transformation $T$ is semantics-preserving if $T(P)$ produces the same observable behavior as $P$ on every input:

• Return the same value or raise the same exception
• Produce the same externally observable effects
• Both halt or both diverge

Operationally, this is validated through functional test-suite equivalence: $T(P)$ yields the same per-test outcome as $P$ across the project's test suite.

Transformation Catalog (14 SPTs)

Two keyword-bound SPTs (Dead String Assignment, Dead Method Injection) require binding to a target keyword from the issue description. These plant "decoys" that surface when the agent searches the repository for that term, testing whether the agent can distinguish decoys from genuine sites of interest.

Validation

SPTs are validated empirically via differential testing against test suites of three projects: SymPy (12,994 tests), sqlfluff (10,060 tests), and xarray (19,917 tests). Every test retained its outcome under all 14 transformations across all three projects.

Random Variant Sampler (Algorithm 1)

For each variant, the sampler makes four random decisions:

1. Which transformations? A subset of $N_t$ transformations drawn uniformly from the catalog $\mathcal{T}$.
2. Which files? For each transformation, an inclusion probability $p_{\text{file}} \sim U(0,1)$ is drawn; each file is included with that probability. Files modified by the gold patch ($F_{\text{gold}}$) are always included — ensuring every variant targets the solution-relevant region.
3. Which sites? Within each included file, a fraction $\phi$ of candidate sites is chosen uniformly.
4. Which keywords? Up to $N_k$ targets drawn uniformly from candidates extracted from the issue description via an LLM call.

Hyperparameters: $N=20$ (variants per instance), $N_t=3$ (transformations per variant), $N_k=5$ (max keywords), $\phi=0.7$ (fraction of candidates transformed), $N_f=10$ (max files per keyword-bound transformation).

The $N_f$ cap is a practical necessity: agents frequently detected and reverted perturbations (e.g., via git reset) when variants contained too many decoys.

Experimental Protocol

For each instance:

• $N=20$ runs on the unperturbed seed
• One run on each of $N=20$ sampled variants
• Every run executes in a freshly provisioned, isolated environment

Design rationale for one run per variant: With budget $R$ and split $R = NK$ (N variants, K runs each), variance is:

$$\text{Var}(\hat{\mu}) = (K\bar{\sigma}^2 + \nu)/R$$

which strictly increases in $K$ whenever $\bar{\sigma}^2 > 0$. A run on a fresh variant samples both variant and agent randomness; re-running a variant resamples only the agent. Hence $K=1$.

Metrics

Degradation (primary metric):

$$\Delta(i) = r_0(i) - r_p(i)$$

where $r_0(i)$ is the baseline resolve rate (fraction of 20 unperturbed runs resolving instance $i$) and $r_p(i)$ is the perturbed resolve rate.

Effort metrics (relative change in steps and cost):

$$\delta_{\text{step}}(i) = \frac{\bar{s}_p(i) - \bar{s}_0(i)}{\bar{s}_0(i)} \times 100\%$$

with $\delta_{\text{cost}}(i)$ defined analogously from token costs.

Statistical protocol: Fixed-population inference with 95% bootstrap percentile intervals ($B = 20{,}000$ resamples); per-instance degradation uses Newcombe intervals.

---

Empirical Validation / Results

RQ1: Degradation of Agent Performance

Key findings:

1. Robustness rankings do not transfer. Perturbation reduces resolve rates in 13 of 16 configurations; intervals exclude zero in 6 of 16. No scaffold-model pair is significant on both benchmarks.

2. Jagged robustness frontier: On SWE-bench Verified, Qwen is the most robust under mini-SWE agent (0.2 points degradation) yet the most brittle under OpenCode (5.5 points). MiniMax moves the other way (2.7 points under mini-SWE agent → 0.5 under OpenCode). Opus under mini-SWE agent degrades 1.8 points on Verified but 6.7 points on Pro — the largest drop in the study.

3. The simpler scaffold is consistently more robust: Averaged over models, mini-SWE agent degrades less than OpenCode on both benchmarks (1.34 vs. 1.88 points on Verified; 1.88 vs. 3.65 on Pro).

4. Degradation is concentrated: Only 14 of 432 instance-configuration items exclude zero in their Newcombe intervals. Three instances alone account for roughly two-thirds of the 6.7-point mean degradation of mini-SWE agent with Opus on SWE-bench Pro.

RQ2: Effort Induced by Perturbation

• Cost rises in all 8 configurations on SWE-bench Verified (4.0% to 22.9%), with intervals excluding zero in every one.
• On SWE-bench Pro, increments are concentrated in OpenCode (8.8–12.6% more cost, all significant); mini-SWE agent configurations sit near zero.
• Cost per step rises in 12 of 16 configurations — the agent consumes more context per turn, not just more turns. Step overhead never exceeds 9.9% while cost overhead reaches 22.9%.
• Opus compresses its trajectory: Under mini-SWE agent on Verified, Opus takes fewer steps on every instance (mean −19.8%) but output tokens per step more than double (median +101%), yielding net +30.4% cost per step and +4.0% total cost.

RQ3: Observed Failure Patterns

Manual analysis of 20 trajectories revealed four patterns:

1. Detecting and reverting SPTs: Agents sometimes recognize "obfuscated" code, check commit history, and revert the repository before making final changes.
2. Degradation of code localization: SPTs dilute grep results or make core files lose credibility — e.g., dead string assignments with the same keyword turn up in searches.
3. Editing/"fixing" perturbed code: Agents occasionally simplify perturbed code during patching, increasing patch volume and failure risk.
4. Corrupted patch validation: An agent misattributed a test failure to a split string literal, reasoned it wasn't its fault, and submitted an incorrect patch.

---

Theoretical and Practical Implications

Theoretical Implications

• Robustness is not a model property alone. The "jagged frontier" demonstrates that robustness emerges from the interaction of model, scaffold, and workload. This challenges the practice of ranking models by robustness on a single benchmark.
• Shortcut learning extends to multi-turn agents. The results confirm that even top frontier models rely on shallow syntactic patterns, echoing Geirhos et al.'s shortcut learning framework and Dziri et al.'s evidence that transformers linearize compositional reasoning.
• Scaffold interaction is a new dimension absent from prior robustness work. The same model can be most robust under one scaffold and most brittle under another.

Practical Implications

• Benchmark numbers may overstate deployment reliability. A practitioner who picks a model for robustness under one scaffold/benchmark may get the opposite outcome after switching.
• Outcome-only views understate perturbation impact. Even when resolve rates are unchanged, agents expend more effort (steps, tokens, cost), increasing operational costs.
• The simpler scaffold (mini-SWE agent) is more robust — a useful heuristic for deployment choices, though absolute capability is benchmark-dependent.
• The non-adversarial, randomized perturbation approach provides lower bounds on impact; feedback-guided adversaries can only do more damage.

---

Conclusion

The paper contributes:

1. A library of 14 semantics-preserving transformations for code repositories and a randomized variant sampler that draws semantically equivalent variants without agent feedback.
2. An experimental methodology isolating perturbation effects from intrinsic LLM stochasticity through paired seed-and-variant runs.
3. The first systematic robustness evaluation of repository-level code agents, revealing a jagged robustness frontier across models, scaffolds, and repositories.

Key takeaways:

• Localized, non-adversarial perturbations cause small but statistically significant degradations in most configurations (up to 6.7 percentage points).
• Perturbations raise agent effort (steps and cost) even when outcomes are unchanged.
• No single model ranking by robustness holds across scaffolds or benchmarks.
• The simpler scaffold is consistently more robust.

Future directions implied by the work include robustness-enhancing designs for models and agentic scaffolds, and understanding why brittleness concentrates in particular instance-configuration pairs.