# CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents

> CrossSWE, a 171-instance benchmark requiring joint modifications across multiple repositories, shows frontier LLMs achieve only 58.48% Pass@1, with cross-repository inconsistency as the dominant failure mode.

- **Source:** [arXiv](https://arxiv.org/abs/2611.01484)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/ylJdoG
- **Whiteboard:** https://picx.dev/p/ylJdoG/image

## Summary

## Summary (Overview)

- **CrossSWE** is a novel benchmark for **cross-repository joint development**, comprising **171 real-world instances** spanning **165 repositories, 20 technical domains, and 13 programming languages**.
- Unlike prior benchmarks where multiple repositories serve only as context, CrossSWE requires agents to **modify multiple repositories jointly** to complete a single task, verified through an explicit **joint-necessity criterion**.
- The benchmark exhibits significantly larger solution scopes than SWE-bench Pro and SWE-Bench ProMax: **56.1% of instances modify >10 files** and **66.7% change >200 lines of code**.
- Evaluation of five frontier LLMs reveals substantial challenges: the strongest model, **GPT-6.1 Sol, achieves only 58.48% Pass@1**, far below its single-repository performance.
- **Cross-repository inconsistency** emerges as the dominant failure mode (**44.2%** of failures), highlighting the difficulty of maintaining consistency across interdependent codebases.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Large language models (LLMs) and coding agents have made rapid progress on repository-level software engineering, with benchmarks like **SWE-Bench** establishing issue resolution in real-world repositories as a standard evaluation setting. Subsequent work expanded this paradigm toward broader language coverage, larger codebases, longer-horizon tasks, and more complex software changes.

However, existing efforts primarily pursue increasing complexity **within individual repositories**. The context an agent needs to understand and the scope of changes it must consider remain bounded by a single repository. Recent benchmarks such as **BeyondSWE** and **DepBench** bring multiple repositories into the picture, but the other repositories merely serve as **context**—the agent reads them to understand dependencies, but the patch and tests remain within one repository.

### The Cross-Repository Gap

Real-world software development often spans multiple interdependent repositories. Changes to APIs, libraries, or protocols in one repository propagate to dependent repositories, requiring coordinated modifications across repository boundaries. This introduces additional challenges in:

- **Dependency reasoning** across codebases
- **Change propagation** through interfaces
- **Interface consistency** maintenance

**Key Example (Figure 2):** A real-world bug involving `oauth2-proxy` and `Higress`. During session refresh, multiple cookies are incorrectly merged into a single HTTP header. Resolving this requires changes to **both** repositories:
- `oauth2-proxy` must preserve cookies as a list
- `Higress` must process and emit each cookie separately

Modifying only one repository leaves the shared interface inconsistent. Only coordinated changes restore intended behavior.

### Task Formulation

Each CrossSWE instance consists of four components:
1. **Pre-configured execution environment** with participating repositories at base commits
2. **Problem statement** describing a task spanning multiple repositories
3. **Test suite** validating target cross-repository behavior
4. **Repository-specific gold patches** recording the original coordinated solution

---

## Methodology

### Benchmark Construction Pipeline

CrossSWE uses a four-stage construction framework (Figure 4):

**Stage 1: Candidate Collection**
- Used GitHub API to identify active repositories
- Mined merged PRs, issues, commits, and release discussions from **January 2025 through September 2026**
- Used cross-repository relationship cues: API evolution, dependency migration, provider–consumer, client–server relations
- Identified candidates through explicit cross-references to issues/PRs/commits in other codebases

**Stage 2: Human Screening**
- Manually inspected candidates to remove non-genuine cross-repository cases
- Retained only when **at least two repositories contain substantive, causally-related implementation changes**
- Excluded candidates with documentation-only, release-notes-only, or non-implementation changes

**Stage 3: Environment Construction & Joint-Necessity Verification**

This is the critical methodological contribution. For an instance with N repositories, each repository has two states: base (0) or gold-patched (1). The framework **exhaustively enumerates all $2^N$ mixed-version configurations**:

For a two-repository instance (A, B), four configurations are tested:
- $(A_0, B_0)$ — both unpatched
- $(A_1, B_0)$ — only A patched
- $(A_0, B_1)$ — only B patched
- $(A_1, B_1)$ — both patched

A candidate satisfies the **joint-necessity criterion** only when:
- Target behavior **fails** in the all-base configuration
- Target behavior **fails** in every partially patched configuration
- Target behavior **succeeds** when all repository patches are applied together

This explicitly distinguishes tasks whose solution merely *involves* multiple repositories from tasks where repository changes are **jointly necessary**.

**Stage 4: Problem Statement Construction**
- Transformed verified candidates into self-contained natural-language task specifications
- Removed solution-specific information (gold patches, fixed commits, evaluator metadata)
- Explicitly specified cross-repository behavior requirements to avoid ambiguity

### Benchmark Composition

CrossSWE covers:
- **8 task-intent categories**: feature development (31.6%), API/protocol evolution (25.1%), bug fixing (12.9%), security/identity/observability (12.3%), deployment/platform/runtime (8.2%), migration/refactoring (5.3%), test/conformance adaptation (4.1%), transitive propagation
- **20 technical domains**: web/application frameworks (23.4%), cloud-native/containers (19.3%), API/protocol ecosystems (18.1%), build/package management (17.5%), etc.
- **13 programming languages**: multi-language (32.2%), Go (29.2%), TypeScript (10.5%), Python (9.4%), Rust (6.4%), JavaScript (5.8%), C/C++/Java/Other (4.1%)

---

## Empirical Validation / Results

### Experimental Setup

- **Agent scaffold**: Mini-SWE-Agent v2.4.6 (lightweight repository-level coding agent)
- **Models evaluated**: GPT-6.1 Sol, Claude Sonnet 5.5, DeepSeek V4.1 Flash, GLM-5.3 Flash, Qwen3.8-Flash-Next
- **Configuration**: High reasoning-effort setting, max 300 interaction steps, $10 cost limit per instance
- **Primary metric**: Pass@1 (instance resolved only when final state satisfies complete evaluation suite)

### Main Results

**Table 1: Overall Performance on CrossSWE**

| Model | Pass@1 (%) | Single-lang. | Multi-lang. | Avg. Input Tokens | Avg. Output Tokens | Avg. Steps | Avg. Cost ($) |
|-------|-----------|-------------|-------------|-------------------|--------------------|------------|---------------|
| **Closed-source models** | | | | | | | |
| GPT-6.1 Sol | **58.48** | **60.34** | **54.55** | 0.97M | 12.5K | **32.9** | **0.3496** |
| Claude Sonnet 5.5 | 50.29 | 52.59 | 43.64 | 1.31M | 16.3K | 29.2 | 0.5217 |
| **Open-source models** | | | | | | | |
| DeepSeek V4.1 Flash | 52.63 | 52.59 | 52.73 | 17.54M | 88.73K | 152.7 | 0.2881 |
| GLM-5.3 Flash | 43.86 | 45.69 | 40.00 | 9.74M | 55.21K | 141.7 | 0.3411 |
| Qwen3.8-Flash-Next | 45.03 | 45.61 | 43.86 | 13.31M | 75.68K | 182.1 | 0.2939 |

**Key findings:**
- GPT-6.1 Sol achieves only **58.48% Pass@1**, substantially below its reported 75.2% on DeepSWE v1.1 — indicating cross-repository tasks introduce significant additional challenges
- GPT-6.1 Sol leads on single-language tasks (60.34%) but DeepSeek V4.1 Flash remains competitive on multi-language tasks (52.73%)
- Closed-source models achieve competitive costs despite higher per-token pricing due to **substantially shorter interaction trajectories** (32.9 vs. 152.7 steps for DeepSeek)

### Joint vs. Independent Development

**Table 2: Comparison of Joint and Single-Side Development**

| Model | Joint Pass@1 (%) | Single-side Pass@1 (%) | Joint Steps | Single-side Steps |
|-------|------------------|----------------------|-------------|-------------------|
| GPT-6.1 Sol | 58.48 | 64.55 (+6.07) | 32.90 | 18.54 (-14.36) |
| DeepSeek V4.1 Flash | 52.63 | 61.81 (+9.18) | 152.70 | 126.94 (-25.76) |

Deriving 330 single-side tasks from the 165 two-repository instances (keeping cross-repository context but requiring modification of only one repo) shows both models improve substantially, confirming that **joint development poses challenges beyond merely accessing multiple repositories**.

### Agent Behavior Analysis

- **Fewer steps ≠ proportionally smaller code changes**: GPT-6.1 Sol and Claude Sonnet 5.5 make concentrated edits in fewer interactions; DeepSeek and GLM perform more exploration without proportionally larger final patches
- **Distinct strategies**: GPT/Claude identify relevant code and modify earlier with larger edits; DeepSeek/GLM spend more time exploring structures and alternate between incremental modifications and testing

### Failure Pattern Analysis

Five failure categories identified through manual trajectory analysis:

**Figure 7: Distribution of Primary Failure Modes**

| Failure Mode | GPT | Claude | DeepSeek | GLM | Qwen |
|-------------|-----|--------|----------|-----|------|
| **Cross-repo inconsistency** | **44.2%** | **41.2%** | **29.2%** | **40.7%** | **41.3%** |
| Incomplete implementation | 14.0% | 29.4% | 25.0% | 25.4% | 17.4% |
| Compatibility regression | 16.3% | 11.8% | 18.8% | 11.9% | 17.4% |
| Logic/state error | 18.6% | 7.8% | 12.5% | 10.2% | 10.9% |
| Boundary handling | 7.0% | 9.8% | 14.6% | 11.9% | 13.0% |

**Cross-repository inconsistency dominates** across all five models (29.2%–44.2% of failures), involving incompatible interfaces, protocols, or artifacts across repositories, including compilation and build failures.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Task complexity taxonomy**: CrossSWE introduces a complementary dimension of task complexity — *joint necessity* across repositories — beyond the established dimensions of codebase size, task horizon, and single-repo modification scope.

2. **Joint-necessity verification**: The $2^N$ exhaustive enumeration framework provides a rigorous, executable method to verify that changes are genuinely jointly necessary, distinguishing "multi-repo involvement" from "multi-repo necessity."

3. **Consistency as a distinct capability**: The dominance of cross-repository inconsistency failures suggests that maintaining interface consistency across codebases is a capability not captured by single-repository benchmarks.

### Practical Implications

1. **Benchmark gap**: The 58.48% vs. 75.2% performance gap (CrossSWE vs. DeepSWE) quantifies the additional difficulty of cross-repository coordination, informing model development priorities.

2. **Cost-efficiency insights**: GPT-6.1 Sol's competitive cost ($0.3496 avg.) despite higher per-token pricing demonstrates that interaction efficiency can offset token pricing — relevant for deployment decisions.

3. **Agent design guidance**: The distinct exploration strategies observed suggest opportunities for hybrid approaches combining efficient code location (GPT/Claude style) with thorough dependency exploration.

---

## Conclusion

### Main Takeaways

1. **CrossSWE** extends software-engineering evaluation to **cross-repository joint development**, comprising 171 real-world instances with executable verification of jointly necessary changes.

2. Current frontier LLMs face **substantial challenges**: the strongest model achieves only 58.48% Pass@1, significantly below single-repository performance.

3. **Cross-repository inconsistency** is the dominant failure mode (44.2%), highlighting limitations in maintaining consistency across repository boundaries.

### Future Directions

1. **Extend to larger repository networks**: Only 6 instances involve 3+ repositories; future work should target larger networks of interdependent repositories.

2. **Repository discovery**: Currently, agents are given the participating repositories. Future extensions could introduce irrelevant repositories to assess **repository discovery and selection** capabilities.

3. **Validation across scaffolds**: Findings are based on Mini-SWE-Agent's Bash-based interactions; direct validation with other harnesses is needed.

### Limitations

- Limited coverage of large-scale multi-repository development (165 of 171 instances are two-repository)
- Participating repositories are provided, excluding repository-discovery challenges
- Unified scaffold and cost/step limits may not capture all agent capabilities

---

_Markdown view of https://picx.dev/p/ylJdoG, served by PicX — AI-generated visual whiteboard summaries of research papers._
