Full text not available for this paper
Summary (Overview)
- CrossSWE is a novel benchmark for cross-repository joint development, comprising 171 real-world instances spanning 165 repositories, 20 technical domains, and 13 programming languages.
- Unlike prior benchmarks where multiple repositories serve only as context, CrossSWE requires agents to modify multiple repositories jointly to complete a single task, verified through an explicit joint-necessity criterion.
- The benchmark exhibits significantly larger solution scopes than SWE-bench Pro and SWE-Bench ProMax: 56.1% of instances modify >10 files and 66.7% change >200 lines of code.
- Evaluation of five frontier LLMs reveals substantial challenges: the strongest model, GPT-6.1 Sol, achieves only 58.48% Pass@1, far below its single-repository performance.
- Cross-repository inconsistency emerges as the dominant failure mode (44.2% of failures), highlighting the difficulty of maintaining consistency across interdependent codebases.
Introduction and Theoretical Foundation
Background and Motivation
Large language models (LLMs) and coding agents have made rapid progress on repository-level software engineering, with benchmarks like SWE-Bench establishing issue resolution in real-world repositories as a standard evaluation setting. Subsequent work expanded this paradigm toward broader language coverage, larger codebases, longer-horizon tasks, and more complex software changes.
However, existing efforts primarily pursue increasing complexity within individual repositories. The context an agent needs to understand and the scope of changes it must consider remain bounded by a single repository. Recent benchmarks such as BeyondSWE and DepBench bring multiple repositories into the picture, but the other repositories merely serve as context—the agent reads them to understand dependencies, but the patch and tests remain within one repository.
The Cross-Repository Gap
Real-world software development often spans multiple interdependent repositories. Changes to APIs, libraries, or protocols in one repository propagate to dependent repositories, requiring coordinated modifications across repository boundaries. This introduces additional challenges in:
- Dependency reasoning across codebases
- Change propagation through interfaces
- Interface consistency maintenance
Key Example (Figure 2): A real-world bug involving oauth2-proxy and Higress. During session refresh, multiple cookies are incorrectly merged into a single HTTP header. Resolving this requires changes to both repositories:
oauth2-proxymust preserve cookies as a listHigressmust process and emit each cookie separately
Modifying only one repository leaves the shared interface inconsistent. Only coordinated changes restore intended behavior.
Task Formulation
Each CrossSWE instance consists of four components:
- Pre-configured execution environment with participating repositories at base commits
- Problem statement describing a task spanning multiple repositories
- Test suite validating target cross-repository behavior
- Repository-specific gold patches recording the original coordinated solution
Methodology
Benchmark Construction Pipeline
CrossSWE uses a four-stage construction framework (Figure 4):
Stage 1: Candidate Collection
- Used GitHub API to identify active repositories
- Mined merged PRs, issues, commits, and release discussions from January 2025 through September 2026
- Used cross-repository relationship cues: API evolution, dependency migration, provider–consumer, client–server relations
- Identified candidates through explicit cross-references to issues/PRs/commits in other codebases
Stage 2: Human Screening
- Manually inspected candidates to remove non-genuine cross-repository cases
- Retained only when at least two repositories contain substantive, causally-related implementation changes
- Excluded candidates with documentation-only, release-notes-only, or non-implementation changes
Stage 3: Environment Construction & Joint-Necessity Verification
This is the critical methodological contribution. For an instance with N repositories, each repository has two states: base (0) or gold-patched (1). The framework exhaustively enumerates all mixed-version configurations:
For a two-repository instance (A, B), four configurations are tested:
- — both unpatched
- — only A patched
- — only B patched
- — both patched
A candidate satisfies the joint-necessity criterion only when:
- Target behavior fails in the all-base configuration
- Target behavior fails in every partially patched configuration
- Target behavior succeeds when all repository patches are applied together
This explicitly distinguishes tasks whose solution merely involves multiple repositories from tasks where repository changes are jointly necessary.
Stage 4: Problem Statement Construction
- Transformed verified candidates into self-contained natural-language task specifications
- Removed solution-specific information (gold patches, fixed commits, evaluator metadata)
- Explicitly specified cross-repository behavior requirements to avoid ambiguity
Benchmark Composition
CrossSWE covers:
- 8 task-intent categories: feature development (31.6%), API/protocol evolution (25.1%), bug fixing (12.9%), security/identity/observability (12.3%), deployment/platform/runtime (8.2%), migration/refactoring (5.3%), test/conformance adaptation (4.1%), transitive propagation
- 20 technical domains: web/application frameworks (23.4%), cloud-native/containers (19.3%), API/protocol ecosystems (18.1%), build/package management (17.5%), etc.
- 13 programming languages: multi-language (32.2%), Go (29.2%), TypeScript (10.5%), Python (9.4%), Rust (6.4%), JavaScript (5.8%), C/C++/Java/Other (4.1%)
Empirical Validation / Results
Experimental Setup
- Agent scaffold: Mini-SWE-Agent v2.4.6 (lightweight repository-level coding agent)
- Models evaluated: GPT-6.1 Sol, Claude Sonnet 5.5, DeepSeek V4.1 Flash, GLM-5.3 Flash, Qwen3.8-Flash-Next
- Configuration: High reasoning-effort setting, max 300 interaction steps, $10 cost limit per instance
- Primary metric: Pass@1 (instance resolved only when final state satisfies complete evaluation suite)
Main Results
Table 1: Overall Performance on CrossSWE
| Model | Pass@1 (%) | Single-lang. | Multi-lang. | Avg. Input Tokens | Avg. Output Tokens | Avg. Steps | Avg. Cost ($) |
|---|---|---|---|---|---|---|---|
| Closed-source models | |||||||
| GPT-6.1 Sol | 58.48 | 60.34 | 54.55 | 0.97M | 12.5K | 32.9 | 0.3496 |
| Claude Sonnet 5.5 | 50.29 | 52.59 | 43.64 | 1.31M | 16.3K | 29.2 | 0.5217 |
| Open-source models | |||||||
| DeepSeek V4.1 Flash | 52.63 | 52.59 | 52.73 | 17.54M | 88.73K | 152.7 | 0.2881 |
| GLM-5.3 Flash | 43.86 | 45.69 | 40.00 | 9.74M | 55.21K | 141.7 | 0.3411 |
| Qwen3.8-Flash-Next | 45.03 | 45.61 | 43.86 | 13.31M | 75.68K | 182.1 | 0.2939 |
Key findings:
- GPT-6.1 Sol achieves only 58.48% Pass@1, substantially below its reported 75.2% on DeepSWE v1.1 — indicating cross-repository tasks introduce significant additional challenges
- GPT-6.1 Sol leads on single-language tasks (60.34%) but DeepSeek V4.1 Flash remains competitive on multi-language tasks (52.73%)
- Closed-source models achieve competitive costs despite higher per-token pricing due to substantially shorter interaction trajectories (32.9 vs. 152.7 steps for DeepSeek)
Joint vs. Independent Development
Table 2: Comparison of Joint and Single-Side Development
| Model | Joint Pass@1 (%) | Single-side Pass@1 (%) | Joint Steps | Single-side Steps |
|---|---|---|---|---|
| GPT-6.1 Sol | 58.48 | 64.55 (+6.07) | 32.90 | 18.54 (-14.36) |
| DeepSeek V4.1 Flash | 52.63 | 61.81 (+9.18) | 152.70 | 126.94 (-25.76) |
Deriving 330 single-side tasks from the 165 two-repository instances (keeping cross-repository context but requiring modification of only one repo) shows both models improve substantially, confirming that joint development poses challenges beyond merely accessing multiple repositories.
Agent Behavior Analysis
- Fewer steps ≠ proportionally smaller code changes: GPT-6.1 Sol and Claude Sonnet 5.5 make concentrated edits in fewer interactions; DeepSeek and GLM perform more exploration without proportionally larger final patches
- Distinct strategies: GPT/Claude identify relevant code and modify earlier with larger edits; DeepSeek/GLM spend more time exploring structures and alternate between incremental modifications and testing
Failure Pattern Analysis
Five failure categories identified through manual trajectory analysis:
Figure 7: Distribution of Primary Failure Modes
| Failure Mode | GPT | Claude | DeepSeek | GLM | Qwen |
|---|---|---|---|---|---|
| Cross-repo inconsistency | 44.2% | 41.2% | 29.2% | 40.7% | 41.3% |
| Incomplete implementation | 14.0% | 29.4% | 25.0% | 25.4% | 17.4% |
| Compatibility regression | 16.3% | 11.8% | 18.8% | 11.9% | 17.4% |
| Logic/state error | 18.6% | 7.8% | 12.5% | 10.2% | 10.9% |
| Boundary handling | 7.0% | 9.8% | 14.6% | 11.9% | 13.0% |
Cross-repository inconsistency dominates across all five models (29.2%–44.2% of failures), involving incompatible interfaces, protocols, or artifacts across repositories, including compilation and build failures.
Theoretical and Practical Implications
Theoretical Implications
-
Task complexity taxonomy: CrossSWE introduces a complementary dimension of task complexity — joint necessity across repositories — beyond the established dimensions of codebase size, task horizon, and single-repo modification scope.
-
Joint-necessity verification: The exhaustive enumeration framework provides a rigorous, executable method to verify that changes are genuinely jointly necessary, distinguishing "multi-repo involvement" from "multi-repo necessity."
-
Consistency as a distinct capability: The dominance of cross-repository inconsistency failures suggests that maintaining interface consistency across codebases is a capability not captured by single-repository benchmarks.
Practical Implications
-
Benchmark gap: The 58.48% vs. 75.2% performance gap (CrossSWE vs. DeepSWE) quantifies the additional difficulty of cross-repository coordination, informing model development priorities.
-
Cost-efficiency insights: GPT-6.1 Sol's competitive cost ($0.3496 avg.) despite higher per-token pricing demonstrates that interaction efficiency can offset token pricing — relevant for deployment decisions.
-
Agent design guidance: The distinct exploration strategies observed suggest opportunities for hybrid approaches combining efficient code location (GPT/Claude style) with thorough dependency exploration.
Conclusion
Main Takeaways
-
CrossSWE extends software-engineering evaluation to cross-repository joint development, comprising 171 real-world instances with executable verification of jointly necessary changes.
-
Current frontier LLMs face substantial challenges: the strongest model achieves only 58.48% Pass@1, significantly below single-repository performance.
-
Cross-repository inconsistency is the dominant failure mode (44.2%), highlighting limitations in maintaining consistency across repository boundaries.
Future Directions
-
Extend to larger repository networks: Only 6 instances involve 3+ repositories; future work should target larger networks of interdependent repositories.
-
Repository discovery: Currently, agents are given the participating repositories. Future extensions could introduce irrelevant repositories to assess repository discovery and selection capabilities.
-
Validation across scaffolds: Findings are based on Mini-SWE-Agent's Bash-based interactions; direct validation with other harnesses is needed.
Limitations
- Limited coverage of large-scale multi-repository development (165 of 171 instances are two-repository)
- Participating repositories are provided, excluding repository-discovery challenges
- Unified scaffold and cost/step limits may not capture all agent capabilities
Related papers
- From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop
RepoFactory's proxy-driven environment evolution loop yields RepoHorizen-35B-A3B, a model improving coding-agent benchmarks by 22.6 points on NL2Repo and 8.5 points on average.
- When Terminal-Agent Training Stalls: Demystifying Data Generation and Verification Challenges
Structural validity alone does not guarantee benchmark validity; task difficulty calibration is critical, as adding hard tasks restores informative RL training failures.
- BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
BuggyMLE generates 6,600+ bug-injected ML repositories and a structural complexity metric (CCI) predicting solve rates with R²=0.951, enabling difficulty-graded benchmarks where a 9B model rivals frontier agents.