Full text not available for this paper

Summary (Overview)

  • CrossSWE is a novel benchmark for cross-repository joint development, comprising 171 real-world instances spanning 165 repositories, 20 technical domains, and 13 programming languages.
  • Unlike prior benchmarks where multiple repositories serve only as context, CrossSWE requires agents to modify multiple repositories jointly to complete a single task, verified through an explicit joint-necessity criterion.
  • The benchmark exhibits significantly larger solution scopes than SWE-bench Pro and SWE-Bench ProMax: 56.1% of instances modify >10 files and 66.7% change >200 lines of code.
  • Evaluation of five frontier LLMs reveals substantial challenges: the strongest model, GPT-6.1 Sol, achieves only 58.48% Pass@1, far below its single-repository performance.
  • Cross-repository inconsistency emerges as the dominant failure mode (44.2% of failures), highlighting the difficulty of maintaining consistency across interdependent codebases.

Introduction and Theoretical Foundation

Background and Motivation

Large language models (LLMs) and coding agents have made rapid progress on repository-level software engineering, with benchmarks like SWE-Bench establishing issue resolution in real-world repositories as a standard evaluation setting. Subsequent work expanded this paradigm toward broader language coverage, larger codebases, longer-horizon tasks, and more complex software changes.

However, existing efforts primarily pursue increasing complexity within individual repositories. The context an agent needs to understand and the scope of changes it must consider remain bounded by a single repository. Recent benchmarks such as BeyondSWE and DepBench bring multiple repositories into the picture, but the other repositories merely serve as context—the agent reads them to understand dependencies, but the patch and tests remain within one repository.

The Cross-Repository Gap

Real-world software development often spans multiple interdependent repositories. Changes to APIs, libraries, or protocols in one repository propagate to dependent repositories, requiring coordinated modifications across repository boundaries. This introduces additional challenges in:

  • Dependency reasoning across codebases
  • Change propagation through interfaces
  • Interface consistency maintenance

Key Example (Figure 2): A real-world bug involving oauth2-proxy and Higress. During session refresh, multiple cookies are incorrectly merged into a single HTTP header. Resolving this requires changes to both repositories:

  • oauth2-proxy must preserve cookies as a list
  • Higress must process and emit each cookie separately

Modifying only one repository leaves the shared interface inconsistent. Only coordinated changes restore intended behavior.

Task Formulation

Each CrossSWE instance consists of four components:

  1. Pre-configured execution environment with participating repositories at base commits
  2. Problem statement describing a task spanning multiple repositories
  3. Test suite validating target cross-repository behavior
  4. Repository-specific gold patches recording the original coordinated solution

Methodology

Benchmark Construction Pipeline

CrossSWE uses a four-stage construction framework (Figure 4):

Stage 1: Candidate Collection

  • Used GitHub API to identify active repositories
  • Mined merged PRs, issues, commits, and release discussions from January 2025 through September 2026
  • Used cross-repository relationship cues: API evolution, dependency migration, provider–consumer, client–server relations
  • Identified candidates through explicit cross-references to issues/PRs/commits in other codebases

Stage 2: Human Screening

  • Manually inspected candidates to remove non-genuine cross-repository cases
  • Retained only when at least two repositories contain substantive, causally-related implementation changes
  • Excluded candidates with documentation-only, release-notes-only, or non-implementation changes

Stage 3: Environment Construction & Joint-Necessity Verification

This is the critical methodological contribution. For an instance with N repositories, each repository has two states: base (0) or gold-patched (1). The framework exhaustively enumerates all 2N2^N mixed-version configurations:

For a two-repository instance (A, B), four configurations are tested:

  • (A0,B0)(A_0, B_0) — both unpatched
  • (A1,B0)(A_1, B_0) — only A patched
  • (A0,B1)(A_0, B_1) — only B patched
  • (A1,B1)(A_1, B_1) — both patched

A candidate satisfies the joint-necessity criterion only when:

  • Target behavior fails in the all-base configuration
  • Target behavior fails in every partially patched configuration
  • Target behavior succeeds when all repository patches are applied together

This explicitly distinguishes tasks whose solution merely involves multiple repositories from tasks where repository changes are jointly necessary.

Stage 4: Problem Statement Construction

  • Transformed verified candidates into self-contained natural-language task specifications
  • Removed solution-specific information (gold patches, fixed commits, evaluator metadata)
  • Explicitly specified cross-repository behavior requirements to avoid ambiguity

Benchmark Composition

CrossSWE covers:

  • 8 task-intent categories: feature development (31.6%), API/protocol evolution (25.1%), bug fixing (12.9%), security/identity/observability (12.3%), deployment/platform/runtime (8.2%), migration/refactoring (5.3%), test/conformance adaptation (4.1%), transitive propagation
  • 20 technical domains: web/application frameworks (23.4%), cloud-native/containers (19.3%), API/protocol ecosystems (18.1%), build/package management (17.5%), etc.
  • 13 programming languages: multi-language (32.2%), Go (29.2%), TypeScript (10.5%), Python (9.4%), Rust (6.4%), JavaScript (5.8%), C/C++/Java/Other (4.1%)

Empirical Validation / Results

Experimental Setup

  • Agent scaffold: Mini-SWE-Agent v2.4.6 (lightweight repository-level coding agent)
  • Models evaluated: GPT-6.1 Sol, Claude Sonnet 5.5, DeepSeek V4.1 Flash, GLM-5.3 Flash, Qwen3.8-Flash-Next
  • Configuration: High reasoning-effort setting, max 300 interaction steps, $10 cost limit per instance
  • Primary metric: Pass@1 (instance resolved only when final state satisfies complete evaluation suite)

Main Results

Table 1: Overall Performance on CrossSWE

ModelPass@1 (%)Single-lang.Multi-lang.Avg. Input TokensAvg. Output TokensAvg. StepsAvg. Cost ($)
Closed-source models
GPT-6.1 Sol58.4860.3454.550.97M12.5K32.90.3496
Claude Sonnet 5.550.2952.5943.641.31M16.3K29.20.5217
Open-source models
DeepSeek V4.1 Flash52.6352.5952.7317.54M88.73K152.70.2881
GLM-5.3 Flash43.8645.6940.009.74M55.21K141.70.3411
Qwen3.8-Flash-Next45.0345.6143.8613.31M75.68K182.10.2939

Key findings:

  • GPT-6.1 Sol achieves only 58.48% Pass@1, substantially below its reported 75.2% on DeepSWE v1.1 — indicating cross-repository tasks introduce significant additional challenges
  • GPT-6.1 Sol leads on single-language tasks (60.34%) but DeepSeek V4.1 Flash remains competitive on multi-language tasks (52.73%)
  • Closed-source models achieve competitive costs despite higher per-token pricing due to substantially shorter interaction trajectories (32.9 vs. 152.7 steps for DeepSeek)

Joint vs. Independent Development

Table 2: Comparison of Joint and Single-Side Development

ModelJoint Pass@1 (%)Single-side Pass@1 (%)Joint StepsSingle-side Steps
GPT-6.1 Sol58.4864.55 (+6.07)32.9018.54 (-14.36)
DeepSeek V4.1 Flash52.6361.81 (+9.18)152.70126.94 (-25.76)

Deriving 330 single-side tasks from the 165 two-repository instances (keeping cross-repository context but requiring modification of only one repo) shows both models improve substantially, confirming that joint development poses challenges beyond merely accessing multiple repositories.

Agent Behavior Analysis

  • Fewer steps ≠ proportionally smaller code changes: GPT-6.1 Sol and Claude Sonnet 5.5 make concentrated edits in fewer interactions; DeepSeek and GLM perform more exploration without proportionally larger final patches
  • Distinct strategies: GPT/Claude identify relevant code and modify earlier with larger edits; DeepSeek/GLM spend more time exploring structures and alternate between incremental modifications and testing

Failure Pattern Analysis

Five failure categories identified through manual trajectory analysis:

Figure 7: Distribution of Primary Failure Modes

Failure ModeGPTClaudeDeepSeekGLMQwen
Cross-repo inconsistency44.2%41.2%29.2%40.7%41.3%
Incomplete implementation14.0%29.4%25.0%25.4%17.4%
Compatibility regression16.3%11.8%18.8%11.9%17.4%
Logic/state error18.6%7.8%12.5%10.2%10.9%
Boundary handling7.0%9.8%14.6%11.9%13.0%

Cross-repository inconsistency dominates across all five models (29.2%–44.2% of failures), involving incompatible interfaces, protocols, or artifacts across repositories, including compilation and build failures.


Theoretical and Practical Implications

Theoretical Implications

  1. Task complexity taxonomy: CrossSWE introduces a complementary dimension of task complexity — joint necessity across repositories — beyond the established dimensions of codebase size, task horizon, and single-repo modification scope.

  2. Joint-necessity verification: The 2N2^N exhaustive enumeration framework provides a rigorous, executable method to verify that changes are genuinely jointly necessary, distinguishing "multi-repo involvement" from "multi-repo necessity."

  3. Consistency as a distinct capability: The dominance of cross-repository inconsistency failures suggests that maintaining interface consistency across codebases is a capability not captured by single-repository benchmarks.

Practical Implications

  1. Benchmark gap: The 58.48% vs. 75.2% performance gap (CrossSWE vs. DeepSWE) quantifies the additional difficulty of cross-repository coordination, informing model development priorities.

  2. Cost-efficiency insights: GPT-6.1 Sol's competitive cost ($0.3496 avg.) despite higher per-token pricing demonstrates that interaction efficiency can offset token pricing — relevant for deployment decisions.

  3. Agent design guidance: The distinct exploration strategies observed suggest opportunities for hybrid approaches combining efficient code location (GPT/Claude style) with thorough dependency exploration.


Conclusion

Main Takeaways

  1. CrossSWE extends software-engineering evaluation to cross-repository joint development, comprising 171 real-world instances with executable verification of jointly necessary changes.

  2. Current frontier LLMs face substantial challenges: the strongest model achieves only 58.48% Pass@1, significantly below single-repository performance.

  3. Cross-repository inconsistency is the dominant failure mode (44.2%), highlighting limitations in maintaining consistency across repository boundaries.

Future Directions

  1. Extend to larger repository networks: Only 6 instances involve 3+ repositories; future work should target larger networks of interdependent repositories.

  2. Repository discovery: Currently, agents are given the participating repositories. Future extensions could introduce irrelevant repositories to assess repository discovery and selection capabilities.

  3. Validation across scaffolds: Findings are based on Mini-SWE-Agent's Bash-based interactions; direct validation with other harnesses is needed.

Limitations

  • Limited coverage of large-scale multi-repository development (165 of 171 instances are two-repository)
  • Participating repositories are provided, excluding repository-discovery challenges
  • Unified scaffold and cost/step limits may not capture all agent capabilities

Related papers