SWE-Nexus: Automated Composition of Software Engineering Tasks from Test Execution

Summary (Overview)

• Novel task construction paradigm: SWE-Nexus introduces a fully automated, four-agent pipeline that constructs repository-level implementation tasks from test execution rather than historical pull requests, enabling task generation beyond individual PR boundaries.
• Compositional benchmark: The resulting benchmark contains 600 tasks from 105 repositories across six programming languages (116 atomic and 484 composite tasks spanning 2–9 test files), with all scopes validated through execution-grounded procedures.
• Execution-footprint validation: Each task scope is retained only if all expected scoring points pass in the original repository and masking the traced function bodies causes at least one failure or error, with deterministic composition that counts shared functions once and revalidates every constituent test file.
• Scope-weighted scoring: Introduces a partial-recovery metric that weights test outcomes by source spans of associated production function bodies, fixed before solving.
• Strong baseline results: Across 12 models and 3 agent scaffolds, the best configuration (Claude Opus 4.8 under Claude Code) resolves 37.8% of tasks in one attempt, with resolved rates declining from 55.2% (Easy) to 7.4% (Hard).

Introduction and Theoretical Foundation

Background and Motivation

Repository-level benchmarks traditionally derive tasks from historical pull requests (PRs), combining issue descriptions, code changes, tests, and execution environments. However, this approach has fundamental limitations:

1. Test–function ambiguity: Code and tests appearing in the same PR do not reveal which implementation each test actually exercises. A test may exercise only some modified functions while calling unchanged helpers.
2. Contamination concerns: Public implementations raise concerns about training data contamination as agents solve more established instances.
3. Scope limitations: A PR or commit captures one historical change, whereas a software requirement may span files and modules.

Core Research Question

Can we automatically construct diverse and challenging repository-level tasks beyond individual PR boundaries, with implementation scopes validated through test execution?

Theoretical Foundation

The approach builds on execution tracing (Rothermel and Harrold, 1997), which directly records the functions each test exercises, combined with masking-based validation to verify that removing the observed scope disrupts tested behavior. The key insight is that test assertions specify expected behavior but do not identify which production code to mask—tracing bridges this gap.

Methodology

Task Definition

Let $R$ denote a repository at a fixed revision with an associated execution environment. A task is defined by:

• Selected test files $T$
• Validated implementation scope $S$ of production functions whose bodies are masked

The solver receives a natural-language specification $q$ and a copy of the masked repository $R^{-}$ with selected test files withheld, then submits a patch $\Delta$ evaluated against hidden tests in a clean environment.

Four-Agent Construction Pipeline

1. Selector Agent: Chooses test files from a prefiltered pool, prioritizing explicit behavioral assertions, stable execution, and focused functional scope.
2. Trace Agent: Executes tests and maps observed functions to editable production function bodies, producing the observed execution footprint $S_t$ for each test file $t$.
3. Validation Agent: Adapts the execution plan and result parser to each repository, establishing a fixed evaluation protocol.
4. Query Agent: Generates the natural-language behavioral specification from original implementations, selected tests, and repository context.

Execution-Grounded Scope Validation

For each selected file $t$ with expected scoring points $P_t$, a scope is retained only if:

$$\forall p \in P_t,\; o_R(p) = \text{pass},$$ $$\exists p \in P_t : o_{R^{-}_t}(p) \in \{\text{fail}, \text{error}\}$$

where $o_X(p)$ denotes the parsed outcome of point $p$ in repository state $X$. The original run checks test validity; the masked run checks whether removing the scope causes observable failure.

Deterministic Scope Composition

For compatible test files $T$ from the same repository revision and environment, the composed scope is the union:

$$S = \bigcup_{t \in T} S_t$$

Shared functions are deduplicated by source identity, and every constituent test file is revalidated under the complete combined mask—all points must pass originally and at least one must fail under the mask.

Fixed Evaluation and Scoring

A task is resolved only when the evaluation run is valid, every expected scoring point passes, and the patch introduces no additional failures. For partial recovery, the scope-weighted score weights test outcomes by source spans of associated function bodies:

$$w_p = \sum_{f \in S : p \in A(f)} \frac{L_f}{|A(f)|}, \quad S_w(\Delta) = \frac{\sum_{p \in P} w_p b_p(\Delta)}{\sum_{p \in P} w_p}$$

where $L_f$ is the body span of function $f$, $A(f)$ is the set of scoring points associated with $f$, and $b_p(\Delta) \in \{0,1\}$ indicates whether point $p$ passes for patch $\Delta$.

Difficulty Tiers

Tasks are assigned Easy, Medium, or Hard tiers using selected test-file count (1–3, 4–6, 7–9) and reference restoration patch size (1–300, 301–999, ≥1,000 changed lines), with the final difficulty being the higher tier.

Empirical Validation / Results

Benchmark Statistics

Main Results (P@1, Claude Code)

Cells report resolved rate / scope-weighted score (%). Bold marks metric maxima.

Key Findings

• Difficulty gradient: Mean resolved rates decline from 55.2% (Easy) to 20.4% (Medium) to 7.4% (Hard) across all 12 models under Claude Code.
• Partial recovery measurement: A FlowFuse patch passing 128 of 141 scoring points achieves 91.6% scope-weighted score; a Jackson Kotlin module patch passes 18 of 20 points (90.4%)—both unresolved but capturing substantial partial recovery.
• Solver activity analysis: Across 126 model–scaffold–language configurations, resolved attempts have lower median tool calls in 96.8% of configurations (median difference: −35.5 calls) and lower modified files in 97.6% (median difference: −4 files).

Theoretical and Practical Implications

Implications for Benchmark Design

• Beyond PR boundaries: SWE-Nexus demonstrates that validated execution footprints can compose tasks spanning functions and modules beyond individual historical changes, addressing contamination and fixed-collection staleness.
• Execution-grounded validation: The independent validation procedure (original pass + masked failure) provides a principled criterion for task quality that does not rely on human review.
• Compositional determinism: The deterministic composition stage with revalidation of every constituent test file ensures composite tasks have coherent, testable requirements.

Implications for Model Training and Evaluation

• Graded rewards: The scope-weighted score could provide graded rewards for reinforcement learning based on source spans of associated function bodies, even when tasks remain unresolved.
• Difficulty calibration: The consistent decline across models and attempt budgets validates the three-tier difficulty distinction, useful for curriculum design.
• Scaffold sensitivity: The leading model varies by scaffold (Claude Opus 4.8 under Claude Code and OpenCode; GPT-5.5/GPT-5.4 tie under Cline), highlighting the importance of evaluating across agent scaffolds.

Conclusion

SWE-Nexus presents a fully automated, four-agent pipeline that constructs repository-level implementation tasks from test execution rather than historical changes. The benchmark contains 600 tasks from 105 repositories across six languages, with all scopes validated through execution-grounded procedures and deterministic composition. Across 12 models and 3 agent scaffolds, the best configuration resolves 37.8% of tasks in one attempt, with scope-weighted scoring capturing partial recovery. Composing validated execution footprints extends tasks beyond individual historical changes to span functions and modules.

Limitations and Future Directions

• Test coverage dependency: Behaviors not exercised by existing tests may be underrepresented; extending to functionality with limited coverage requires developing and validating additional tests.
• Language coverage: Only six programming languages are currently supported; broader coverage requires adapting tracing, source mapping, and result parsing to additional runtimes and test frameworks.
• Future work: Applying the scope-weighted score as a graded reward for reinforcement learning training, and extending the pipeline to additional languages and test frameworks.

Ethical Considerations

The benchmark derives tasks from public repositories, and use/redistribution remains subject to original licenses. Code execution during construction and evaluation should run in isolated environments since tests and setup commands can have side effects. Benchmark results verify selected test outcomes and do not establish security or deployment readiness.