Summary (Overview)

• SWE-Bench ProMax is a new expert-curated, multilingual code refactoring benchmark comprising 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, Rust) and 70 repositories.
• The benchmark addresses three critical gaps: benchmark saturation (frontier agents exceed 75% on SWE-bench Verified), evaluation quality issues (nearly 60% of unsolved SWE-bench Verified instances contain flawed tests), and the absence of refactoring as a benchmark domain.
• Instances are substantially larger than existing benchmarks: averaging 11.4 modified files and 261.6 lines of code per instance, with the most complex instance touching 244 files.
• The best model (GPT-5.2) achieves only 41.2% resolve rate, confirming the benchmark presents a meaningful, unsaturated challenge.
• Key finding: open-weight models are highly competitive with proprietary ones at a fraction of the cost (GLM-5 achieves 36.5% at $0.24/instance vs. Claude Sonnet 4.6 at 38.8$4.77/instance).

Introduction and Theoretical Foundation

Motivation

The paper identifies three fundamental weaknesses in current coding benchmarks:

1. Difficulty saturation: Frontier agents now exceed 75% resolve rate on SWE-bench Verified, and the gap among top systems continues to narrow. Most existing benchmark instances involve modifications to a small number of files with limited lines of code.

2. Evaluation quality defects: Test suites suffer from two complementary defects:

   • Overly narrow tests that enforce specific implementation details and reject functionally correct solutions (35.5% of unsolved SWE-bench Verified instances)
   • Overly broad tests that check behavior not specified in the task description (18.8% of unsolved instances)

3. Data contamination: Growing evidence of benchmark contamination from public repositories erodes the correlation between benchmark scores and genuine capability.

Why Refactoring?

Code refactoring—the disciplined process of restructuring existing code without changing its external behavior—is identified as a prototypical long-horizon task:

• One of the most frequent activities in professional software development
• A key mechanism for managing technical debt in evolving codebases
• Industry leaders (OpenAI, Cursor) highlight "project-scale refactors" as primary use cases for sustained, multi-context-window agents
• Requires understanding large codebases, coordinating changes across many files, and preserving subtle behavioral invariants

Theoretical Foundation

The benchmark formulation is outcome-driven: an instance is resolved if and only if the agent's modifications pass every test in the suite, evaluating the final state of the repository rather than intermediate steps.

Methodology

Task Formulation

Each instance consists of four components:

1. Pre-configured Docker environment with repository at pre-refactoring commit
2. Issue description specifying the intended refactoring in precise natural language
3. Test suite validating correct refactoring application
4. Gold patch recording the original developer's solution

Three-Stage Construction Pipeline

Stage 1: Data Collection

• GitHub API used to identify candidate repositories with:
  • At least 500 stars
  • Approved open-source license
  • Primary language ≥80% of codebase among seven target languages
• Commits submitted after January 2025 with "refactor" (not "bug fix") in message
• Must modify both test and non-test files
• Initial pool: 29,782 candidates

Stage 2: Environment Construction

• Isolated Docker environments built using SWE-Factory
• Gold patch applied, full test suite executed
• Instances where environment cannot be established or gold patch fails are discarded

Stage 3: Filtering and Problem Rewriting (expert-driven, four steps):

1. Commit analysis: Experts analyze diffs with LLM assistance
2. Quality filtering: Remove insufficient complexity tasks; remove overly narrow and overly broad tests
3. Problem statement rewriting: Rewrite issue descriptions from scratch as precise, self-contained specifications
4. Human verification: Ensure description is both necessary and sufficient for gold patch

Dataset Statistics

Empirical Validation / Results

Experimental Setup

• Two agent scaffolds: mini-swe-agent (minimal SWE-agent reimplementation) and OpenHands (richer runtime with sandboxed execution)
• Step limit: 300; Cost limit: $10 per instance
• Six frontier models: Gemini-3-Pro, Claude Sonnet 4.6, GPT-5.2 (proprietary); GLM-5, Kimi-K2.5, Qwen3.5 (open-weight)
• Primary metric: Resolve rate (Pass@1) on all 170 instances

Main Results

Key Findings

1. Unsaturated challenge: Best model achieves only 41.2% vs. 75%+ on SWE-bench Verified
2. Open-weight competitiveness: GLM-5 and Qwen3.5 (36.5%) come within 5 points of GPT-5.2 (41.2%) at ~1/15th the cost
3. Scaffold matters: Most models improve substantially with OpenHands (e.g., GPT-5.2: 21.8% → 41.2%)
4. Language variance: No single model dominates all languages; TypeScript and Rust show surprising variance across models

Agent Behavior Analysis

The dominant failure mode is incomplete refactoring:

• Agents modify fewer files than the gold patch requires, with the gap widening for larger patches
• Agents correctly locate and edit core files but fail to propagate changes to peripheral call sites, documentation, configuration files, and test fixtures
• Failed attempts consume substantially more interaction rounds than successful ones, indicating unproductive exploration cycles (repeated edit–revert cycles)

Theoretical and Practical Implications

For Benchmark Design

• Quality over quantity: The rigorous multi-stage curation pipeline (29,782 → 170 instances) demonstrates that reliable evaluation requires expert verification of both problem descriptions and test suites
• Difficulty calibration: SWE-Bench ProMax establishes that refactoring tasks with large cross-file scope provide a meaningful difficulty frontier that current benchmarks fail to capture

For Agent Development

• Cross-file coordination bottleneck: The primary limitation is not reasoning ability per se, but the capacity to maintain a coherent plan across many files and persist through cascading consequences of large-scale structural changes
• Cost-efficiency insights: Higher spending often reflects unproductive exploration rather than genuine progress; open-weight models offer competitive performance at dramatically lower cost
• Scaffold design: Richer runtime tooling (OpenHands) provides substantial benefits for large-scale refactoring tasks

For Language-Specific Research

• The multilingual coverage enables analysis of how language-specific features (Rust's ownership model, C's manual memory management, Java's type hierarchies) affect agent performance
• Results suggest model performance varies more with training data composition than inherent language difficulty

Conclusion

SWE-Bench ProMax addresses three critical gaps in AI coding evaluation: benchmark saturation, evaluation quality assurance, and the absence of refactoring as a benchmark domain. The benchmark's 170 expert-curated instances across seven languages require coordinated changes averaging 11.4 files and 261.6 lines of code—substantially exceeding existing benchmarks.

The evaluation reveals that:

1. SWE-Bench ProMax remains far from saturated (best model: 41.2% resolve rate)
2. Open-weight models are highly competitive with proprietary ones at a fraction of the cost
3. Incomplete refactoring—failing to propagate changes across all required files—is the dominant failure mode, indicating that sustained cross-file coordination remains a fundamental bottleneck for current AI coding agents

Future directions implied by this work include developing agents with better long-horizon planning and cross-file coordination capabilities, and expanding the benchmark to cover additional refactoring patterns and language ecosystems.

Dataset: https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax