# SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

> SWE-Bench ProMax, a 170-instance multilingual refactoring benchmark, shows frontier agents resolve only 41.2% of tasks, with open-weight models competitive at a fraction of the cost.

- **Source:** [arXiv](https://arxiv.org/abs/2608.09802)
- **Published:** 2026-08-12
- **Permalink:** https://picx.dev/p/o3LZSK
- **Whiteboard:** https://picx.dev/p/o3LZSK/image

## Summary

## Summary (Overview)

- **SWE-Bench ProMax** is a new expert-curated, multilingual code refactoring benchmark comprising **170 instances** drawn from real commits across **seven programming languages** (Python, Java, TypeScript, Go, C, C++, Rust) and 70 repositories.
- The benchmark addresses three critical gaps: **benchmark saturation** (frontier agents exceed 75% on SWE-bench Verified), **evaluation quality issues** (nearly 60% of unsolved SWE-bench Verified instances contain flawed tests), and the **absence of refactoring** as a benchmark domain.
- Instances are substantially larger than existing benchmarks: averaging **11.4 modified files** and **261.6 lines of code** per instance, with the most complex instance touching **244 files**.
- The best model (GPT-5.2) achieves only **41.2% resolve rate**, confirming the benchmark presents a meaningful, unsaturated challenge.
- Key finding: **open-weight models are highly competitive** with proprietary ones at a fraction of the cost (GLM-5 achieves 36.5% at $0.24/instance vs. Claude Sonnet 4.6 at 38.8% and $4.77/instance).

## Introduction and Theoretical Foundation

### Motivation

The paper identifies three fundamental weaknesses in current coding benchmarks:

1. **Difficulty saturation**: Frontier agents now exceed 75% resolve rate on SWE-bench Verified, and the gap among top systems continues to narrow. Most existing benchmark instances involve modifications to a small number of files with limited lines of code.

2. **Evaluation quality defects**: Test suites suffer from two complementary defects:
   - **Overly narrow tests** that enforce specific implementation details and reject functionally correct solutions (35.5% of unsolved SWE-bench Verified instances)
   - **Overly broad tests** that check behavior not specified in the task description (18.8% of unsolved instances)

3. **Data contamination**: Growing evidence of benchmark contamination from public repositories erodes the correlation between benchmark scores and genuine capability.

### Why Refactoring?

Code refactoring—the disciplined process of restructuring existing code without changing its external behavior—is identified as a prototypical long-horizon task:

- One of the most frequent activities in professional software development
- A key mechanism for managing technical debt in evolving codebases
- Industry leaders (OpenAI, Cursor) highlight "project-scale refactors" as primary use cases for sustained, multi-context-window agents
- Requires understanding large codebases, coordinating changes across many files, and preserving subtle behavioral invariants

### Theoretical Foundation

The benchmark formulation is **outcome-driven**: an instance is resolved if and only if the agent's modifications pass every test in the suite, evaluating the final state of the repository rather than intermediate steps.

## Methodology

### Task Formulation

Each instance consists of four components:
1. **Pre-configured Docker environment** with repository at pre-refactoring commit
2. **Issue description** specifying the intended refactoring in precise natural language
3. **Test suite** validating correct refactoring application
4. **Gold patch** recording the original developer's solution

### Three-Stage Construction Pipeline

**Stage 1: Data Collection**
- GitHub API used to identify candidate repositories with:
  - At least 500 stars
  - Approved open-source license
  - Primary language ≥80% of codebase among seven target languages
- Commits submitted after January 2025 with "refactor" (not "bug fix") in message
- Must modify both test and non-test files
- Initial pool: **29,782 candidates**

**Stage 2: Environment Construction**
- Isolated Docker environments built using SWE-Factory
- Gold patch applied, full test suite executed
- Instances where environment cannot be established or gold patch fails are discarded

**Stage 3: Filtering and Problem Rewriting** (expert-driven, four steps):
1. **Commit analysis**: Experts analyze diffs with LLM assistance
2. **Quality filtering**: Remove insufficient complexity tasks; remove overly narrow and overly broad tests
3. **Problem statement rewriting**: Rewrite issue descriptions from scratch as precise, self-contained specifications
4. **Human verification**: Ensure description is both necessary and sufficient for gold patch

### Dataset Statistics

| Metric | Mean | Max |
|--------|------|-----|
| **Issue Description** | | |
| Tokens | 685.3 | 2,092 |
| **Gold Patch (source)** | | |
| # Files | 11.4 | 182 |
| Lines of code | 261.6 | 4,503 |
| Tokens | 8,179.5 | 72,623 |
| **Test Patch** | | |
| # Files | 4.5 | 66 |
| Lines of code | 185.5 | 1,959 |
| Tokens | 3,980.8 | 52,031 |
| **Total** | | |
| # Files | 15.9 | 244 |

## Empirical Validation / Results

### Experimental Setup

- **Two agent scaffolds**: mini-swe-agent (minimal SWE-agent reimplementation) and OpenHands (richer runtime with sandboxed execution)
- **Step limit**: 300; **Cost limit**: $10 per instance
- **Six frontier models**: Gemini-3-Pro, Claude Sonnet 4.6, GPT-5.2 (proprietary); GLM-5, Kimi-K2.5, Qwen3.5 (open-weight)
- **Primary metric**: Resolve rate (Pass@1) on all 170 instances

### Main Results

| Model | Overall | Avg. Steps | Avg. Cost | Py | Java | TS | Go | C | C++ | Rust |
|-------|---------|------------|-----------|-----|------|-----|-----|-----|------|------|
| **Mini-SWE-Agent** | | | | | | | | | | |
| Gemini-3-Pro | 26.5 | 58.0 | $0.60 | 17.2 | 15.4 | 14.3 | 26.1 | 50.0 | 45.5 | 27.3 |
| Claude Sonnet 4.6 | 30.6 | 99.5 | $2.32 | 13.8 | 19.2 | 32.1 | 13.0 | 65.0 | 40.9 | 40.9 |
| GPT-5.2 | 21.8 | 25.2 | $0.19 | 17.2 | 15.4 | 21.4 | 13.0 | 45.0 | 31.8 | 13.6 |
| GLM-5 | 22.9 | 108.9 | $0.10 | 13.8 | 7.7 | 25.0 | 21.7 | 50.0 | 22.7 | 27.3 |
| Kimi-K2.5 | 26.5 | 85.3 | $0.37 | 17.2 | 23.1 | 21.4 | 17.4 | 60.0 | 31.8 | 22.7 |
| Qwen3.5 | 20.6 | 155.4 | $0.93 | 17.2 | 7.7 | 10.7 | 13.0 | 45.0 | 27.3 | 31.8 |
| **OpenHands** | | | | | | | | | | |
| Gemini-3-Pro | 19.4 | 51.2 | $1.49 | 13.8 | 19.2 | 0.0 | 8.7 | 45.0 | 36.4 | 22.7 |
| Claude Sonnet 4.6 | 38.8 | 117.9 | $4.77 | 17.2 | 30.8 | 53.6 | 26.1 | 50.0 | 36.4 | 63.6 |
| GPT-5.2 | **41.2** | 115.1 | $3.60 | **48.3** | 19.2 | 35.7 | 26.1 | **75.0** | 36.4 | 54.5 |
| GLM-5 | 36.5 | 114.2 | $0.24 | 20.7 | **34.6** | 28.6 | 34.8 | 65.0 | 45.5 | 36.4 |
| Kimi-K2.5 | 32.9 | 99.6 | $0.72 | 24.1 | 30.8 | 10.7 | **43.5** | 70.0 | 45.5 | 18.2 |
| Qwen3.5 | 36.5 | 141.2 | $0.78 | 37.9 | 26.9 | 17.9 | 39.1 | 65.0 | **54.5** | 22.7 |

### Key Findings

1. **Unsaturated challenge**: Best model achieves only 41.2% vs. 75%+ on SWE-bench Verified
2. **Open-weight competitiveness**: GLM-5 and Qwen3.5 (36.5%) come within 5 points of GPT-5.2 (41.2%) at ~1/15th the cost
3. **Scaffold matters**: Most models improve substantially with OpenHands (e.g., GPT-5.2: 21.8% → 41.2%)
4. **Language variance**: No single model dominates all languages; TypeScript and Rust show surprising variance across models

### Agent Behavior Analysis

The dominant failure mode is **incomplete refactoring**:
- Agents modify fewer files than the gold patch requires, with the gap widening for larger patches
- Agents correctly locate and edit core files but fail to propagate changes to peripheral call sites, documentation, configuration files, and test fixtures
- Failed attempts consume substantially more interaction rounds than successful ones, indicating **unproductive exploration cycles** (repeated edit–revert cycles)

## Theoretical and Practical Implications

### For Benchmark Design

- **Quality over quantity**: The rigorous multi-stage curation pipeline (29,782 → 170 instances) demonstrates that reliable evaluation requires expert verification of both problem descriptions and test suites
- **Difficulty calibration**: SWE-Bench ProMax establishes that refactoring tasks with large cross-file scope provide a meaningful difficulty frontier that current benchmarks fail to capture

### For Agent Development

- **Cross-file coordination bottleneck**: The primary limitation is not reasoning ability per se, but the capacity to maintain a coherent plan across many files and persist through cascading consequences of large-scale structural changes
- **Cost-efficiency insights**: Higher spending often reflects unproductive exploration rather than genuine progress; open-weight models offer competitive performance at dramatically lower cost
- **Scaffold design**: Richer runtime tooling (OpenHands) provides substantial benefits for large-scale refactoring tasks

### For Language-Specific Research

- The multilingual coverage enables analysis of how language-specific features (Rust's ownership model, C's manual memory management, Java's type hierarchies) affect agent performance
- Results suggest model performance varies more with training data composition than inherent language difficulty

## Conclusion

SWE-Bench ProMax addresses three critical gaps in AI coding evaluation: benchmark saturation, evaluation quality assurance, and the absence of refactoring as a benchmark domain. The benchmark's 170 expert-curated instances across seven languages require coordinated changes averaging 11.4 files and 261.6 lines of code—substantially exceeding existing benchmarks.

The evaluation reveals that:
1. **SWE-Bench ProMax remains far from saturated** (best model: 41.2% resolve rate)
2. **Open-weight models are highly competitive** with proprietary ones at a fraction of the cost
3. **Incomplete refactoring**—failing to propagate changes across all required files—is the dominant failure mode, indicating that sustained cross-file coordination remains a fundamental bottleneck for current AI coding agents

**Future directions** implied by this work include developing agents with better long-horizon planning and cross-file coordination capabilities, and expanding the benchmark to cover additional refactoring patterns and language ecosystems.

**Dataset**: https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax

---

_Markdown view of https://picx.dev/p/o3LZSK, served by PicX — AI-generated visual whiteboard summaries of research papers._