# SWE-at-Scale: Scalable Issue Synthesis from Historical Bug Fixes

> SWE-at-Scale reverses real-world bug fixes to train bug-inducing models, generating SWE tasks that are more realistic, diverse, and challenging than existing synthetic baselines.

- **Source:** [arXiv](https://arxiv.org/abs/2610.21359)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/E1nL7v
- **Whiteboard:** https://picx.dev/p/E1nL7v/image

## Summary

## Summary (Overview)

- **SWE-at-Scale** is a novel framework that synthesizes realistic software engineering (SWE) tasks by **reversing real-world bug fixes** to train bug-inducing models, rather than relying solely on general-purpose LLM prompting.
- The framework introduces two synthesis variants: **SWE-at-Scale** (direct bug-inducing patch generation) and **SWE-at-Scale+** (explicit natural-language bug pattern generation followed by code realization).
- Trained on **13,407 real-fix transformations from 688 repositories**, the models produce tasks that are more **realistic** (55.9%–61.2% Human Preference Rate), **diverse** (88.27% unique patches at 200 generations/file), and **challenging** (Pass@1 between 16.81%–48.48%) than existing synthesis baselines.
- The framework demonstrates **scalable bug diversity**: MBTE (Mean Bug Type Entropy) increases from 0.583 to 2.063 bits as sampling budget grows from K=5 to K=200, without saturation.
- SWE-at-Scale+ achieves a **94.0% Usability Rate** with a **61.2% HPR**, outperforming all synthetic baselines (SWE-smith, BugPilot) in the usability–realism trade-off space.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Repository-level software engineering (SWE) tasks have become a central testbed for evaluating coding agents. A useful SWE task requires more than a code change—it must:

1. Preserve a realistic repository context
2. Introduce an observable behavioral bug
3. Provide a meaningful issue description
4. Remain executable within the repository's testing infrastructure

**The scaling problem:** Real-world SWE benchmarks (e.g., SWE-bench) are inherently difficult to scale because:
- New tasks emerge only at the pace of real software development
- Collection is constrained by repository availability and environment construction costs

### Limitations of Existing Synthetic Approaches

Existing LLM-based synthesis methods (SWE-smith, SWE-Synth, BugPilot) rely on:
- **Heuristic prompting** of general-purpose LLMs
- **Pretrained knowledge** already encoded in the models

> Without dedicated training for bug injection, these approaches largely depend on the bug-generation knowledge already encoded in general-purpose LLMs and elicited through prompting, making repeated generation more likely to revisit familiar bug patterns rather than continually expanding the diversity of synthesized tasks.

### Key Insight

> Historical fixes encapsulate deep mechanistic knowledge of bug formation: **inverting a fix transforms repair supervision into bug-induction supervision** grounded in actual development scenarios.

This motivates a different scaling strategy: rather than repeatedly prompting a general-purpose LLM, **reverse real-world bug fixes** to construct correct-to-buggy supervision and train a reusable bug-injection model that can generate diverse bugs in new code contexts.

---

## Methodology

### Stage 1: Learn From Real Fixes

**Training Data Collection:**
- Sources: SWE-Fixer, SWE-rebench, SWE-bench-Live, SWE-Gym (134K+ issue–fix instances total)
- For each real-fix instance:
  1. Apply the original golden patch to the buggy repository to reconstruct fixed source files
  2. Compute a new unified diff from each fixed file back to its original buggy version (reverse-direction diff)
  3. This reversed diff serves as the **bug-inducing patch** for training

**Quality Filters Applied:**
- Remove missing/unparseable patches, non-code files, import/comment/whitespace-only changes
- Discard structurally incomplete patches
- Verify round-trip patch application (forward and reverse)
- Annotate bug types (following BugPilot)
- Remove examples exceeding 64K-token limit
- Distribution-aware sampling to balance bug types, repositories, and context lengths
- Manual inspection of selected patches

**Final dataset:** 13,407 real-fix transformations from 688 repositories

**Bug type distribution in training data:** Logic (47.6%), Interface (17.1%), Boundary (9.9%), Protocol (9.4%), Attribute (6.3%), Copy Semantics (4.3%), Bookkeeping (2.4%), Filesystem (1.8%), Arguments (0.8%)

### Stage 2: Dual-Mode Bug Synthesis

**SWE-at-Scale (Direct Patch Generation):**
- Backbone: **Qwen3.5-9B**
- Fine-tuned via supervised fine-tuning (SFT) with LoRA
- Input: Complete golden source file with line numbers
- Target: Unified diff that transforms correct code into buggy state
- The model learns jointly *where* to modify and *how* to realize bug-inducing changes

**SWE-at-Scale+ (Bug Pattern Realization):**
- Uses **DeepSeek-V4-Flash-0731** as an expert teacher to infer natural-language bug patterns
- Each pattern specifies: task category, target location, current behavior, specific code modification, implementation constraints, expected observable effect
- Trained to predict the pattern instead of the unified diff
- At synthesis time, an **executor** realizes the specified modifications (presented as ordinary code-modification instructions to avoid prior bug-generation knowledge)

### Task Assembly and Validation

1. Apply generated code changes to the golden repository
2. Execute test suite; retain only candidates causing at least one **Pass-to-Fail (P2F)** test
3. Generate GitHub-style issue description using an LLM (provided with code diff, P2F test source, execution output, and two demonstrations)
4. Pair issue description with validated buggy repository to form the final SWE task

### Training Configuration

| Hyperparameter | Value |
|---|---|
| Base model | Qwen3.5-9B |
| Fine-tuning method | LoRA supervised fine-tuning |
| LoRA rank r | 16 |
| LoRA scaling α | 32 |
| LoRA dropout | 0.05 |
| Target modules | All linear layers |
| Training examples | 13,407 |
| Maximum sequence length | 64,000 tokens |
| Number of epochs | 1 |
| Learning rate | 1 × 10⁻⁴ |
| Scheduler | Cosine |
| Warmup ratio | 0.03 |
| Numerical precision | bfloat16 |

---

## Empirical Validation / Results

### Synthesis Quality and Task Difficulty

**Usability vs. Realism (Figure 5):**

| Method | Usability Rate | Human Preference Rate (HPR) |
|---|---|---|
| SWE-at-Scale+ | **94.0%** | **61.2%** |
| SWE-at-Scale | 72.6% | 55.9% |
| Best LLM-Rewrite | 72.2% | 51.8% |
| LLM-Modify | — | Substantially lower HPR |

**Downstream Issue Resolution (Pass@1, %):**

| Method | Developer | DS-V4-Flash | GPT-5.4-mini | GLM-5.3-Flash | Qwen3.8-Flash |
|---|---|---|---|---|---|
| **Real-world Tasks** | | | | | |
| DeepSWE | Human | 54.40 | 24.30 | 63.40 | 58.70 |
| SWE-bench Verified | Human | 91.80 | 69.60 | 94.00 | 97.00 |
| **Synthetic Tasks** | | | | | |
| LLM-Modify | DS-V4-Flash | 79.85 | 35.82 | 57.78 | 63.96 |
| LLM-Rewrite | DS-V4-Flash | 67.80 | 25.14 | 47.36 | 56.32 |
| FeatAdd | DS-V4-Flash | 58.62 | 37.93 | 62.07 | 62.07 |
| **SWE-at-Scale** | — | **48.48** | **31.68** | **42.15** | **44.44** |
| **SWE-at-Scale+** | — | **33.19** | **16.81** | **35.32** | **37.02** |

SWE-at-Scale+ approaches the difficulty of human-expert-designed DeepSWE tasks.

### Scaling Bug Diversity

**Repository coverage effect (Figure 6):** MBTE increases as training pool expands from 5 to 100 repositories, while usability remains stable.

**Sampling temperature effect (Figure 7):**
- MBTE increases from 0.527 (t=0.0) to peak of 1.173 (t=0.6)
- Usability decreases from 69.0% (t=0.0) to 23.4% (t=1.0)
- **Moderate temperatures (0.4–0.6)** provide the most favorable diversity–usability trade-off

**Sampling budget scaling (Figure 8):**

| Budget K | MBTE (SWE-at-Scale) | MBTE (LLM-Modify) |
|---|---|---|
| 5 | 0.583 | — |
| 25 | — | — |
| 50 | — | — |
| 100 | ~1.9 | — |
| 200 | **2.063** | 1.22 |

- At K=200: **88.27%** of usable patches remain unique (vs. 61.74% for LLM-Modify)
- Usability stabilizes at ~47% beyond K=50

### Bug Complexity and Synthesis Cost

- **Mean changed lines:** SWE-at-Scale: 18.793, SWE-at-Scale+: 18.536, SWE-bench Verified: 14.334
- **Token consumption per instance:** SWE-at-Scale: 17.45k, LLM-Modify: 4.46–10.23k, LLM-Rewrite: 34.92–62.82k, FeatAdd: 2.29M (extreme outlier)

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Reverse-learning paradigm:** Demonstrates that inverting real bug fixes provides a principled, scalable source of supervision for learning bug-inducing transformations—transforming a finite collection of real fixes into reusable knowledge.

2. **Explicit vs. implicit bug patterns:** SWE-at-Scale+ shows that externalizing bug-inducing logic as natural-language patterns provides stronger learning signal than implicit patch generation, aligning better with LLM language-modeling capabilities.

3. **Bug diversity quantification:** Introduces rigorous evaluation of within-file bug diversity via MBTE, showing that learned bug distributions can be explored at inference time without collapse.

### Practical Implications

- **Scalable benchmark construction:** SWE-at-Scale turns finite real-world fixes into exponential-scale SWE task production, addressing the fundamental bottleneck of naturally occurring issue rates.
- **Model-agnostic difficulty:** The consistent difficulty across four resolution models suggests robustness against model-specific biases.
- **Cost efficiency:** Compared to FeatAdd (2.29M tokens/instance), SWE-at-Scale requires only 17.45k tokens per instance—over two orders of magnitude cheaper.

---

## Conclusion

### Main Takeaways

SWE-at-Scale presents a scalable paradigm for SWE task synthesis by:
1. **Reversing real bug fixes** as training supervision for bug-inducing models
2. Learning a **reusable distribution of real-world bug patterns**
3. Enabling **repeated sampling** of diverse bug types from the same code context

The framework produces tasks that are realistic, diverse, and challenging—approaching the difficulty of human-expert-designed benchmarks while remaining substantially more scalable.

### Future Directions

- **Cross-language generalization:** Current work focuses exclusively on Python; extending to other programming languages with different syntax and semantics remains an open question.
- **Test coverage limitations:** Realistic bugs not covered by available test suites may be discarded, limiting task construction from repositories with weak test coverage.
- **Potential improvements:** Investigating whether bug-inducing models can operate across multiple languages to enable scalable cross-language SWE task synthesis.

### Reproducibility

Code and models are available at:
- https://github.com/cslsolow/SWE-at-scale
- https://huggingface.co/SWE-at-Scale

---

_Markdown view of https://picx.dev/p/E1nL7v, served by PicX — AI-generated visual whiteboard summaries of research papers._