Summary (Overview)

• SWE-at-Scale is a novel framework that synthesizes realistic software engineering (SWE) tasks by reversing real-world bug fixes to train bug-inducing models, rather than relying solely on general-purpose LLM prompting.
• The framework introduces two synthesis variants: SWE-at-Scale (direct bug-inducing patch generation) and SWE-at-Scale+ (explicit natural-language bug pattern generation followed by code realization).
• Trained on 13,407 real-fix transformations from 688 repositories, the models produce tasks that are more realistic (55.9%–61.2% Human Preference Rate), diverse (88.27% unique patches at 200 generations/file), and challenging (Pass@1 between 16.81%–48.48%) than existing synthesis baselines.
• The framework demonstrates scalable bug diversity: MBTE (Mean Bug Type Entropy) increases from 0.583 to 2.063 bits as sampling budget grows from K=5 to K=200, without saturation.
• SWE-at-Scale+ achieves a 94.0% Usability Rate with a 61.2% HPR, outperforming all synthetic baselines (SWE-smith, BugPilot) in the usability–realism trade-off space.

---

Introduction and Theoretical Foundation

Background and Motivation

Repository-level software engineering (SWE) tasks have become a central testbed for evaluating coding agents. A useful SWE task requires more than a code change—it must:

1. Preserve a realistic repository context
2. Introduce an observable behavioral bug
3. Provide a meaningful issue description
4. Remain executable within the repository's testing infrastructure

The scaling problem: Real-world SWE benchmarks (e.g., SWE-bench) are inherently difficult to scale because:

• New tasks emerge only at the pace of real software development
• Collection is constrained by repository availability and environment construction costs

Limitations of Existing Synthetic Approaches

Existing LLM-based synthesis methods (SWE-smith, SWE-Synth, BugPilot) rely on:

• Heuristic prompting of general-purpose LLMs
• Pretrained knowledge already encoded in the models

Without dedicated training for bug injection, these approaches largely depend on the bug-generation knowledge already encoded in general-purpose LLMs and elicited through prompting, making repeated generation more likely to revisit familiar bug patterns rather than continually expanding the diversity of synthesized tasks.

Key Insight

Historical fixes encapsulate deep mechanistic knowledge of bug formation: inverting a fix transforms repair supervision into bug-induction supervision grounded in actual development scenarios.

This motivates a different scaling strategy: rather than repeatedly prompting a general-purpose LLM, reverse real-world bug fixes to construct correct-to-buggy supervision and train a reusable bug-injection model that can generate diverse bugs in new code contexts.

---

Methodology

Stage 1: Learn From Real Fixes

Training Data Collection:

• Sources: SWE-Fixer, SWE-rebench, SWE-bench-Live, SWE-Gym (134K+ issue–fix instances total)
• For each real-fix instance:
  1. Apply the original golden patch to the buggy repository to reconstruct fixed source files
  2. Compute a new unified diff from each fixed file back to its original buggy version (reverse-direction diff)
  3. This reversed diff serves as the bug-inducing patch for training

Quality Filters Applied:

• Remove missing/unparseable patches, non-code files, import/comment/whitespace-only changes
• Discard structurally incomplete patches
• Verify round-trip patch application (forward and reverse)
• Annotate bug types (following BugPilot)
• Remove examples exceeding 64K-token limit
• Distribution-aware sampling to balance bug types, repositories, and context lengths
• Manual inspection of selected patches

Final dataset: 13,407 real-fix transformations from 688 repositories

Bug type distribution in training data: Logic (47.6%), Interface (17.1%), Boundary (9.9%), Protocol (9.4%), Attribute (6.3%), Copy Semantics (4.3%), Bookkeeping (2.4%), Filesystem (1.8%), Arguments (0.8%)

Stage 2: Dual-Mode Bug Synthesis

SWE-at-Scale (Direct Patch Generation):

• Backbone: Qwen3.5-9B
• Fine-tuned via supervised fine-tuning (SFT) with LoRA
• Input: Complete golden source file with line numbers
• Target: Unified diff that transforms correct code into buggy state
• The model learns jointly where to modify and how to realize bug-inducing changes

SWE-at-Scale+ (Bug Pattern Realization):

• Uses DeepSeek-V4-Flash-0731 as an expert teacher to infer natural-language bug patterns
• Each pattern specifies: task category, target location, current behavior, specific code modification, implementation constraints, expected observable effect
• Trained to predict the pattern instead of the unified diff
• At synthesis time, an executor realizes the specified modifications (presented as ordinary code-modification instructions to avoid prior bug-generation knowledge)

Task Assembly and Validation

1. Apply generated code changes to the golden repository
2. Execute test suite; retain only candidates causing at least one Pass-to-Fail (P2F) test
3. Generate GitHub-style issue description using an LLM (provided with code diff, P2F test source, execution output, and two demonstrations)
4. Pair issue description with validated buggy repository to form the final SWE task

Training Configuration

---

Empirical Validation / Results

Synthesis Quality and Task Difficulty

Usability vs. Realism (Figure 5):

Downstream Issue Resolution (Pass@1, %):

SWE-at-Scale+ approaches the difficulty of human-expert-designed DeepSWE tasks.

Scaling Bug Diversity

Repository coverage effect (Figure 6): MBTE increases as training pool expands from 5 to 100 repositories, while usability remains stable.

Sampling temperature effect (Figure 7):

• MBTE increases from 0.527 (t=0.0) to peak of 1.173 (t=0.6)
• Usability decreases from 69.0% (t=0.0) to 23.4% (t=1.0)
• Moderate temperatures (0.4–0.6) provide the most favorable diversity–usability trade-off

Sampling budget scaling (Figure 8):

• At K=200: 88.27% of usable patches remain unique (vs. 61.74% for LLM-Modify)
• Usability stabilizes at ~47% beyond K=50

Bug Complexity and Synthesis Cost

• Mean changed lines: SWE-at-Scale: 18.793, SWE-at-Scale+: 18.536, SWE-bench Verified: 14.334
• Token consumption per instance: SWE-at-Scale: 17.45k, LLM-Modify: 4.46–10.23k, LLM-Rewrite: 34.92–62.82k, FeatAdd: 2.29M (extreme outlier)

---

Theoretical and Practical Implications

Theoretical Contributions

1. Reverse-learning paradigm: Demonstrates that inverting real bug fixes provides a principled, scalable source of supervision for learning bug-inducing transformations—transforming a finite collection of real fixes into reusable knowledge.

2. Explicit vs. implicit bug patterns: SWE-at-Scale+ shows that externalizing bug-inducing logic as natural-language patterns provides stronger learning signal than implicit patch generation, aligning better with LLM language-modeling capabilities.

3. Bug diversity quantification: Introduces rigorous evaluation of within-file bug diversity via MBTE, showing that learned bug distributions can be explored at inference time without collapse.

Practical Implications

• Scalable benchmark construction: SWE-at-Scale turns finite real-world fixes into exponential-scale SWE task production, addressing the fundamental bottleneck of naturally occurring issue rates.
• Model-agnostic difficulty: The consistent difficulty across four resolution models suggests robustness against model-specific biases.
• Cost efficiency: Compared to FeatAdd (2.29M tokens/instance), SWE-at-Scale requires only 17.45k tokens per instance—over two orders of magnitude cheaper.

---

Conclusion

Main Takeaways

SWE-at-Scale presents a scalable paradigm for SWE task synthesis by:

1. Reversing real bug fixes as training supervision for bug-inducing models
2. Learning a reusable distribution of real-world bug patterns
3. Enabling repeated sampling of diverse bug types from the same code context

The framework produces tasks that are realistic, diverse, and challenging—approaching the difficulty of human-expert-designed benchmarks while remaining substantially more scalable.

Future Directions

• Cross-language generalization: Current work focuses exclusively on Python; extending to other programming languages with different syntax and semantics remains an open question.
• Test coverage limitations: Realistic bugs not covered by available test suites may be discarded, limiting task construction from repositories with weak test coverage.
• Potential improvements: Investigating whether bug-inducing models can operate across multiple languages to enable scalable cross-language SWE task synthesis.

Reproducibility

Code and models are available at:

• https://github.com/cslsolow/SWE-at-scale
• https://huggingface.co/SWE-at-Scale