Summary (Overview)

  • RepoFactory is an automated pipeline that converts real-world repositories into 10,159 verifiable training environments across 17 software domains, where agents must reconstruct complete projects from scratch given a language specification.
  • The Environment Evolution Loop optimizes a multiplicative objective Q=A⋅C⋅SQ = A \cdot C \cdot S combining validity (AA), structural coupling (CC), and implementation scope (SS), using timely, low-cost proxy feedback that avoids expensive model rollouts.
  • Training Qwen3.5-35B-A3B on RepoFactory yields RepoHorizen-35B-A3B, achieving gains of 22.6 percentage points on NL2Repo-Bench and 8.5 points on average across five benchmarks covering long-horizon coding and agentic tasks.
  • Environment evolution contributes an additional 5.1 points on NL2Repo and 2.1 points on average over training without evolution, validating the effectiveness of the proxy metrics.
  • RepoFactory outperforms six existing open SWE training datasets (OpenSWE, R2E, ScaleSWE, SWE-smith, SWE-rebench, SWE-rebench-v2) by 21.2 points on NL2Repo under controlled comparisons with the same harness.

Introduction and Theoretical Foundation

Background and Motivation

Large language models (LLMs) are increasingly deployed as agents that interact with real-world environments over long horizons. Unlike static language tasks, agentic work requires coherent progress toward completing long-horizon tasks—not just producing correct answers. Software engineering has become a major application domain, yet existing training pipelines face critical limitations:

  1. Rigid construction pipelines (e.g., DeNovoSWE, MindForge) follow linear processes that discard unsuitable repositories rather than repairing them, limiting scalability.
  2. Feedback focused on reliability overlooks training value—signals like execution diagnostics and coverage feedback (used in Program-Gym, RefactorForge) improve reliability without explicitly optimizing whether tasks require substantial implementation or coordination.
  3. Model-execution feedback is costly and time-consuming—approaches like BenchEvolver and SETA require model rollouts on candidate tasks, which is prohibitive for long-horizon coding where each rollout involves extensive implementation.

Key Insight

The paper proposes that timely, reliable proxy signals—derived from static code analysis and verifier execution rather than model rollouts—can guide environment evolution to improve both reliability and training value at scale.


Methodology

Problem Formulation

Each training environment is defined as e=(I,W,V,O)\boldsymbol{e} = (I, W, V, O) where:

  • II = language specification
  • WW = isolated workspace and runtime
  • VV = verifier
  • OO = source repository's oracle implementation

An agent π\pi receives (I,W)(I, W) and reconstructs a project P=π(I,W)P = \pi(I, W), scored by V(P)∈[0,1]V(P) \in [0, 1].

Environment Objective

The multiplicative objective is defined as:

Q=A⋅C⋅SQ = A \cdot C \cdot S

where:

Validity requires full reward for the oracle and zero reward for the unchanged workspace:

A={1,V(O)=1 and V(P0)=0,0,otherwise.A = \begin{cases} 1, & V(O) = 1 \text{ and } V(P_0) = 0, \\ 0, & \text{otherwise}. \end{cases}

Structural coupling uses normalized propagation cost:

C=RN2C = \frac{R}{N^2}

where NN = number of functions and RR = number of reachable ordered function pairs in the static internal call graph (including self-pairs).

Implementation scope combines code size and dependency breadth:

S=log⁡10(LOC)+log⁡10(1+D)−2S = \log_{10}(\mathrm{LOC}) + \log_{10}(1 + D) - 2

where LOC\mathrm{LOC} = source code size and DD = number of direct runtime dependencies.

Construction Pipeline

  1. Repository Search and Filtering: Searches public code-hosting platforms, excluding repositories in established benchmarks. Requires package metadata, nonempty test suites, 5+ implementation files, <40,000 lines of Python, and no external services or specialized hardware.

  2. Runtime Construction: Builds Docker images from dependency manifests, installs dependencies, identifies test commands, and validates that all tests pass in a fresh container with bounded resources and no network access.

  3. Repository Analysis: Uses AST-based scanners to extract public classes, functions, and methods with signatures, exceptions, and return information. Builds an implementation tree excluding test files.

  4. Task Synthesis: Creates language specifications and API manifests describing the repository's functionality and public interfaces. Checks against repository symbols imported by tests, returning missing symbols for revision.

  5. Environment Evolution Loop: Iteratively revises environments up to 12 rounds:

    • First checks validity AA; repairs candidates that fail
    • Compares SS and CC against thresholds (0.5 and 0.01 respectively)
    • Prioritizes terms below threshold; retains revisions only if scores strictly increase
    • Uses verifier execution and code analysis only—no solver rollouts or downstream training

All construction agents use Qwen3.8-Max.


Empirical Validation / Results

Data Statistics

  • 10,159 environments across 17 software domains (largest category only 12.5%)
  • Median oracle: 4.2k lines of source code, 26 source files
  • 2.09 million statically identified test definitions across instances
  • Median SFT trajectory: 64 assistant turns (10,947 samples from 5,577 environments)
  • Environments with larger scope show lower success rates (23.0% → 13.4% from smallest to largest scope group), validating SS as a difficulty proxy

Main Results (Table 1)

ModelParamsNL2RepoSWE-bench VerifiedSWE-bench ProSWE-bench MultilingualTerminal-Bench 2.1Avg.
RepoHorizen-35B-A3B35B-A3B46.4 (+22.6)72.2 (+6.4)47.6 (+5.2)61.0 (+1.3)40.5 (+6.8)53.5 (+8.5)
w/o environment evolution35B-A3B41.3 (+17.5)70.9 (+5.1)45.1 (+2.7)60.8 (+1.1)39.3 (+5.6)51.5 (+6.4)
Qwen3.5-35B-A3B (base)35B-A3B23.865.842.459.733.745.1

RepoHorizen surpasses GPT-5.5 (45.1) and Gemini-3.1 Pro (33.4) on NL2Repo, as well as larger open-weight models like DeepSeek-V4-Pro (38.5) and MiniMax-M3 (42.1).

Environment Scaling

Performance increases steadily with training set size (23.8% → 46.4% with 5,577 environments), with no evidence of saturation at the largest scale tested. Gains become more pronounced beyond 3,000 environments.

Model Size Scaling

NL2Repo gains of 12.2 to 22.6 percentage points across 9B, 27B, and 35B-A3B models. The 35B-A3B model gains the most despite the 27B model's higher baseline.

Data Source Comparison (Table 2)

Under controlled comparisons with the same Claude Code harness, RepoFactory leads all six open-data baselines on all five benchmarks, with a 21.2-point advantage over OpenSWE (the strongest baseline) on NL2Repo.

Ablation of Evolution Objective

  • Recovery: Evolution increases validated environments from 169 to 347 out of 500 (33.8% → 69.4% yield)
  • Scope optimization: NL2Repo accuracy rises from 30.2% (Level 1) to 32.9% (Level 4) with longer interactions
  • Coupling optimization: Accuracy reaches 32.5% at Level 4 despite shorter trajectories, showing training value depends on component relationships, not just implementation extent

Theoretical and Practical Implications

Theoretical Contributions

  1. Proxy-based environment evolution: Demonstrates that static code metrics and verifier execution can substitute for expensive model rollouts in guiding environment synthesis, making recursive self-improvement practical for long-horizon tasks.

  2. Multiplicative objective design: The product form Q=A⋅C⋅SQ = A \cdot C \cdot S ensures validity is a hard gate (any invalid environment has Q=0Q = 0) while allowing trade-offs between coupling and scope for valid environments.

  3. Training value quantification: Provides evidence that structural properties of environments (coupling, scope) directly correlate with downstream training effectiveness, offering a principled basis for environment design.

Practical Implications

  • Scalable environment generation: The 69.4% recovery rate through evolution means far fewer repositories are wasted, expanding the usable repository pool.
  • Cross-task transfer: Training on full-repository generation transfers to issue resolution (SWE-bench) and terminal tasks, suggesting that repository-level reconstruction builds broadly useful agentic skills.
  • Cost efficiency: The evolution loop avoids solver rollouts entirely, making large-scale environment synthesis computationally feasible.

Conclusion

RepoFactory demonstrates that repository-level environments with verifiable feedback significantly improve coding-agent training. The Environment Evolution Loop's low-cost proxy objective provides a practical mechanism for scaling environment synthesis while improving both reliability and training value. Key future directions include:

  • Extending language coverage beyond Python
  • Incorporating more detailed capability requirements into proxy metrics for more targeted training
  • Exploring whether the proxy-based evolution approach generalizes to other long-horizon agentic domains beyond software engineering

Limitations

  • Language coverage: Currently focused on Python projects only
  • Proxy granularity: Metrics characterize task content at a broad structural level and do not explicitly distinguish the specific capabilities required by individual tasks
  • Extending language coverage and incorporating capability-level information could support wider development settings and more targeted training

Related papers