# From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop

> RepoFactory's proxy-driven environment evolution loop yields RepoHorizen-35B-A3B, a model improving coding-agent benchmarks by 22.6 points on NL2Repo and 8.5 points on average.

- **Source:** [arXiv](https://arxiv.org/abs/2611.21152)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/bfLowk
- **Whiteboard:** https://picx.dev/p/bfLowk/image

## Summary

## Summary (Overview)

- **RepoFactory** is an automated pipeline that converts real-world repositories into 10,159 verifiable training environments across 17 software domains, where agents must reconstruct complete projects from scratch given a language specification.
- The **Environment Evolution Loop** optimizes a multiplicative objective $Q = A \cdot C \cdot S$ combining validity ($A$), structural coupling ($C$), and implementation scope ($S$), using timely, low-cost proxy feedback that avoids expensive model rollouts.
- Training Qwen3.5-35B-A3B on RepoFactory yields **RepoHorizen-35B-A3B**, achieving gains of **22.6 percentage points on NL2Repo-Bench** and **8.5 points on average** across five benchmarks covering long-horizon coding and agentic tasks.
- Environment evolution contributes an additional **5.1 points on NL2Repo** and **2.1 points on average** over training without evolution, validating the effectiveness of the proxy metrics.
- RepoFactory outperforms six existing open SWE training datasets (OpenSWE, R2E, ScaleSWE, SWE-smith, SWE-rebench, SWE-rebench-v2) by 21.2 points on NL2Repo under controlled comparisons with the same harness.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Large language models (LLMs) are increasingly deployed as agents that interact with real-world environments over long horizons. Unlike static language tasks, agentic work requires **coherent progress toward completing long-horizon tasks**—not just producing correct answers. Software engineering has become a major application domain, yet existing training pipelines face critical limitations:

1. **Rigid construction pipelines** (e.g., DeNovoSWE, MindForge) follow linear processes that discard unsuitable repositories rather than repairing them, limiting scalability.
2. **Feedback focused on reliability overlooks training value**—signals like execution diagnostics and coverage feedback (used in Program-Gym, RefactorForge) improve reliability without explicitly optimizing whether tasks require substantial implementation or coordination.
3. **Model-execution feedback is costly and time-consuming**—approaches like BenchEvolver and SETA require model rollouts on candidate tasks, which is prohibitive for long-horizon coding where each rollout involves extensive implementation.

### Key Insight

The paper proposes that **timely, reliable proxy signals**—derived from static code analysis and verifier execution rather than model rollouts—can guide environment evolution to improve both reliability and training value at scale.

---

## Methodology

### Problem Formulation

Each training environment is defined as $\boldsymbol{e} = (I, W, V, O)$ where:
- $I$ = language specification
- $W$ = isolated workspace and runtime
- $V$ = verifier
- $O$ = source repository's oracle implementation

An agent $\pi$ receives $(I, W)$ and reconstructs a project $P = \pi(I, W)$, scored by $V(P) \in [0, 1]$.

### Environment Objective

The multiplicative objective is defined as:

$$
Q = A \cdot C \cdot S
$$

where:

**Validity** requires full reward for the oracle and zero reward for the unchanged workspace:

$$
A = \begin{cases} 1, & V(O) = 1 \text{ and } V(P_0) = 0, \\ 0, & \text{otherwise}. \end{cases}
$$

**Structural coupling** uses normalized propagation cost:

$$
C = \frac{R}{N^2}
$$

where $N$ = number of functions and $R$ = number of reachable ordered function pairs in the static internal call graph (including self-pairs).

**Implementation scope** combines code size and dependency breadth:

$$
S = \log_{10}(\mathrm{LOC}) + \log_{10}(1 + D) - 2
$$

where $\mathrm{LOC}$ = source code size and $D$ = number of direct runtime dependencies.

### Construction Pipeline

1. **Repository Search and Filtering**: Searches public code-hosting platforms, excluding repositories in established benchmarks. Requires package metadata, nonempty test suites, 5+ implementation files, <40,000 lines of Python, and no external services or specialized hardware.

2. **Runtime Construction**: Builds Docker images from dependency manifests, installs dependencies, identifies test commands, and validates that all tests pass in a fresh container with bounded resources and no network access.

3. **Repository Analysis**: Uses AST-based scanners to extract public classes, functions, and methods with signatures, exceptions, and return information. Builds an implementation tree excluding test files.

4. **Task Synthesis**: Creates language specifications and API manifests describing the repository's functionality and public interfaces. Checks against repository symbols imported by tests, returning missing symbols for revision.

5. **Environment Evolution Loop**: Iteratively revises environments up to 12 rounds:
   - First checks validity $A$; repairs candidates that fail
   - Compares $S$ and $C$ against thresholds (0.5 and 0.01 respectively)
   - Prioritizes terms below threshold; retains revisions only if scores strictly increase
   - Uses verifier execution and code analysis only—**no solver rollouts or downstream training**

All construction agents use Qwen3.8-Max.

---

## Empirical Validation / Results

### Data Statistics

- **10,159 environments** across 17 software domains (largest category only 12.5%)
- **Median oracle**: 4.2k lines of source code, 26 source files
- **2.09 million** statically identified test definitions across instances
- **Median SFT trajectory**: 64 assistant turns (10,947 samples from 5,577 environments)
- Environments with larger scope show lower success rates (23.0% → 13.4% from smallest to largest scope group), validating $S$ as a difficulty proxy

### Main Results (Table 1)

| Model | Params | NL2Repo | SWE-bench Verified | SWE-bench Pro | SWE-bench Multilingual | Terminal-Bench 2.1 | Avg. |
|---|---|---|---|---|---|---|---|
| **RepoHorizen-35B-A3B** | 35B-A3B | **46.4 (+22.6)** | 72.2 (+6.4) | 47.6 (+5.2) | 61.0 (+1.3) | 40.5 (+6.8) | **53.5 (+8.5)** |
| w/o environment evolution | 35B-A3B | 41.3 (+17.5) | 70.9 (+5.1) | 45.1 (+2.7) | 60.8 (+1.1) | 39.3 (+5.6) | 51.5 (+6.4) |
| Qwen3.5-35B-A3B (base) | 35B-A3B | 23.8 | 65.8 | 42.4 | 59.7 | 33.7 | 45.1 |

RepoHorizen surpasses GPT-5.5 (45.1) and Gemini-3.1 Pro (33.4) on NL2Repo, as well as larger open-weight models like DeepSeek-V4-Pro (38.5) and MiniMax-M3 (42.1).

### Environment Scaling

Performance increases steadily with training set size (23.8% → 46.4% with 5,577 environments), with **no evidence of saturation** at the largest scale tested. Gains become more pronounced beyond 3,000 environments.

### Model Size Scaling

NL2Repo gains of **12.2 to 22.6 percentage points** across 9B, 27B, and 35B-A3B models. The 35B-A3B model gains the most despite the 27B model's higher baseline.

### Data Source Comparison (Table 2)

Under controlled comparisons with the same Claude Code harness, RepoFactory leads all six open-data baselines on all five benchmarks, with a **21.2-point advantage over OpenSWE** (the strongest baseline) on NL2Repo.

### Ablation of Evolution Objective

- **Recovery**: Evolution increases validated environments from 169 to 347 out of 500 (33.8% → 69.4% yield)
- **Scope optimization**: NL2Repo accuracy rises from 30.2% (Level 1) to 32.9% (Level 4) with longer interactions
- **Coupling optimization**: Accuracy reaches 32.5% at Level 4 despite shorter trajectories, showing training value depends on component relationships, not just implementation extent

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Proxy-based environment evolution**: Demonstrates that static code metrics and verifier execution can substitute for expensive model rollouts in guiding environment synthesis, making recursive self-improvement practical for long-horizon tasks.

2. **Multiplicative objective design**: The product form $Q = A \cdot C \cdot S$ ensures validity is a hard gate (any invalid environment has $Q = 0$) while allowing trade-offs between coupling and scope for valid environments.

3. **Training value quantification**: Provides evidence that structural properties of environments (coupling, scope) directly correlate with downstream training effectiveness, offering a principled basis for environment design.

### Practical Implications

- **Scalable environment generation**: The 69.4% recovery rate through evolution means far fewer repositories are wasted, expanding the usable repository pool.
- **Cross-task transfer**: Training on full-repository generation transfers to issue resolution (SWE-bench) and terminal tasks, suggesting that repository-level reconstruction builds broadly useful agentic skills.
- **Cost efficiency**: The evolution loop avoids solver rollouts entirely, making large-scale environment synthesis computationally feasible.

---

## Conclusion

RepoFactory demonstrates that **repository-level environments with verifiable feedback** significantly improve coding-agent training. The Environment Evolution Loop's low-cost proxy objective provides a practical mechanism for scaling environment synthesis while improving both reliability and training value. Key future directions include:

- Extending language coverage beyond Python
- Incorporating more detailed capability requirements into proxy metrics for more targeted training
- Exploring whether the proxy-based evolution approach generalizes to other long-horizon agentic domains beyond software engineering

---

## Limitations

- **Language coverage**: Currently focused on Python projects only
- **Proxy granularity**: Metrics characterize task content at a broad structural level and do not explicitly distinguish the specific capabilities required by individual tasks
- Extending language coverage and incorporating capability-level information could support wider development settings and more targeted training

---

_Markdown view of https://picx.dev/p/bfLowk, served by PicX — AI-generated visual whiteboard summaries of research papers._
