# ProgramGym: Scaling Training Environments for Long-Horizon Program Reproduction Agents

> ProgramGym builds 1,648 verifiable program-reproduction environments and a role-isolated agent workflow, boosting a 27B model from 34.87% to 56.25% tests passed, exceeding flagship models.

- **Source:** [arXiv](https://arxiv.org/abs/2610.13395)
- **Published:** 2026-10-12
- **Permalink:** https://picx.dev/p/1mY1KG
- **Whiteboard:** https://picx.dev/p/1mY1KG/image

## Summary

## Summary

- **ProgramGym** is a framework for scalable training environments for long-horizon program reproduction agents, consisting of two components: **Program-Env** (environment construction) and **Program-Agent** (agent workflow).
- **Program-Env** is a feedback-driven pipeline that automatically converts open-source repositories into verifiable training environments (reference executable, runtime sandbox, documentation, and black-box test suites), producing **1,648 environments** spanning C, C++, Go, and Rust.
- **Program-Agent** combines a role-isolated iterative workflow (Analysis → Implement → Review) with revision management for compilation validation and failure recovery, enabling sustained long-horizon development.
- Fine-tuning on Program-Env trajectories raises **Qwen3.8-27B** from 34.87% to 42.84% Tests Passed on ProgramBench; Program-Agent further raises base and SFT models to **55.71%** and **56.25%**, both exceeding flagship models like Claude Opus 4.7 (50.90%).
- The framework will be released to the community to support research on coding agents that build programs from scratch.

---

## Introduction and Theoretical Foundation

### Background
Current coding agents typically handle two task settings:
- **Issue-resolution** (e.g., SWE-bench): agent patches a given repository based on an issue.
- **Repository generation** (e.g., NL2Repo-Bench): agent implements a program from a requirements document.

Neither setting captures the *from-scratch* scenario where a software engineer must **discover what is required** and **decide how the system should be designed**. Program reproduction tasks (e.g., ProgramBench) address this by asking agents to rebuild a program's observable behavior from its usage documentation and a reference executable, without access to source code or network.

### Key Challenges
1. **Scaling verifiable training environments**: Building reference executables requires accommodating repository-specific toolchains and dependencies; black-box tests must cover diverse functionality while rejecting incorrect reproductions.
2. **Sustaining long-horizon development**: General-purpose agent harnesses do not keep exploration and implementation coordinated; agents may begin implementing before sufficient exploration and forget previously observed requirements as context grows.

### Formal Definition
The environment-construction pipeline is formally defined as:

$$
e _ {i} = \Phi (P _ {i}) = (B _ {i} ^ {\star}, D _ {i}, \mathcal {T} _ {i}, \mathcal {S} _ {i}), \qquad \mathcal {E} = \{e _ {i} \} _ {i = 1} ^ {N}.\tag{1}
$$

where:
- $P_i$: source-available codebase
- $B_i^{\star}$: execute-only reference program built from $P_i$
- $D_i$: usage documentation
- $\mathcal{T}_i$: black-box behavioral test suite
- $\mathcal{S}_i$: network-isolated runtime sandbox
- $\mathcal{E}$: collection of $N$ program-reproduction environments

The agent produces a candidate executable $\widehat{B}_i$, and the test suite $\mathcal{T}_i$ compares its externally observable behavior against $B_i^{\star}$.

---

## Methodology

### Program-Env: Environment Construction

**Stage 1 — Repository Collection:**
- Stratified search via GitHub Search API, partitioned by programming language and star-count buckets.
- Excludes archived/forked repositories, deduplicates, validates licensing, and excludes repos used by existing benchmarks.
- Yields $M = 30{,}459$ candidate repositories, each anchored to a fixed commit.

**Stage 2 — Executable and Sandbox Generation:**
- Uses Claude Code as the underlying agent harness with an **outer feedback loop**.
- The agent generates `download.sh` (online dependency preparation) and `compile.sh` (offline compilation).
- The loop proceeds only if both scripts succeed and `./executable` passes a loadability check; otherwise, diagnostics are returned for revision.
- The validated executable becomes $B_i^{\star}$; the prepared image becomes the runtime sandbox $\mathcal{S}_i$.

**Stage 3 — Black-Box Test Construction:**
- The dispatcher splits the codebase into functional modules and assigns each to a parallel subagent.
- Each subagent iteratively writes tests under structured feedback, organized into three documents in `Feedback Docs/`:
  - **Summary**: records progress, unresolved items, and rationale.
  - **Program analysis**: locates uncovered functionality via dynamic analysis and traces to public entry points via static analysis.
  - **Validation**: four checks — *pass check* (fails on $B_i^{\star}$), *double check* (inconsistent across runs), *dummy check* (accepts dummy implementation), *weak-assert check* (insufficient assertions).
- Subagents terminate at coverage threshold or round limit; suites are merged and re-validated.

### Program-Agent: Agent Workflow

**Role-Isolated Iterative Workflow:**
- Decouples into two logical roles (analyst and implementer) and three stages, each in an **independent session** to eliminate cross-stage memory contamination.

| Stage | Role | Key Actions |
|-------|------|-------------|
| **Analysis** | Analyst | Reads documentation, explores reference program, stores behavioral evidence in `analysis-initial/analysis.md` (read-only) |
| **Implement** | Implementer | Develops candidate programs based on frozen handoff artifacts, documents progress in `implementation-k/delivery.md` |
| **Review** | Analyst | Compares reference vs. candidate program, records verified discrepancies in `review-k/review.md` |

- **Iteration budget**: fixed budget of K implementation rounds, resolving premature termination and unbounded exploration.

**Revision Management:**
- Independent compilation as the gate for each round — exports a clean snapshot, compiles outside the workspace.
- Successful snapshots are archived in `Revisions` and forwarded to Review.
- Timeouts or failures trigger retries in fresh sessions; persistent failures roll back to the last validated revision.

---

## Empirical Validation / Results

### Experimental Settings
- **Environments**: 1,648 training environments across C, C++, Go, Rust, using DeepSeek-V4-Flash-0731 as the underlying model.
- **Test construction**: 5,918 subagents; mean coverage 72.99% (median 81.63%); 665,512 validated test cases (mean 404 per repository).
- **Training**: Qwen3.8-27B fine-tuned via ms-swift framework → Qwen3.8-27B-SFT.
- **Evaluation**: ProgramBench metrics — % Tests Passed, % Almost Resolved (≥95% tests), % Resolved (all tests, no cheating flag).

### Main Results (Table 2)

| Models | Agent Scaffolds | Tests Passed (%↑) | Δ | Almost Resolved (%↑) | Δ | Resolved (%↑) | Δ |
|---|---|---|---|---|---|---|---|
| **Flagship Models** (mini-SWE-agent) | | | | | | | |
| GPT 5 mini | mini-SWE-agent | 16.00 | - | 0.00 | - | 0.00 | - |
| Claude Haiku 4.5 | mini-SWE-agent | 30.00 | - | 0.00 | - | 0.00 | - |
| Gemini 3.1 Pro | mini-SWE-agent | 36.40 | - | 0.00 | - | 0.00 | - |
| GPT 5.4 | mini-SWE-agent | 37.70 | - | 0.00 | - | 0.00 | - |
| Claude Sonnet 4.6 | mini-SWE-agent | 47.50 | - | 1.00 | - | 0.00 | - |
| Claude Opus 4.7 | mini-SWE-agent | 50.90 | - | 3.00 | - | 0.00 | - |
| Claude Opus 4.7 (xhigh) | mini-SWE-agent | 55.10 | - | 4.50 | - | 0.00 | - |
| **DeepSeek-V4-Flash-0731** | | | | | | | |
| DS-V4-Flash-0731 | Claude Code | 57.68 | - | 4.50 | - | 0.00 | - |
| DS-V4-Flash-0731 | Program-Agent | 58.97 | 1.29 | 5.0 | 0.5 | 0.00 | 0.00 |
| **Qwen3.8-27B** | | | | | | | |
| Qwen3.8-27B | Claude Code | 34.87 | - | 3.50 | - | 0.50 | - |
| Qwen3.8-27B | Program-Agent | 55.71 | 20.84 | 5.50 | 2.00 | 0.50 | 0.00 |
| Qwen3.8-27B-SFT | Claude Code | 42.84 | 7.97 | 1.50 | -2.00 | 0.00 | -0.50 |
| Qwen3.8-27B-SFT | Program-Agent | 56.25 | 21.38 | 4.50 | 1.0 | 0.50 | 0.00 |

### Key Findings (RQ1–RQ4)

**RQ1 (Effectiveness):**
- Program-Env SFT improves Qwen3.8-27B from 34.87% → 42.84% (Δ=+7.97), exceeding GPT 5.4 (37.70%).
- Program-Agent raises Qwen3.8-27B to 55.71% and Qwen3.8-27B-SFT to 56.25%, both exceeding Claude Opus 4.7 (50.90%) and Gemini 3.5 Flash (53.60%).
- **Fully reproducing programs remains unsolved**: Resolved stays near 0% for all models.

**RQ2 (Environment Analysis):**
- 1,648 tasks across 8 functional categories (developer tools and CLI/system utilities being largest).
- Black-box interaction categories: I/O (99.24%), Arguments (82.82%), Help (68.77%), Configuration (40.35%), Subcommands (22.05%), TUI (12.10%).
- Mean coverage 72.99%, median 81.63%; total construction cost 136.77B tokens.

**RQ3 (Test-Time Scaling):**
- Tests Passed rises from 43.49% (K=1) → 54.13% (K=3) → 56.25% (K=5); most gains arrive by K=3.

**RQ4 (Agent Behavior):**
- Program-Agent increases `read` and `git` actions, reshaping agents toward exploration and revision, especially after Program-Env training.

---

## Theoretical and Practical Implications

### Theoretical Implications
- **Program reproduction from scratch** shifts the challenge from "implement a specified repository" to "iteratively discover what the program should do and decide how to rebuild it," more closely mirroring the work of experienced software engineers.
- The framework demonstrates that **scalable verifiable training environments** can be automatically constructed from repositories, addressing a critical bottleneck in training autonomous coding agents.
- The role-isolated workflow with revision management provides a principled solution to the **context-compression problem** in long-horizon agent tasks.

### Practical Implications
- **Training**: Program-Env provides reliable behavioral supervision (black-box tests) for fine-tuning, improving model performance by ~8 percentage points without changing the agent scaffold.
- **Inference**: Program-Agent's iterative workflow provides a substantial test-time scaling benefit (~20 percentage points for base models), suggesting that workflow design is as important as model scale.
- **Deployment**: The 1,648 environments across four languages offer a diverse, open resource for the community to develop and evaluate coding agents that build programs from scratch.

---

## Conclusion

ProgramGym addresses two fundamental challenges in training program reproduction agents:
1. **Program-Env** scales verifiable training environments through feedback-driven build repair and test refinement, yielding 1,648 environments across C, C++, Go, and Rust.
2. **Program-Agent** sustains long-horizon development through role-isolated workflows and revision management.

Key results:
- Fine-tuning on Program-Env trajectories raises Qwen3.8-27B from 34.87% to 42.84%.
- Program-Agent further raises models to 55.71% (base) and 56.25% (SFT), exceeding several flagship models.
- Additional iteration budgets continue to recover program behavior.

**Future directions** include releasing ProgramGym to the community to support further research on coding agents that discover requirements and design systems from scratch. The authors will release environments, construction code, and training artifacts for reproducibility.

---

_Markdown view of https://picx.dev/p/1mY1KG, served by PicX — AI-generated visual whiteboard summaries of research papers._