Summary

• ProgramGym is a framework for scalable training environments for long-horizon program reproduction agents, consisting of two components: Program-Env (environment construction) and Program-Agent (agent workflow).
• Program-Env is a feedback-driven pipeline that automatically converts open-source repositories into verifiable training environments (reference executable, runtime sandbox, documentation, and black-box test suites), producing 1,648 environments spanning C, C++, Go, and Rust.
• Program-Agent combines a role-isolated iterative workflow (Analysis → Implement → Review) with revision management for compilation validation and failure recovery, enabling sustained long-horizon development.
• Fine-tuning on Program-Env trajectories raises Qwen3.8-27B from 34.87% to 42.84% Tests Passed on ProgramBench; Program-Agent further raises base and SFT models to 55.71% and 56.25%, both exceeding flagship models like Claude Opus 4.7 (50.90%).
• The framework will be released to the community to support research on coding agents that build programs from scratch.

---

Introduction and Theoretical Foundation

Background

Current coding agents typically handle two task settings:

• Issue-resolution (e.g., SWE-bench): agent patches a given repository based on an issue.
• Repository generation (e.g., NL2Repo-Bench): agent implements a program from a requirements document.

Neither setting captures the from-scratch scenario where a software engineer must discover what is required and decide how the system should be designed. Program reproduction tasks (e.g., ProgramBench) address this by asking agents to rebuild a program's observable behavior from its usage documentation and a reference executable, without access to source code or network.

Key Challenges

1. Scaling verifiable training environments: Building reference executables requires accommodating repository-specific toolchains and dependencies; black-box tests must cover diverse functionality while rejecting incorrect reproductions.
2. Sustaining long-horizon development: General-purpose agent harnesses do not keep exploration and implementation coordinated; agents may begin implementing before sufficient exploration and forget previously observed requirements as context grows.

Formal Definition

The environment-construction pipeline is formally defined as:

$$e _ {i} = \Phi (P _ {i}) = (B _ {i} ^ {\star}, D _ {i}, \mathcal {T} _ {i}, \mathcal {S} _ {i}), \qquad \mathcal {E} = \{e _ {i} \} _ {i = 1} ^ {N}.\tag{1}$$

where:

• $P_i$: source-available codebase
• $B_i^{\star}$: execute-only reference program built from $P_i$
• $D_i$: usage documentation
• $\mathcal{T}_i$: black-box behavioral test suite
• $\mathcal{S}_i$: network-isolated runtime sandbox
• $\mathcal{E}$: collection of $N$ program-reproduction environments

The agent produces a candidate executable $\widehat{B}_i$, and the test suite $\mathcal{T}_i$ compares its externally observable behavior against $B_i^{\star}$.

---

Methodology

Program-Env: Environment Construction

Stage 1 — Repository Collection:

• Stratified search via GitHub Search API, partitioned by programming language and star-count buckets.
• Excludes archived/forked repositories, deduplicates, validates licensing, and excludes repos used by existing benchmarks.
• Yields $M = 30{,}459$ candidate repositories, each anchored to a fixed commit.

Stage 2 — Executable and Sandbox Generation:

• Uses Claude Code as the underlying agent harness with an outer feedback loop.
• The agent generates download.sh (online dependency preparation) and compile.sh (offline compilation).
• The loop proceeds only if both scripts succeed and ./executable passes a loadability check; otherwise, diagnostics are returned for revision.
• The validated executable becomes $B_i^{\star}$; the prepared image becomes the runtime sandbox $\mathcal{S}_i$.

Stage 3 — Black-Box Test Construction:

• The dispatcher splits the codebase into functional modules and assigns each to a parallel subagent.
• Each subagent iteratively writes tests under structured feedback, organized into three documents in Feedback Docs/:
  • Summary: records progress, unresolved items, and rationale.
  • Program analysis: locates uncovered functionality via dynamic analysis and traces to public entry points via static analysis.
  • Validation: four checks — pass check (fails on $B_i^{\star}$), double check (inconsistent across runs), dummy check (accepts dummy implementation), weak-assert check (insufficient assertions).
• Subagents terminate at coverage threshold or round limit; suites are merged and re-validated.

Program-Agent: Agent Workflow

Role-Isolated Iterative Workflow:

• Decouples into two logical roles (analyst and implementer) and three stages, each in an independent session to eliminate cross-stage memory contamination.

• Iteration budget: fixed budget of K implementation rounds, resolving premature termination and unbounded exploration.

Revision Management:

• Independent compilation as the gate for each round — exports a clean snapshot, compiles outside the workspace.
• Successful snapshots are archived in Revisions and forwarded to Review.
• Timeouts or failures trigger retries in fresh sessions; persistent failures roll back to the last validated revision.

---

Empirical Validation / Results

Experimental Settings

• Environments: 1,648 training environments across C, C++, Go, Rust, using DeepSeek-V4-Flash-0731 as the underlying model.
• Test construction: 5,918 subagents; mean coverage 72.99% (median 81.63%); 665,512 validated test cases (mean 404 per repository).
• Training: Qwen3.8-27B fine-tuned via ms-swift framework → Qwen3.8-27B-SFT.
• Evaluation: ProgramBench metrics — % Tests Passed, % Almost Resolved (≥95% tests), % Resolved (all tests, no cheating flag).

Main Results (Table 2)

Key Findings (RQ1–RQ4)

RQ1 (Effectiveness):

• Program-Env SFT improves Qwen3.8-27B from 34.87% → 42.84% (Δ=+7.97), exceeding GPT 5.4 (37.70%).
• Program-Agent raises Qwen3.8-27B to 55.71% and Qwen3.8-27B-SFT to 56.25%, both exceeding Claude Opus 4.7 (50.90%) and Gemini 3.5 Flash (53.60%).
• Fully reproducing programs remains unsolved: Resolved stays near 0% for all models.

RQ2 (Environment Analysis):

• 1,648 tasks across 8 functional categories (developer tools and CLI/system utilities being largest).
• Black-box interaction categories: I/O (99.24%), Arguments (82.82%), Help (68.77%), Configuration (40.35%), Subcommands (22.05%), TUI (12.10%).
• Mean coverage 72.99%, median 81.63%; total construction cost 136.77B tokens.

RQ3 (Test-Time Scaling):

• Tests Passed rises from 43.49% (K=1) → 54.13% (K=3) → 56.25% (K=5); most gains arrive by K=3.

RQ4 (Agent Behavior):

• Program-Agent increases read and git actions, reshaping agents toward exploration and revision, especially after Program-Env training.

---

Theoretical and Practical Implications

Theoretical Implications

• Program reproduction from scratch shifts the challenge from "implement a specified repository" to "iteratively discover what the program should do and decide how to rebuild it," more closely mirroring the work of experienced software engineers.
• The framework demonstrates that scalable verifiable training environments can be automatically constructed from repositories, addressing a critical bottleneck in training autonomous coding agents.
• The role-isolated workflow with revision management provides a principled solution to the context-compression problem in long-horizon agent tasks.

Practical Implications

• Training: Program-Env provides reliable behavioral supervision (black-box tests) for fine-tuning, improving model performance by ~8 percentage points without changing the agent scaffold.
• Inference: Program-Agent's iterative workflow provides a substantial test-time scaling benefit (~20 percentage points for base models), suggesting that workflow design is as important as model scale.
• Deployment: The 1,648 environments across four languages offer a diverse, open resource for the community to develop and evaluate coding agents that build programs from scratch.

---

Conclusion

ProgramGym addresses two fundamental challenges in training program reproduction agents:

1. Program-Env scales verifiable training environments through feedback-driven build repair and test refinement, yielding 1,648 environments across C, C++, Go, and Rust.
2. Program-Agent sustains long-horizon development through role-isolated workflows and revision management.

Key results:

• Fine-tuning on Program-Env trajectories raises Qwen3.8-27B from 34.87% to 42.84%.
• Program-Agent further raises models to 55.71% (base) and 56.25% (SFT), exceeding several flagship models.
• Additional iteration budgets continue to recover program behavior.

Future directions include releasing ProgramGym to the community to support further research on coding agents that discover requirements and design systems from scratch. The authors will release environments, construction code, and training artifacts for reproducibility.