CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

Summary (Overview)

• CodeMidas is an agentic pipeline that automatically constructs executable reinforcement learning (RL) environments for coding agents, using source code as its only task-specific input—requiring no issues, pull requests, commits, existing tests, or written documentation.
• The pipeline converts implemented functionality in existing codebases into behavioral task statements, execution-grounded tests, and containerized development environments, yielding 5,545 verifiable training tasks from 3,185 open-source codebases spanning 23 programming languages and 15 technical domains.
• Training MiMo-V2.5 on CodeMidas tasks with GRPO improves performance on all five external benchmarks: DeepSWE (+11.7 pp), ProgramBench (+17.0 pp), Terminal-Bench v2.1 (+8.5 pp), SWE-bench Pro (+4.1 pp), and RepoZero C2Rust (+11.3 pp).
• Ablation studies show that increasing high-quality task scale improves performance, and a smaller filtered dataset (5k) outperforms a larger unfiltered one (8k) by up to 4.59 percentage points.
• Trajectory analysis reveals that RL-trained agents exhibit better behaviors: increased codebase exploration, more code drafting (reasoning before edits), and more diverse self-verification—changes that generalize to held-out tasks.

---

Introduction and Theoretical Foundation

Background and Motivation

Large language models are increasingly capable of agentic coding—completing substantial real-world software work autonomously over long horizons. Effective RL for coding requires two critical ingredients:

1. Task diversity — supports generalization across different software tasks
2. Reliable rewards — reinforce correct work and reject incorrect solutions

The central challenge is turning real codebases into a broad supply of training tasks with trustworthy verifiers.

Limitations of Existing Approaches

Existing pipelines construct environments from development artifacts, which ties task creation to the coverage of recorded changes, tests, or documentation:

Table 1: Task-specific input requirements. ✓ = not required; ♦ = required by part of the pipeline; ✗ = required.

Key Insight

Open-source codebases provide a rich foundation: implemented functionality provides both the basis for a task and a candidate solution. Public interfaces and observable behavior define what an agent should implement, while executing the original code provides evidence for test expectations. The key requirements are:

• Making required behavior explicit in the task statement while leaving internal implementation choices open
• Building tests that reject incorrect solutions while still accepting alternative correct implementations

---

Methodology

CodeMidas consists of four modules, with agentic compute allocated to every stage of environment construction:

3.1 Task Design and Codebase Adaptation

An agent inspects codebase structure and build metadata to identify functionality with public entry points and observable outcomes. Supported interfaces include:

• Command-line tools — assessed through process outputs
• Pure library functions — assessed through return values
• Stateful library APIs — assessed through state changes across calls

For each candidate, the agent:

1. Traces public entry points and shared dependencies to define task scope
2. Removes the selected core implementation
3. Adjusts remaining code to form a coherent starting point
4. Retains the original implementation separately as a reference solution

The task statement defines inputs, observable behavior, and required public interfaces, while solvers choose their own internal helpers and algorithms.

3.2 Execution-grounded Test Construction

An agent maps behavioral requirements to test inputs and boundary cases, invoking public entry points in a reference copy of the codebase:

• CLI tools: command executions
• Pure functions: input-output cases
• Stateful APIs: sequences of calls (including ordering and cleanup)

Key principle: For outputs fixed by the statement, assertions use reference execution to establish expected values. For unspecified aspects, assertions check only stated constraints (e.g., enforcing exception type without fixing message wording).

A review step replaces restrictions unsupported by the statement (exact wording, incidental ordering, internal structure) with behavioral checks. Tasks are rejected if an assertion depends on a private symbol with no behavioral substitute.

3.3 Environment Preparation

• Dependency installation: From a uniform base container, an agent installs dependencies per project declarations
• Cleanup: Removes artifacts that could reveal the deleted implementation (compiled outputs, cached copies, construction-agent files)
• Execution consistency check: Each task runs in six fresh containers:
  • 2 with starting codebase → both must fail
  • 4 with reference solution → all must pass

This verifies the expected fail-to-pass transition and screens for unstable execution outcomes.

3.4 Post-rollout Environment Filtering

Three filtering steps using agent rollouts:

1. Leakage filtering: Adversarial rollouts attempt to exploit residual leakage (compiled artifacts, caches, installed copies). A separate review checks evidence against the reference solution and verifier; tasks with confirmed exploitable leakage are rejected.

2. Agreement on agent solutions: A coding agent attempts each task four times. A reviewing agent examines trajectories (submitted code, test outputs) against the statement, verifier, and reference solution, flagging:

   • False positives: incorrect implementations that pass tests
   • False negatives: correct implementations that fail tests
   • Tasks with verifier defects are rejected

3. Rollout outcome filtering: A frontier model makes several attempts per task. Only tasks with both successful and failed attempts are kept, filtering out tasks that are too easy or too hard (which may indicate defects).

3.5 Dataset Overview

The retained dataset: 5,545 tasks from 3,185 codebases, 23 languages, 15 technical domains

Language coverage (top 10):

Technical domains: Systems (17.4%), Web (14.6%), Dev Tools (13.6%), AI/ML (9.1%), Data Science (7.6%), Multimedia (7.1%), Specialized (6.9%), Cloud/DevOps (4.0%), Blockchain (3.6%), Desktop (3.3%), Security (3.3%), Game Dev (3.1%), Hardware (2.7%), Automation (2.0%), Mobile (1.6%).

Reference solution size: Median = 142 lines (IQR: 66–305). Patches touch ≥2 source files in 65.9% of tasks.

---

Empirical Validation / Results

4.1 Experimental Setup

• Base model: MiMo-V2.5
• RL algorithm: GRPO with binary execution rewards, batch size 32, 32 rollouts per task
• External benchmarks: SWE-bench Pro, DeepSWE v1.1, ProgramBench, RepoZero C2Rust, Terminal-Bench v2.1
• CodeMidas Val: 200 held-out tasks, 3 attempts each

4.2 Main Results

Performance gains from RL on CodeMidas:

Figure 5: Gains span repository repair, code translation, program construction, and terminal work.

Learning dynamics on CodeMidas Val: Pass rate rises from 35.0% to 44.7%, staying ~8–10 pp above the initial rate from step 40 onward, accompanied by longer trajectories.

5.1 Task Scale and Quality

Effect of task scale (1k, 3k, 5k high-quality pools vs. 8k vanilla unfiltered):

Figure 8: The full 5k dataset outperforms vanilla 8k by 0.59, 4.59, and 4.49 pp on the three evaluations. Even the 3k subset outperforms vanilla 8k, demonstrating the value of cleaning, execution checks, and post-rollout filtering.

5.2 Behavioral Changes and Generalization

Changes during RL training (early → late):

Table 2: Agents explore more, show greater overlap between written code and preceding reasoning, and execute more diverse verification commands.

Self-verification is associated with higher pass rates: Rollouts with agent-written checks have mean pass rates 4.2 pp higher (95% CI: 1.8–6.6) than those without.

Behavioral generalization to held-out tasks:

Table 3: Exploration increases across all benchmarks; interaction length varies by task type (increases for issue repair, decreases for whole-program construction).

---

Theoretical and Practical Implications

Theoretical Contributions

1. Source code as a scalable RL data source: CodeMidas demonstrates that implemented functionality—not just development artifacts—can be systematically transformed into verifiable learning environments. This decouples RL environment construction from the coverage of issues, PRs, commits, tests, or documentation.

2. Execution-grounded verification: By deriving test expectations from actual execution of reference code, CodeMidas addresses the test oracle problem (Barr et al., 2015), producing verifiers that reject incorrect solutions while accepting alternative correct implementations.

3. Quality over quantity in RL data: The finding that a filtered 5k dataset outperforms an unfiltered 8k dataset provides empirical evidence that environment reliability and training suitability matter more than raw task count.

Practical Implications

1. Broad language and domain coverage: With 23 languages and 15 technical domains, CodeMidas enables RL training for coding agents across diverse software ecosystems, unlike prior pipelines limited to 1–20 languages.

2. Fully automated pipeline: The agentic approach allocates compute to every construction stage, enabling large-scale dataset creation without human annotation.

3. Transferable behaviors: RL training on CodeMidas induces generalizable behavioral changes (exploration, drafting, self-verification) that appear on external benchmarks, suggesting that source-derived tasks teach broadly applicable coding strategies.

4. Verifier reliability: The multi-stage filtering (execution consistency, leakage checks, solution audits, rollout outcome filtering) provides a practical recipe for constructing trustworthy RL rewards at scale.

---

Conclusion

CodeMidas demonstrates that existing codebases serve as a valuable source of coding RL data. Key takeaways:

• Training on 5,545 source-derived tasks improves MiMo-V2.5 across all five external benchmarks, with gains spanning issue repair, whole-program construction, code translation, and terminal work.
• Scaling high-quality tasks improves performance: 1k → 3k → 5k pools yield progressively higher scores.
• Filtering matters: A smaller filtered dataset outperforms a larger unfiltered one, validating the cleaning, execution consistency checks, and post-rollout filtering steps.
• Behavioral improvements generalize: Agents explore codebases more, draft code more (reasoning before editing), and perform more diverse self-verification—changes that transfer to held-out tasks.

Future Directions

The paper suggests that source code itself can serve as a scalable foundation for constructing RL environments. Future work could explore:

• Scaling to even larger code corpora and more languages
• Extending to additional task types beyond CLI tools, pure functions, and stateful APIs
• Investigating how different filtering strategies affect downstream performance
• Combining source-derived tasks with development-artifact-based tasks for complementary coverage

CodeMidas applies a Midas touch to existing codebases, turning implemented functionality into effective training environments and providing a practical route to scaling coding RL from code itself.