Executable Environments Begin to Act as Dense Reward Signals
Highlights of This Issue
The clearest signal this issue is that training signals are shifting from "result filtering" to "process signals." At least four works are no longer satisfied with using pass/fail or trajectory quality to filter data; instead, they turn executable environments and tests themselves into dense rewards and curriculum sources. Counterfactual Rollout Replay uses forkable environments to replay counterfactual branches at decision points, using "realized trajectory terminal return − counterfactual branch return" as per-step advantage, without process labels or learned reward models; WEAVE organizes real PR history into executable dependency DAGs and introduces Graph Credit as a dependency-consistent dense verifiable reward; CodeLift turns the test suite itself into an adaptive curriculum, using behavior balancing for dynamic weighting and snapshot slicing to assign intermediate credit to effective actions before regression. All three share a judgment: the verifier is not just a scorer but a credit allocator that can feed back into training signals.
The second theme is the "zero-advantage trap" and the scalability of task synthesis. MagicBeanStalk systematically quantifies that about 88% of existing corpora provide no learning signal to the target policy under a fixed budget in RLVR, and proposes treating hard problems as "reasoning blueprint" seeds for top-down decomposition and bottom-up recursive synthesis, rather than simple filtering or variant generation—this insight also applies to code trajectories. SWE-at-Scale uses "reversed real bug fixes" as supervision to train a reusable bug injection model, replacing prompt-based injection with general LLMs; CodeMidas shifts the input for environment construction entirely from development artifacts to pure source code, covering 23 languages. Task synthesis is moving from "prompting LLMs to create tasks" toward "distilling task structures from code and historical fixes."
Utilization of failure trajectories is also advancing. ReTerm advances recovery training from proactive error injection (SFT only) to online RL based on the agent's own failure states, forking multiple independent recovery attempts from real failure states—this is a substantial advance over FC-SWE from the previous issue in failure-conditioned RL. On the community side, Repo2RLEnv unifies 23 existing generators under a verifier quality check layer, while Environment Scaling for Agentic RL proposes "reward completeness" as an acceptance gate independent of trajectory quality filtering. The quality of verifiers themselves continues to be scrutinized: SWEPatchTest uses the JVR metric to prove that single-reference patch evaluation masks many failures—the best configuration reproduces the reference transformation on 59.29% of tasks, yet JVR is only 9.21%.
Community and Developments
Repo2RLEnv is a system-oriented integration tool: it unifies 23 generators such as SWE-smith, R2E-Gym, SWE-gen, and CodeMidas into the Harbor task format, then applies control checks (oracle scores 1, no-op scores 0) and leakage detection for unified verifier quality control, releasing 21 datasets and 1930 tasks on the Hub. It does not provide new data generation algorithms itself, but standardizes the "generator–verifier" interface, offering direct practical value for scaling executable environment construction.
Microsoft's How to Learn from Negative Trajectories in Coding Agents proposes "inoculation": extracting factual errors from failure trajectories and injecting them as privileged information into the SFT user prompt, so the model treats failed actions as "behavior in that error context" rather than as objects to imitate. On SWE-bench Verified, with fewer than 4.5K trajectories, it improves 4.1% over positive-sample SFT, and ablations decompose the gain into "negative samples themselves" and "privileged information labeling" layers—complementing this issue's ReTerm failure-utilization theme, but error extraction relies on LLMs, so robustness against contamination needs independent verification.
Environment Scaling for Agentic RL is a practitioner survey that organizes environment construction into an eight-step pipeline of Generate→Build→Verify→Filter→Train, proposing reward completeness audits as an independent acceptance gate and recommendations like "report the funnel, not the container count." It has no new experiments, but its observations on "generator–verifier asymmetry" and "semantic-level leakage is rarely measured" directly relate to the attribution demands of several works this issue.
Open Questions
-
CRR's counterfactual advantage and CodeLift's snapshot slicing both come at the cost of additional environment executions (the latter averages 2.35 more test executions per rollout). Can the credit allocation gains of dense process rewards still hold under constrained compute budgets? Are there approximations that do not require replay or self-testing?
-
MagicBeanStalk's "zero-advantage trap" measures about 88% no-signal in mathematical corpora, with difficulty being a relative property of the triple (problem, policy, budget). Do code agent trajectories exhibit the same proportion? Can hard SWE problems serve as "reasoning blueprint" seeds for recursive synthesis like hard math problems?
-
WEAVE's Graph Credit and CodeLift's behavior balancing both derive dense rewards from test/patch structure, but the former allocates by dependency DAG and the latter by behavior-group dynamic weighting. Can "dependency-consistent credit" and "difficulty-adaptive weighting" be unified into a single reward framework?
-
As Environment Scaling for Agentic RL points out, semantic/embedding-level overlap between synthetic tasks and held-out benchmarks is rarely measured. Once Repo2RLEnv and CodeMidas scale environment construction, how can contamination audits (rather than merely excluding a few repositories) become a standard acceptance gate?
Papers in this issue
ReTerm trains terminal agents to recover from their own execution failures via self-generated recovery data and online RL, boosting Terminal-Bench 2.0 Pass@1 by 14.6 points.
Editor's noteAdvances recovery training from TermiGen's proactive error injection (SFT only) to online RL based on the agent's own failure states: forking multiple independent recovery attempts from real failure states and using a recovery task queue to preserve failure states across training steps, allowing recovery ability and task-solving ability to co-evolve. Compared to FC-SWE from Issue 1, the increment lies in branching multiple recovery paths from real failure states and using a queue to mitigate the problem of insufficient failure samples in a single batch; public data, code, and models, with clear attribution evidence from SFT/GRPO/ReTerm-GRPO ablations on Terminal-Bench 2.0 and SWE-bench Verified.
Counterfactual Rollout Replay improves software engineering agent training by using forkable environments to compute step-level advantages from counterfactual rollouts, boosting SWE-bench Verified pass@1 to 41.7% without process labels.
Editor's noteProposes a training-time credit allocation method: using forkable executable environments to replay counterfactual branches at selected decision points, with "realized trajectory terminal return − counterfactual branch return" as per-step advantage, requiring no manual process labels or learned reward models—a direct attempt to use executable environments as a source of dense process rewards. Compared to Tree-GRPO's group-relative advantage and Entropy-Guided Branching's failure trajectory repair, CRR preserves the realized trajectory and only rescoring original actions with counterfactuals, providing unbiasedness theory results for fixed-index comparisons and variance decomposition diagnostics. Equal-wall-clock comparisons and ablations isolating the credit allocation mechanism are exactly the attribution evidence valued in this direction; limitations include dependence on deterministic environments and replay compute costs.
CodeMidas automatically converts existing codebases into 5,545 verifiable RL training tasks, improving coding agent performance across all five external benchmarks without requiring issues, tests, or documentation.
Editor's noteShifts the input for environment construction entirely from development artifacts (issue/commit/test/documentation) to pure source code: generating behavioral specifications through agent exploration of implemented functionality, constructing tests based on original code execution, and introducing adversarial rollout leakage detection and de-review as post-rollout filtering. Compared to R2E-Gym and SWE-smith, it does not depend on commits or existing tests, covers 23 languages, and is the first large-scale environment construction pipeline with source code as the sole input. Ablations show high-quality 5k tasks outperform unfiltered 8k, with gains validated on multiple held-out benchmarks, meeting this direction's requirements for data attribution and contamination control.
WEAVE turns real multi-PR histories into dependency-structured code agent tasks, improving dependency recovery and training while exposing phantom credit in flat evaluation.
Editor's noteAdvances multi-PR dependency structure from coarse-grained aggregation to executable DAG task construction: automatically recovering dependency edges with repository-calibrated typed change loci, organizing related PRs into bounded DAGs instantiated as unified executable tasks, while using Graph Credit to provide dependency-consistent dense, verifiable rewards. Compared to SWE-Milestone's LLM-constructed DAGs and daVinci-Agency's explicit reference chains, WEAVE uses the DAG structure for both evaluation and RL rewards, validating gains in both SFT and RL, with ablations attributing gains to task construction rather than the base model. Limitations include a self-built benchmark not yet widely independently validated.
SWE-at-Scale reverses real-world bug fixes to train bug-inducing models, generating SWE tasks that are more realistic, diverse, and challenging than existing synthetic baselines.
Editor's noteThe core novelty is using "reversed real bug fixes" as training supervision to learn bug injection, rather than relying on prompt-based injection with general LLMs: explicitly training a reusable bug-inducing model and validating that sampling temperature and budget can scale bug type diversity in fixed code contexts (MBTE grows with sampling budget up to 200 generations without saturation). Compared to SWE-smith's LLM-Modify/Rewrite and BugPilot's FeatAdd, it distills bug patterns from historical fixes and transfers them to new code contexts; code and models are open-sourced. Limitations include Python-only, no training validation, and Pass@1 comparisons not controlling distribution differences between synthetic and real tasks.
MagicBeanStalk converts hard RLVR problems into calibrated training data via top-down decomposition and bottom-up synthesis, boosting math benchmarks by +8.6 points.
Editor's noteProposes and systematically quantifies the "zero-advantage trap": about 88% of existing mathematical corpora provide no learning signal to the target policy under a fixed budget, with difficulty being a relative property of the triple (problem, policy, budget). The core increment is treating hard problems as "reasoning blueprint" seeds, generating new problems through top-down decomposition (with effectivity testing) and bottom-up recursive synthesis (preserving blueprints, calibrating difficulty), rather than simple filtering or direct variant generation—compared to Issue 1's FrogNano online task synthesis, MBS explicitly uses the reasoning structure of hard seeds for structured decomposition and recombination. Attribution ablations compare full corpora, Goldilocks filtering, direct synthesis, and MBS, validating cross-policy transfer and held-out benchmarks; limitations include no open-sourced data and reliance on proprietary teacher models.
CodeLift's behavior-balanced test weighting and snapshot slicing deliver denser, better-calibrated rewards, improving SWE-bench Verified pass@1 by 3.67 points over binary reward baselines.
Editor's noteProposes behavior balancing and snapshot slicing mechanisms to turn test suites into adaptive curricula: dynamically weighting by behavior grouping of tests and patches to address diversity, progress-regression, and difficulty imbalances in pass-rate rewards; snapshot slicing uses agent self-test points to evaluate intermediate states, assigning credit to effective actions before regression. Compared to Issue 1's FC-SWE failure-conditioned RL and Issue 1's TQS structured trajectory scoring, CodeLift's increment lies in online dynamic weighting from the test behavior matrix and intermediate snapshot credit allocation, without requiring additional rollouts. Trained on R2E-Gym/SWE-rebench and validated on SWE-bench Verified/Pro, including paired bootstrap intervals and multi-base-model robustness; limitations include snapshot slicing requiring extra test executions and no open-sourced code.
