Environment Evolution Loops: Cheap Agent Feedback Replaces Expensive Rollouts
Highlights of This Issue
The clearest signal this issue is that environment construction is shifting from a linear pipeline of "validate once and done" to a closed loop of "continuously evolve with cheap feedback." Three independent works advance "making trainable environments" into "evolving trainable environments" from different angles. RepoFactory is the first to drive an environment evolution loop purely with agent metrics (validity × structural coupling × implementation scope), iteratively revising oracles, specifications, and tests without solver rollouts or downstream training, recovering repositories that were previously discarded. Ablations show the evolution contributes 5.1 points of NL2Repo gain. POSEIDON decomposes scientific workflows into four complementary task types (reproduction, subprocess, completion, correction) and uses a recursive generator-verifier loop to recover 49.4% of initially rejected environments, producing 9.3 times more environments than D3-Gym under matched budgets. Together, they show that the bottleneck in environment construction is not "generation" but "recovery and evolution"—echoing the progressive construction in SWE-Stream (Issue 4) and the "repair broken tasks" path in SWE-rebench V2.1 RL (Issue 5).
The second theme is "difficulty" becoming an explicitly controllable and measurable axis. BuggyMLE proposes the Code Complexity Index (CCI), a difficulty metric that predicts benchmark solve rates purely from code structure (R²=0.951), and uses it to construct difficulty-graded curricula via bug injection, reducing the hardest tier solve rate from 95% to 9% across 14 models. SWE-Growth uses "teacher-student task-level capability gap" as an explicit curriculum routing signal, quantifying the silent problem of synthetic tasks under binary rewards—79.4% of 63 student groups have no group-level relative advantage—and revives these silent tasks with reliable successes from a frozen teacher. This echoes the difficulty axis in How Data Scales (Issue 4): difficulty is no longer a byproduct of "post-hoc filtering" but an explicit control variable at "construction time."
The third theme continues the focus on reward reliability from the previous issue but advances the perspective from "evaluating verifiers" to "causal attribution observable during training." Optimistic Verifiers Get Hacked is the first to decompose verifier degradation into seven semantic axes in a setting where process-level exact rewards are computable, and records stepwise differences between exact and proxy rewards in parallel during training—advancing reward hacking from post-hoc evaluation in VerifierBench (Issue 5) to per-rollout observability. When Terminal-Agent Training Stalls systematically decomposes failure modes in terminal agent training data generation and verification into four often-confused dimensions (structural validity, verifier fidelity, policy solvability, infrastructure reliability) with quantitative evidence: Docker heredoc parsing errors account for 74%, and concurrent containers cause evaluation scores to plummet from 76.8% to 4.6%.
Community and Developments
rlgate offers a pre-deployment verifier audit tool: it proactively injects adversarial exploits to detect weak verifiers, leaked answers, and forgeable graders, and categorizes evidence into "proven exploits," "inspectable facts," and "model judgments." It reproduced all 8 of 8 git leak tasks on SWE-bench Verified and demonstrated that a ten-line conftest.py can make a do-nothing patch pass—the rebuilt :v2 image fixed the leak but not the scoring path. Limitations include 44% static recall and no training validation; read it as a diagnostic tool rather than a data generation method.
Harness adapters propose a recipe for converting public agent trajectories into RL environments: replay file writing, detection work units, and applying oracle/NOP/mutation gating before training—a path to construct environments from existing trajectories rather than generating new tasks, with gating mechanisms directly addressing environment hackability.
Training an LLM to use a Debugger reports a cautionary negative result: result-filtered SFT amplifies bad intermediate behaviors, while trajectory-level filtering cleans data but does not improve pass@k—because filtering cannot add solutions the teacher never found. This directly addresses the assumption about the "trajectory cleaning vs. coverage" trade-off in this direction and serves as a sanity check when designing synthetic trajectory pipelines.
Open Questions
-
RepoFactory's evolution loop uses proxy metrics (coupling, scope) instead of solver rollouts, but the mapping between proxy metrics and training gains remains indirect. When the evolution target shifts from "full repository reconstruction" to SWE patch-type tasks, does the product objective of validity × coupling × scope still hold, or are new proxy signals needed?
-
BuggyMLE's CCI predicts solve rates on ML repositories with R²=0.951. Can this structural difficulty metric transfer to SWE-bench-like tasks, and how does it complement or conflict with the difficulty axis in How Data Scales (Issue 4) and the teacher-student gap signal in SWE-Growth?
-
The seven semantic degradation axes in Optimistic Verifiers Get Hacked are validated on authorization policy domains. When the verifier is a test suite rather than an enumerable oracle, how can the attribution method of "fully observable exact rewards during training" be approximated? Can rlgate's exploit audit serve as a pre-training screening for such approximation?
-
SWE-Growth uses reliable successes from a frozen teacher to revive all-fail silent tasks. When the teacher-student capability gap is too large or too small, how should the curriculum routing signal be calibrated? Is there a unified difficulty-signal theory with the bluffing threshold condition in Honesty Is a Race (Issue 5)?
Papers in this issue
RepoFactory's proxy-driven environment evolution loop yields RepoHorizen-35B-A3B, a model improving coding-agent benchmarks by 22.6 points on NL2Repo and 8.5 points on average.
Editor's noteFirst to advance environment construction from "linear pipeline + post-hoc validation" to "evolution loop based on proxy metrics": using a triple product objective of validity × structural coupling × implementation scope, it iteratively revises oracles, specifications, and tests via low-cost code metric feedback without solver rollouts or downstream training, recovering repositories that were previously discarded. Ablations show the evolution contributes 5.1 points of NL2Repo gain, and controlled comparisons with the same teacher and harness across six public data sources (OpenSWE/R2E/SWE-smith) provide solid attribution evidence. Limitation: Python-only and NL2Repo is a self-built benchmark, but as an environment evolution methodology it is worth close reading.
BuggyMLE generates 6,600+ bug-injected ML repositories and a structural complexity metric (CCI) predicting solve rates with R²=0.951, enabling difficulty-graded benchmarks where a 9B model rivals frontier agents.
Editor's noteProposes the Code Complexity Index (CCI), a difficulty metric that predicts benchmark solve rates purely from code structure (R²=0.951), and uses it to construct difficulty-graded curricula via bug injection in ML repositories, reducing the hardest tier solve rate from 95% to 9% across 14 models. Compared to mutation-based synthesis in SWE-smith and SERA, the increment lies in explicit difficulty control and oracle attribution (distinguishing impactful bugs from decoys), supporting curriculum learning with per-bug verification rewards. Trained models are competitive with frontier models on held-out BuggyMLEBench-Lite, and OOD transfer to SWE-bench Lite and LiveCodeBench aligns with this direction's requirements for attribution and contamination control; limitation: data/code not open-sourced.
POSEIDON's automated pipeline generates 97,659 verifiable scientific-agent training environments from 7,110 repositories, and fine-tuning on them consistently boosts model performance across six benchmarks.
Editor's noteDecomposes scientific workflows into four complementary task types (reproduction, subprocess, completion, correction), producing 9.3 times more environments than D3-Gym under matched budgets, and introduces a recursive generator-verifier loop that recovers 49.4% of initially rejected environments. Controlled ablations (repository diversity, difficulty, task type removal) show each task type provides non-redundant supervision, and the scale of 97,659 environments makes data yield and quality evidence solid. Limitation: data/code not open-sourced, but the methodology is worth borrowing for any scientific agent training pipeline.
SWE-Growth uses a frozen teacher to guide self-play curricula, improving software agents to 31.7% on SWE-bench Pro and 17.2% on TerminalBench, surpassing student-only training.
Editor's noteUses "teacher-student task-level capability gap" as an explicit curriculum routing signal, simultaneously driving SFT demonstration selection, GRPO group filtering, and Injector rewards, forming a closed loop. Compared to historical-guided collection in SWE-Trail (Issue 4), the increment lies in first quantifying the silent problem of all-fail groups in repository-level self-play (79.4% of groups have no group-level relative advantage) and reviving these silent tasks with reliable successes from a frozen teacher. Ablations (SFT only, GRPO only, No refresh, Fixed-pair gap) and held-out TerminalBench 2.1 validation align with attribution preferences, and 101 seed tasks suffice for continuous improvement; limitation: teacher distillation and self-play contributions are not fully separated.
Optimistic process verifiers get hacked via padding and cause 2.3-3.2x more damage per unit of ranking corruption than pessimistic ones, while outcome-only verifiers starve training entirely.
Editor's noteFirst to decompose verifier degradation into seven semantic axes in a setting where process-level exact rewards are computable (executable authorization policy gates, 124,416 contexts), and records stepwise differences between exact and proxy rewards in parallel during training, advancing reward hacking from post-hoc evaluation in VerifierBench (Issue 5) to per-rollout observable causal attribution. Core findings—optimistic verifiers are hacked by padding attacks, outcome-only leads to training starvation, ranking-preserving is harmless—provide direct guidance for reward design in synthetic data pipelines. Limitation: task domain is authorization policies rather than code patching, and training scale is small, but attribution rigor aligns with this direction's taste.
Structural validity alone does not guarantee benchmark validity; task difficulty calibration is critical, as adding hard tasks restores informative RL training failures.
Editor's noteSystematically decomposes failure modes in terminal agent training data generation and verification into four often-confused dimensions (structural validity, verifier fidelity, policy solvability, infrastructure reliability) with quantitative evidence: difficulty calibration's impact on RL signals (E2 to E3 drops pass@2 from 81.3% to 20.6%), Docker heredoc parsing errors account for 74%, and concurrent containers cause evaluation scores to plummet from 76.8% to 4.6%. Compared to VerifierBench (Issue 5) which focuses on the verifier itself, this paper more comprehensively covers measurement confounds across the entire pipeline of generation, environment, and evaluation, and provides training experiments validating the importance of difficulty calibration.
RIPPLE internalizes prefix-only recovery guidance into a single policy via stage-specific credit, improving coding agents by 4.8 points across four benchmarks without deployment-time critics.
Editor's noteAdvances recovery training from failure-state recovery RL in ReTerm (Issue 2) to "three-stage (trigger, diagnose, correct) process-level credit assignment": using prefix-only teachers to provide evidence-anchored recovery supervision, and local validity gating to decide whether each recovery stage inherits positive or negative components of trajectory-level advantage. Snapshot-test F2P/P2F and attribution judges measure the actual effect of corrective actions rather than relying solely on terminal rewards, with leave-one-stage-out and matched-prefix ablations being clear. Limitation: data/code not open-sourced, and the gain confidence interval on Verified crosses zero.






