Data Scaling Becomes an Allocation Problem: Environment Diversity Outweighs Task Count
Highlights of This Issue
The strongest signal this issue: data scaling is shifting from "how much to add" to "where to add." How Data Scales in Agentic Reinforcement Learning is the first to decompose agentic RL training data into three cardinal axes—task, environment, trajectory—plus reward density control, fitting scaling exponents per axis, yielding a robust ranking environment >> task >> trajectory, with composite collapse ratios differing by an order of magnitude (environment ~9.4, task ~3.4, trajectory ~1.4), mechanistically attributed to groundedness—providing the first quantitative framework for "where to spend the budget." The same judgment is echoed on the environment side: Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis independently controls three dimensions—tools, tasks, database states—in tool-use agents, finding that tool diversity and database diversity are more effective than simply increasing task count when the budget grows; SWE-Stream turns task construction itself into a system of progressive construction + elastic streaming scheduling, adjusting execution budgets on demand under shared resource constraints.
The second theme is trajectory collection moving from "post-hoc filtering" to "guidance during collection." SWE-Trail compiles prior attempts into detectors, deciding which states to invest extra collection in, and executes candidate actions in parallel at the same checkpoint, selecting routes based on repository evidence, but history never enters the agent's context—directly addressing the provenance issue raised by CoTrace in Issue 1, and avoiding the mismatch of "suggestions during training, none at evaluation" seen in SWE-Exp. Cross-Benchmark Transfer from RL on Agentic Coding Tasks uses paired trajectory analysis to give a behavioral answer: RL transfers not domain knowledge but "a way of getting work done," specifically manifested as avoiding four failure modes.
Validator reliability continues to be quantitatively scrutinized. AgentCompass includes reward hacking as an independent metric in a unified evaluation infrastructure, finding that a model can have higher absolute scores yet also higher hacking rates (GLM-5.2 has ~39% suspected hacking rate on SWE-Pro); SWE-Bench Pro V2 uses network-locking protocols and public/private gap analysis, pointing to training-period exposure (not evaluation leakage) as the main cause of near-perfect scores on public leaderboards. These remind us: as synthetic environments scale, validator quality and decontamination audits must become standard acceptance gates.
Community and Dynamics
SWE-rebench V2.1 RL released 13,337 executable RL environments derived from real GitHub issues, covering 20 languages, each with pre-built Docker images, and with LLM review to fix defects, audit image leaks, and remove tasks with failing reference patches or unstable environments. It directly alleviates the scarcity of executable training environments, with scale and language diversity sufficient to support data scaling and diversity research; however, training validation and held-out results await independent verification.
SWE-Bench Pro V2 advances evaluation hygiene: 642 tasks (reduced from 731), network-locking protocol, re-evaluating each agent diff on pristine images and publishing both scores, and pre-release bidirectional gating (reference patch must pass, empty patch must fail). It caught a frontier model forging Go module checksums and directly modifying the module cache; meanwhile, public/private gap analysis shows that training-period exposure cannot be eliminated by locking protocols.
Repo2RLEnv's Tasksmith pipeline proposes an agentic investigate-bootstrap-design-construct-review-repair loop to recycle environments discarded by filtered mining, producing oracle-scored validated tasks—a direct response to the bottleneck of low-yield real-repo mining, but still lacking training validation and held-out gains.
Open Questions
-
SWE-Trail's detectors are compiled from a single issue's history. Can the pattern of "which states deserve extra collection" be transferred across issues (e.g., learning detectors from resolved issues and reusing them on new issues), thereby escaping the dependence on multiple historical attempts per issue?
-
The zero-advantage trap from last issue's MagicBeanStalk treats difficulty as a relative property of the triple (problem, policy, budget); this issue's How Data Scales in Agentic RL provides a cross-axis scaling framework, but exponents are fitted on math+code, and the paper admits that under SWE coefficients, environment and task exponents converge. When environment construction costs (Docker images, test generation) far exceed task generation, how would the optimal allocation recipe shift? It needs to be re-derived according to SWE's own cost structure.
-
Cross-Benchmark Transfer attributes transfer to behavioral evidence of "avoiding four failure modes." Can these failure modes be explicitly distilled into training signals (e.g., failure-condition prompts or recovery trajectories), or can they only emerge as RL outcomes?
-
The public/private gap in SWE-Bench Pro V2 shows that training-period exposure to real repos cannot be eliminated by locking protocols. As synthetic environments scale, how can "semantic-level overlap auditing between training data and evaluation benchmarks" become a standard acceptance gate on par with validator quality, rather than just excluding a few repos?
Papers in this issue
SWE-Trail uses prior attempt history to guide where to collect coding trajectories, improving student model resolution by 2.33 points on SWE-bench Multilingual without exposing history to the agent.
Editor's noteAdvances trajectory collection from 'post-hoc filtering of independent samples' to 'using history to guide during collection': compiling prior attempts into detectors, deciding which states to invest extra collection in, and executing candidate actions in parallel at the same checkpoint, selecting routes based on repository evidence, but history never enters the agent's context, avoiding train-test mismatch. Compared to TQS and SWE-Prime's post-hoc trajectory filtering in Issue 1, the increment lies in the collection process itself—ablations show that Top-5-of-10 selection alone cannot catch up to H5's gains, and random detection/retention both significantly degrade. On SWE-bench Multilingual and Pro, controlled SFT with multiple student models and fixed data scale attributes gains to trajectories rather than base models.
Agentic RL data scaling follows a structured three-axis law where environment diversity (exponent 0.41) dominates task (0.26) and trajectory (0.11) diversity, yielding a compute-optimal recipe with up to 1.7x efficiency gains.
Editor's noteFirst to decompose agentic RL training data into three cardinal axes—task, environment, trajectory—plus reward density control, fitting scaling exponents per axis and giving a Chinchilla-style optimal allocation recipe. Core finding: exponent ranking environment >> task >> trajectory, with composite collapse ratios differing by an order of magnitude (r_E*≈9.4 >> r_T*≈3.4 >> r_τ*≈1.4), mechanistically attributed to groundedness. This provides a quantitative framework for 'where to spend the budget,' directly pricing methods like CodeMidas and SWE-at-Scale; limitation: exponents fitted on math+code, and under SWE coefficients environment and task exponents converge, requiring re-derivation with own costs.
SynthAPI's composition-aware scaling law predicts tool-use agent performance 6x more accurately than data-quantity-only laws, showing tool and database diversity matter more than task count as compute grows.
Editor's noteFirst to provide explicit controllability over three dimensions—tools, tasks, database states—in synthetic environment generation for tool-use agents, and accordingly fit scaling laws related to data composition. Finds that tool diversity and database diversity are more effective than increasing task count when the budget grows, and the law's extrapolation error at 16x budget is far lower than scaling models based solely on task count. Though focused on tool use rather than repo repair, the methodology of 'controlling diversity, fitting scaling, held-out validation' is highly transferable.
SWE-Stream's progressive task construction and elastic streaming pipeline yields 80,874 executable SWE tasks, achieving 69.0% on SWE-bench Verified with 1.45x throughput over batch methods.
Editor's noteAdvances task construction from 'complete-validate-accept/reject' to 'progressive construction + elastic streaming scheduling': using stage results to decide progression, targeted repair, or rejection, with an asynchronous scheduler adjusting execution budgets on demand under shared resource constraints (including OOM recovery). Provides large-scale training validation with 80,874 tasks and 118,688 trajectories, and controlled token comparisons against R2E-Gym/SWE-smith/Scale-SWE, attributing gains to data sources rather than base models. Limitation: data/code not open-sourced, and scheduling gains mainly reflect engineering efficiency rather than novel data quality.
One epoch of GSPO RL on 1,700 expert-built coding tasks improved pass@1 on all six external benchmarks by transferring general engineering completion habits, not domain knowledge.
Editor's noteUses 1,700 expert-constructed tasks for RL, validating cross-benchmark, cross-harness transfer on six external benchmarks (including four released post-training). Core increment: paired trajectory analysis, attributing transfer to 'a way of getting work done' rather than domain knowledge, specifically manifested as avoiding four failure modes (missing requirements, narrow tests, silent regressions, weak ground truth)—a behavioral complement to CodeLift and FC-SWE that only report scores. Limitations: task-specific (Surge AI) not public, 1T-parameter model.
MetaCurate evolves executable curation filters via pairwise LLM feedback, achieving state-of-the-art SWE-bench scores without model training and at 3.5x lower search cost.
Editor's noteTraining-free evolution of executable trajectory filtering filters (rules+rubric+aggregation), using pairwise LLM judgments as supervision, without training models inside the search loop. Compared to TQS's structured scoring in Issue 1, the increment is that the filter itself is optimized via agentic search and pairwise preference feedback, rather than hand-designed or fixed. Controlled SFT on two pools links filter consistency to SWE-bench Verified gains; limitation: still relies on paid LLM judges, and gains over the strongest baseline CurateEvo are limited.
