# How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies

> Agentic RL data scaling follows a structured three-axis law where environment diversity (exponent 0.41) dominates task (0.26) and trajectory (0.11) diversity, yielding a compute-optimal recipe with up to 1.7x efficiency gains.

- **Source:** [arXiv](https://arxiv.org/abs/2610.23946)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/yM8Sgn
- **Whiteboard:** https://picx.dev/p/yM8Sgn/image

## Summary

## Summary (Overview)

- **Key Contribution**: This paper reframes agentic reinforcement learning (RL) data scaling from a scalar abstraction (token count) to a structured design space with three cardinality axes—task, environment, and trajectory—plus a reward-density control variable. The authors fit per-axis scaling laws and derive a compute-optimal allocation recipe.

- **Central Finding**: Across 21 settings (model sizes, families, domains, RL algorithms), the scaling exponents robustly order as **environment > task > trajectory** (e.g., $\alpha_E = 0.41$, $\alpha_T = 0.26$, $\alpha_\tau = 0.11$ on Qwen3-8B math+code). When environment diversity is decomposed, world diversity ($\alpha_W = 0.44$) drives the advantage, while environment-structure diversity ($\alpha_V = 0.28$) is comparable to task diversity.

- **Synthesis Pricing**: Using fitted laws as a common ruler, the paper measures synthesis method efficiency via a per-axis discount factor $\kappa \in [0,1]$ and identifies critical synthetic-to-real collapse ratios that differ by an order of magnitude: $r_E^\star \approx 9.4 \gg r_T^\star \approx 3.4 \gg r_\tau^\star \approx 1.4$.

- **Compute-Optimal Recipe**: Solving budget allocation as constrained optimization yields the scaling laws $D_E^\star \propto C^{0.34}$, $D_T^\star \propto C^{0.27}$, $D_\tau^\star \propto C^{0.13}$, $N^\star \propto C^{0.26}$, validated at held-out scale with up to **1.7× compute efficiency** over the dominant "scale-the-trajectories" practice.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Pretraining scaling laws (Chinchilla [16], Kaplan et al. [19]) treat training data as a one-dimensional quantity—token count. Recent agentic RL works (DeepSeek-R1 [13], ScaleRL [20], Mai et al. [30]) inherited this scalar abstraction, plotting performance against rollout tokens or prompt counts as if interchangeable.

However, agentic data is fundamentally different: a single "data point" simultaneously encodes a **task specification**, an **environment** (API surface, dynamics), a **trajectory** of actions and tool feedback, and a **reward signal**—each generated by different pipelines at different costs. Modern synthesis pipelines make data effectively unbounded, shifting the central question from *"how much data to collect"* to *"where to spend the synthesis budget."*

The paper demonstrates this concretely: two GRPO runs on Qwen3-8B with identical total tokens and FLOPs, differing only in budget allocation across axes, reach success rates differing by **14 percentage points**. A single-axis power law cannot fit both.

### Theoretical Foundation

The authors define the training corpus as:
$$\mathcal{D} = \left( \mathcal{S}_T, \mathcal{S}_E, \{(t_i, e_i, \tau_i)\}_{i=1}^n, \rho \right)$$

where $\mathcal{S}_T$ is the set of training tasks, $\mathcal{S}_E$ is the set of environment instances, each trajectory $\tau_i$ is paired with a task-environment pair $(t_i, e_i)$, and $\rho$ is the reward function.

The data tensor induces **three cardinality axes and one control variable**:

1. **Task axis** $D_T$: number of unique task specifications (deduplicated by sentence-embedding cosine distance > 0.15)
2. **Environment axis** $D_E$: number of distinct environment instances (identified by canonical hash over tool/API surface, state schema, transition semantics, verifier interface)
3. **Trajectory axis** $D_\tau$: average rollouts per covered task-environment cell, $D_\tau = \frac{1}{|\Omega|} \sum_{(t,e) \in \Omega} n_{t,e}$
4. **Reward control** $D_R$: categorical sparse/dense control (dense provides ≥3 verifiable intermediate signals per trajectory)

The paper proposes an **additive power-law functional form**:
$$L(D_T, D_E, D_\tau) - L_\infty = A \cdot D_T^{-\alpha_T} + B \cdot D_E^{-\alpha_E} + C \cdot D_\tau^{-\alpha_\tau} + I(D_T, D_E, D_\tau) \tag{1}$$

where $L_\infty$ is the irreducible loss and $I$ captures cross-axis interactions. Since RL with verifiable rewards has no smooth loss, they use $L \equiv -\log p_{\text{success}}$ following [20, 47].

---

## Methodology

### Experimental Testbeds

Five domains stress-test domain invariance:

| Domain | Training | Evaluation |
|--------|----------|------------|
| Math+Code (TIR) | Synthetic math problems (ZeroTIR recipe) | AIME24/25, MATH500, AMC23 |
| Code agents (SWE) | SWE-Gym-style environments | SWE-bench Verified |
| Web agents | WebArena-Lite + BrowserGym | Task-held-out |
| UI/Tool agents | AppWorld, ALFWorld | World-held-out |
| Search agents (robustness) | Wikipedia retrieval + web-search API | Bamboogle, multi-hop split |

### Models and Training

- **Size sweep**: Qwen3-4B/8B/14B (Qwen3-8B default)
- **Model families**: Llama-3.1-8B, Qwen2.5-7B
- **RL algorithms**: GRPO (group size 8, no critic) and PPO (separate critic)
- **Framework**: verl with VerlTool for multi-tool rollout
- All results are means over 3 seeds with bootstrap 95% CIs

### Per-Axis Scaling Protocol

For each axis, the other two are held at calibration-sweep values $(D_E, D_T, D_\tau) = (256, 10240, 8)$, sweeping the target axis over 6–8 nested grid points spanning 32×. Fits use $L = KD^{-\alpha} + L_\infty$ via nonlinear weighted least squares, profiling $L_\infty$ over a grid and solving the inner log-linear problem in closed form.

### Environment Diversity Decomposition

| Level | What varies | Example | Exponent |
|-------|------------|---------|----------|
| L0: content | seed/initial state within same repository | new AIME problem | $0.07 \pm 0.02$ (placebo) |
| L1: structure | tool/API surface, state schema, semantics | stateless vs. persistent REPL | $\alpha_V = 0.28 \pm 0.03$ |
| L2: world ($D_W$) | repository, corpus, app, site | unseen repository | $\alpha_W = 0.44 \pm 0.04$ |

### Synthesis Pricing Framework

The paper posits:
$$L(D_r, D_s) = f\left(D_r + \kappa(r) \cdot D_s\right), \quad r \equiv D_s / D_r \tag{3}$$

where $f$ is the fitted single-axis law and $\kappa(r) \in [0,1]$ is the real-data-equivalent yield. For each method, total $D = D_r + D_s$ is held constant while sweeping $r \in \{0, 0.25, 1, 4, 16\}$. The critical collapse ratio $r^\star$ is defined as the ratio at which normalized success falls 5% below the real-only baseline.

### Compute-Optimal Optimization

The cost model is linear and axis-additive:
$$\Phi_{\text{compute}}(N, D_E, D_T, D_\tau) = \frac{c_E}{\kappa_E} D_E + \frac{c_T}{\kappa_T} D_T + \frac{c_\tau}{\kappa_\tau} D_\tau + c_N N \tag{5}$$

The constrained optimization becomes:
$$\min_{N, D_E, D_T, D_\tau} K_E D_E^{-\alpha_E} + K_T D_T^{-\alpha_T} + K_\tau D_\tau^{-\alpha_\tau} + K_N N^{-\alpha_N} + L_\infty \quad \text{s.t.} \quad \Phi_{\text{compute}} \leq C \tag{6}$$

KKT conditions yield per-variable scaling:
$$D_i^\star \propto \left(\frac{\kappa_i \alpha_i K_i}{c_i}\right)^{\frac{1}{\alpha_i + 1}} \cdot \lambda^{-\frac{1}{\alpha_i + 1}} \tag{7}$$

---

## Empirical Validation / Results

### Per-Axis Scaling Exponents

The fitted exponents (Qwen3-8B, math+code):
$$\alpha_E = 0.41 \pm 0.03, \quad \alpha_T = 0.26 \pm 0.02, \quad \alpha_\tau = 0.11 \pm 0.02 \tag{2}$$

The ordering $\alpha_E > \alpha_T > \alpha_\tau$ holds with non-overlapping 95% CIs at every model size, across all 18 (model×algo×domain) cells, and in all 21 settings including Llama-3.1-8B (0.36, 0.23, 0.09), Qwen2.5-7B (0.38, 0.24, 0.10), and the search domain (0.42, 0.25, 0.12). Paired bootstrap differences: $\alpha_E - \alpha_T = 0.148$ [0.109, 0.187], $\alpha_T - \alpha_\tau = 0.150$ [0.116, 0.184].

### Fitted Exponents Across Settings (Table 2 excerpt)

| Model | Algo. | Domain | $\alpha_E$ | $\alpha_T$ | $\alpha_\tau$ |
|-------|-------|--------|-----------|-----------|--------------|
| Qwen3-4B | GRPO | Math+Code† | 0.38 ± 0.04 | 0.23 ± 0.03 | 0.10 ± 0.02 |
| Qwen3-4B | GRPO | SWE | 0.36 ± 0.05 | 0.21 ± 0.04 | 0.09 ± 0.03 |
| Qwen3-8B | GRPO | Math+Code† | 0.41 ± 0.03 | 0.26 ± 0.02 | 0.11 ± 0.02 |
| Qwen3-8B | GRPO | UI/Tool | 0.45 ± 0.03 | 0.27 ± 0.02 | 0.13 ± 0.02 |
| Qwen3-14B | GRPO | Math+Code† | 0.43 ± 0.03 | 0.27 ± 0.02 | 0.12 ± 0.02 |

†No separable world axis: α is a structure exponent.

### Cross-Axis Interactions

At low $D_E$, scaling $D_T$ yields almost no gain. A multiplicative correction $I = -\gamma \cdot (D_E D_T)^{-\beta}$ improves $R^2$ from 0.93 to 0.98 ($\gamma = 0.18$, $\beta = 0.21$). **Key finding**: environment diversity gates the value of task diversity.

### Synthesis Pricing Results (Table 3 excerpt)

| Method | Primary axis | $\kappa@r=1$ | $r^\star$ | Cost/unit |
|--------|-------------|-------------|-----------|-----------|
| AWM | $D_E$ | 0.71 ± 0.05 | 9.4 ± 0.9 | 4.10 USD/env |
| GenEnv | $D_E$ | 0.66 ± 0.06 | 8.1 ± 1.0 | 3.80 USD/env |
| AutoPlay | $D_T$ | 0.54 ± 0.04 | 3.1 ± 0.4 | 0.18 USD/task |
| SCA | $D_T$ | 0.51 ± 0.05 | 2.8 ± 0.4 | 0.21 USD/task |
| SWiRL | $D_\tau$ | 0.34 ± 0.06 | 1.4 ± 0.3 | 0.02 USD/traj |
| TDScaling | $D_\tau$ | 0.42 ± 0.05 | 1.7 ± 0.3 | 0.02 USD/traj |

Axis membership explains 97% of κ variance (ANOVA $F(2,5) = 76.6$, $p < 0.001$).

### Compute-Optimal Recipe

The fitted recipe:
$$D_E^\star \propto C^{0.34}, \quad D_T^\star \propto C^{0.27}, \quad D_\tau^\star \propto C^{0.13}, \quad N^\star \propto C^{0.26} \tag{8}$$

Exponents sum to ≈1.00. The combined data exponent (0.74) exceeds Chinchilla's 0.5, with model size correspondingly smaller (0.26 vs. 0.5), indicating agentic RL benefits more from data variety than raw model scale.

### Held-Out Validation

At $C_{\text{test}} = 32C_0$ on Qwen3-14B (held out in both compute and model size):
- **Predicted**: 51.7% ± 1.6%
- **Measured**: 51.3% ± 1.4% (within prediction interval)
- **Naive recipe**: 42.9% ± 1.3%
- **Chinchilla-transferred**: 47.3% ± 1.5%

The recipe achieves **1.4×–1.7× compute efficiency** over baselines. The recipe is robust: exponents shift < ±0.04 on SWE/web data, < 0.05 under ±50% cost perturbations.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Reframing scaling laws**: The paper demonstrates that the scalar abstraction of data scaling is insufficient for agentic RL. The data axis must be decomposed into structured design dimensions, each with its own scaling exponent and synthesis discount.

2. **Mechanistic explanation via groundedness**: The collapse ratio ordering aligns with how strongly each axis is grounded by external truth—environments are checked by interpreters at every step, tasks only by verifiable rewards, trajectories only by the model's own distribution (the agentic analog of recursive collapse in pretraining [9, 41, 46]).

3. **Prerequisite structure**: Environment diversity gates the value of task diversity—a structural finding that explains why task-only synthesis pipelines saturate early.

### Practical Implications

1. **Allocation over quantity**: The central practical claim is that trajectory resampling ranks last in both regimes (synthesis and world acquisition). Pipelines should shift budget toward environment and task diversity.

2. **Cost-aware decision making**: The paper provides a framework for pricing synthesis methods on a common scale, enabling practitioners to compute whether purchasing new worlds (α_W steepest) or synthesizing tasks (23× cheaper per unit) is the better investment.

3. **Regime-dependent caveats**: The environment axis loses its top rank when fully loaded cost per environment exceeds ~3.4× the measured maximum (11.8 USD-equivalent on SWE). Under SWE coefficients, environment and task exponents become indistinguishable (0.28 vs. 0.29).

---

## Conclusion

The paper argues that agentic RL data is not a scalar but a structured design space whose axes carry distinct scaling exponents and synthesis discounts. The three tightly coupled contributions—per-axis decomposition, synthesis pricing via discount factors, and compute-optimal allocation—together reframe the central question from "how much data to collect" to "where to spend the synthesis budget."

**Key takeaways**:
1. The exponent ordering $\alpha_E > \alpha_T > \alpha_\tau$ is robust across models, algorithms, domains, and families
2. Synthesis collapse ratios differ by an order of magnitude across axes, mirroring the exponent ordering
3. The validated compute-optimal recipe achieves up to 1.7× compute efficiency over dominant practice

**Future directions**:
- (i) Extending the law with a continuous, fully orthogonal treatment of the reward axis and richer interaction terms
- (ii) Verifying the recipe at frontier model scale
- (iii) Coupling axis-level decomposition with curriculum methods that schedule within an axis

---

_Markdown view of https://picx.dev/p/yM8Sgn, served by PicX — AI-generated visual whiteboard summaries of research papers._