# Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

> SynthAPI's composition-aware scaling law predicts tool-use agent performance 6x more accurately than data-quantity-only laws, showing tool and database diversity matter more than task count as compute grows.

- **Source:** [arXiv](https://arxiv.org/abs/2610.37648)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/UFlvb6
- **Whiteboard:** https://picx.dev/p/UFlvb6/image

## Summary

# Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

## Summary (Overview)

- **Novel controllable synthesis pipeline (SynthAPI)**: Introduces a pipeline that generates executable tool-use environments with explicit control over diversity across three axes: tools (apps), tasks, and database states (data profiles), enabling systematic study of how data composition affects downstream agent performance.

- **Scalability analysis at unprecedented scale**: The study generates 458,752 tasks across 64 apps (10,472 APIs) and trains 849 models across four model sizes (2B–27B), systematically varying data composition controls to derive scaling relationships.

- **Data composition-dependent scaling law**: Proposes a predictive scaling law (Equation 4.2) that accounts for tool diversity (A), database breadth (S), and database depth (D), achieving a held-out mean absolute error of **0.0285** vs. **0.1706** for a data-quantity-only law—a 6× improvement in extrapolation accuracy.

- **Key finding on data allocation**: As compute budgets grow, scaling tool diversity and database diversity becomes *more* effective than simply increasing the number of tasks, and increasing data complexity yields larger gains.

- **State-of-the-art transfer results**: Agents trained with the optimal recipe outperform prior large-scale tool-use data (Nemotron) by **30.5 percentage points** on average across benchmarks, and surpass per-benchmark baselines by 7.6–11.6 points on AppWorld Test-Challenge and 2.0–3.7 points on BFCL.

## Introduction and Theoretical Foundation

### Motivation

Large language models (LLMs) show promise as tool-use agents for complex, stateful environments, but progress depends on scaling high-quality, diverse training data. Multi-step tool interactions are scarce in internet text and require realistic tasks, diverse tools, and rich environment data.

### Challenges

- **Simulating tool responses** (e.g., Nemotron, SimSAT): LLM-simulated outputs struggle with complex state transitions and become unreliable as complexity grows.
- **Executable environments** (e.g., EnvScaler, EnvFactory): Prior work emphasizes quantity but leaves open *which properties* of generated data drive performance.

### Key Insight

Tool-use agent data varies along three dimensions:
1. **Tool diversity** (number of apps/APIs)
2. **Task diversity** (number of tasks per app)
3. **Database diversity** (number of initial states per task, distractor data amount)

Understanding how these properties affect generalization is essential for determining what data to generate as training scales.

### Theoretical Foundation

The work builds on scaling law literature (Kaplan et al., 2020; Hofmann et al., 2022) but extends it from language-model pre-training to *tool-use agent training*, predicting downstream task success from environment-generation factors rather than benchmark loss.

## Methodology

### SynthAPI Pipeline Architecture

The pipeline consists of two stages:

#### Stage 1: API Generation

1. **App metadata generation**: The builder LLM (Gemini-3.1 Pro) generates app name, description, taxonomy category (shopping, music, productivity, finance, fitness, messaging, utilities, social), and functionalities.

2. **Database schema generation** (`db.py`): Python `Enum` and `dataclass` definitions specifying data types for the app (e.g., products, inventory, users, orders for a shopping app).

3. **API layer generation** (`apis.py`): Python functions operating on the database schema. Each function takes a database object plus input arguments, reading or modifying instantiated data.

#### Stage 2: Task Generation

A task is defined as a tuple:
$$T = (x, v, Db^0, \pi^*)$$

where:
- $x$ = task instruction
- $Db^0$ = starting database state
- $v$ = verifier code (determines task completion)
- $\pi^*$ = oracle solver program

**Data profile construction**: The initial database state is built as:
$$Db^0 = Db^0_{\text{core}} \cup Db^0_1 \cup \ldots \cup Db^0_{D-1}$$

where $Db^0_{\text{core}}$ is the minimal state for task feasibility and $Db^0_i$ are distractor data batches.

**Filtering**: Tasks are rejected unless $v(Db^0) = 0$ and $v(\text{Exec}(\pi^*, Db^0)) = 1$.

### Controllable Data Composition

The task set is characterized by:
- **A** = number of apps (tool diversity)
- **T** = number of tasks per app (task diversity)
- **S** = number of initial databases per task instruction (database breadth)
- **D** = amount of distractor data per initial state (database depth)

Total tasks: $N = A \cdot T \cdot S$

### Training Setup

- **Expert**: GLM-5.2 collects trajectories (up to 4 attempts per episode, first successful attempt retained)
- **Students**: Qwen3.5 family (2B–27B), trained via SFT with next-token prediction on reasoning and action tokens
- **Data scale**: 5.9B total tokens (721M supervision tokens)
- **Evaluation**: AppWorld, BFCLv3 Multi-Turn, ASTRA-Bench, and SynthAPI-Holdout

### Normalized Score

For model $\theta^M$ at size $M$, the normalized score aggregates across $K$ diagnostic benchmarks:

$$P(\theta^M) = \frac{1}{K} \sum_{i=1}^{K} \frac{\text{Succ}_i(\theta^M) - \text{Succ}_i(\theta_0^M)}{\max_{\theta' \in \Theta_i} \text{Succ}_i(\theta') - \text{Succ}_i(\theta_0^M)}$$

## Empirical Validation / Results

### Task Property Analysis

#### Apps vs. Tasks Allocation

Fitted model (generalized logistic):
$$\hat{P}_{M,A}(N) = U \cdot \sigma\left(b + b_N \log_2 N\right)^\nu \tag{4.1}$$

**Key findings** (Figure 2a):
- At low budgets (64–512 tasks): $A = 8$ performs best
- At high budgets (≥4096 tasks): $A = 64$ is clearly best
- Performance ceiling increases with $A$; $A = 8$ saturates at substantially lower values
- Even at $T = 1024$, only 51% of tools are used per app (no task exhaustion)

#### Database Breadth (S)

**Key findings** (Figure 2b):
- 32–512 instances: $S = 1$ best
- 512–4096: $S = 2$ best
- Beyond 4096: $S = 4$ best
- Ceiling increases with $S$; learning same task across different states is valuable

#### Database Depth (D)

**Key findings** (Figure 2c):
- Large improvement from $D = 1, 2$ to $D = 4$ at high budgets
- Distractor data improves downstream scaling

### Scaling Law

**Data composition-dependent law**:
$$\hat{P}_M^1(A, T, S, D) = \sigma(c_0 + c_A \log_2(A \cdot S \cdot D)) \cdot \sigma(b_0 + b_T \log_2 T)^\nu \tag{4.2}$$

**Comparison** to data-quantity-only law $\hat{P}_M^0(N) = U\sigma(b_0 + b_N \log_2 N)^\nu$:

| Metric | Composition-aware | Quantity-only |
|--------|-------------------|---------------|
| Held-out MAE (top 25% FLOPs) | **0.0285** | 0.1706 |
| Extrapolation range | 16× FLOP budget | — |

The law reveals that app count, data profiles per task, and data complexity set the **performance ceiling**, while tasks per app control the **rate of progress** toward that ceiling.

### Benchmark Results

**Table 1: Unnormalized success rates**

| Approach | AppWorld (Test-C) 4B/9B/27B | ASTRA-Bench 4B/9B/27B | BFCL 4B/9B/27B | SynthAPI-Holdout 4B/9B/27B |
|----------|------------------------------|------------------------|-----------------|----------------------------|
| Zero-shot | 14.7/26.4/61.5 | 57.4/71.8/80.2 | 27.3/39.3/54.4 | 0.3/1.5/20.0 |
| Nemotron | 17.5/20.8/- | 54.4/67.7/- | 15.9/39.7/- | 1.0/0.5/- |
| SynthAPI-SubOpt | 38.4/53.0/73.8 | 71.1/74.2/77.3 | 44.9/47.5/59.6 | 51.3/56.0/64.0 |
| SynthAPI-Opt | 45.0/56.2/74.1 | 72.3/75.5/79.3 | 48.8/49.9/61.5 | 56.0/63.0/68.0 |
| SynthAPI-Big | **66.4/77.0/81.5** | **74.2/76.8/82.3** | **51.1/53.6/65.0** | 53.0/64.0/69.8 |

**Table 2: Comparison to prior baselines**

| BFCL | 4B | 8B | | AppWorld | 4B | 9B |
|------|-----|-----|-|----------|-----|-----|
| Zero-shot | 33.5 | 41.3 | | Zero-shot | 14.7 | 26.4 |
| Agent World Model | 40.8 | 42.3 | | ToolACE | — | — |
| EnvScaler | 45.0 | 51.9 | | ToolAlpaca | — | — |
| EnvFactory | 48.5 | 49.0 | | SimSAT | 58.8 | 65.4 |
| SynthAPI-Big | **50.5** | **55.6** | | SynthAPI-Big | **66.4** | **77.0** |

Key results:
- **SynthAPI-Opt vs. SubOpt**: Optimal recipe consistently outperforms suboptimal at fixed budget (especially large deltas in SynthAPI-Holdout and 4B model)
- **SynthAPI-Big vs. Nemotron**: +30.5 points average improvement; Nemotron underperforms zero-shot in 5/8 settings
- **AppWorld**: SynthAPI-Big exceeds SimSAT by 7.6 (4B) and 11.6 (9B) points on Test-C
- **BFCL**: Surpasses strongest baseline (EnvFactory at 4B, EnvScaler at 8B) by 2.0 and 3.7 points

## Theoretical and Practical Implications

### Theoretical Implications

1. **Data composition matters as much as scale**: The 6× improvement in extrapolation accuracy demonstrates that *what* data is generated matters as much as *how much*—a critical insight for the agent-training community.

2. **Composition-optimal allocation shifts with budget**: The finding that tool diversity and database diversity become increasingly valuable as budgets grow challenges the common practice of scaling task count alone.

3. **Ceiling vs. rate decomposition**: The scaling law's structure suggests a principled decomposition—diversity factors (A, S, D) determine the achievable ceiling, while task count (T) determines convergence rate toward that ceiling.

4. **Generalization of scaling laws**: Extends scaling law methodology from pre-training loss prediction to *downstream task success* prediction, connecting environment generation factors to agent generalization.

### Practical Implications

1. **Data generation guidance**: Provides concrete guidance for practitioners: at small budgets, concentrate tasks within fewer tools; at large budgets, prioritize tool diversity and multiple database states per task.

2. **Compute-efficient training**: The scaling law enables predicting performance at 16× larger compute budgets, allowing researchers to select optimal data compositions before committing expensive training runs.

3. **Executable environments over simulation**: The strong results over LLM-simulation approaches (Nemotron, SimSAT) validate database-backed executable environments as a more reliable path to scalable, complex tool-use supervision.

## Conclusion

### Main Takeaways

1. **SynthAPI** enables controllable synthesis of executable tool-use environments, allowing systematic study of data composition effects.

2. **Scaling law** (Equation 4.2) accurately predicts downstream performance from data composition factors, with 16× extrapolation capability.

3. **Optimal allocation shifts**: As budgets grow, prioritize tool diversity (A), database breadth (S), and database depth (D) over task count (T).

4. **Practical superiority**: The optimal recipe outperforms prior large-scale datasets (Nemotron) and per-benchmark baselines (SimSAT, EnvScaler, EnvFactory) across all evaluated benchmarks.

### Limitations

- No user-in-the-loop simulation (agents cannot ask clarifying questions)
- Limited information retrieval tasks with long, complex documents (lacks semantic depth of real user content)
- Scaling law validated on a single data pipeline; transferability to other pipelines unverified

### Future Directions

- Multi-app tasks (current tasks involve tools from a single app)
- User-in-the-loop interaction synthesis
- Integration of real-world documents and data for richer semantic content
- Validation of scaling law transfer across different synthesis pipelines and training settings

---

_Markdown view of https://picx.dev/p/UFlvb6, served by PicX — AI-generated visual whiteboard summaries of research papers._