Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

Summary (Overview)

• Novel controllable synthesis pipeline (SynthAPI): Introduces a pipeline that generates executable tool-use environments with explicit control over diversity across three axes: tools (apps), tasks, and database states (data profiles), enabling systematic study of how data composition affects downstream agent performance.

• Scalability analysis at unprecedented scale: The study generates 458,752 tasks across 64 apps (10,472 APIs) and trains 849 models across four model sizes (2B–27B), systematically varying data composition controls to derive scaling relationships.

• Data composition-dependent scaling law: Proposes a predictive scaling law (Equation 4.2) that accounts for tool diversity (A), database breadth (S), and database depth (D), achieving a held-out mean absolute error of 0.0285 vs. 0.1706 for a data-quantity-only law—a 6× improvement in extrapolation accuracy.

• Key finding on data allocation: As compute budgets grow, scaling tool diversity and database diversity becomes more effective than simply increasing the number of tasks, and increasing data complexity yields larger gains.

• State-of-the-art transfer results: Agents trained with the optimal recipe outperform prior large-scale tool-use data (Nemotron) by 30.5 percentage points on average across benchmarks, and surpass per-benchmark baselines by 7.6–11.6 points on AppWorld Test-Challenge and 2.0–3.7 points on BFCL.

Introduction and Theoretical Foundation

Motivation

Large language models (LLMs) show promise as tool-use agents for complex, stateful environments, but progress depends on scaling high-quality, diverse training data. Multi-step tool interactions are scarce in internet text and require realistic tasks, diverse tools, and rich environment data.

Challenges

• Simulating tool responses (e.g., Nemotron, SimSAT): LLM-simulated outputs struggle with complex state transitions and become unreliable as complexity grows.
• Executable environments (e.g., EnvScaler, EnvFactory): Prior work emphasizes quantity but leaves open which properties of generated data drive performance.

Key Insight

Tool-use agent data varies along three dimensions:

1. Tool diversity (number of apps/APIs)
2. Task diversity (number of tasks per app)
3. Database diversity (number of initial states per task, distractor data amount)

Understanding how these properties affect generalization is essential for determining what data to generate as training scales.

Theoretical Foundation

The work builds on scaling law literature (Kaplan et al., 2020; Hofmann et al., 2022) but extends it from language-model pre-training to tool-use agent training, predicting downstream task success from environment-generation factors rather than benchmark loss.

Methodology

SynthAPI Pipeline Architecture

The pipeline consists of two stages:

Stage 1: API Generation

1. App metadata generation: The builder LLM (Gemini-3.1 Pro) generates app name, description, taxonomy category (shopping, music, productivity, finance, fitness, messaging, utilities, social), and functionalities.

2. Database schema generation (db.py): Python Enum and dataclass definitions specifying data types for the app (e.g., products, inventory, users, orders for a shopping app).

3. API layer generation (apis.py): Python functions operating on the database schema. Each function takes a database object plus input arguments, reading or modifying instantiated data.

Stage 2: Task Generation

A task is defined as a tuple:

$$T = (x, v, Db^0, \pi^*)$$

where:

• $x$ = task instruction
• $Db^0$ = starting database state
• $v$ = verifier code (determines task completion)
• $\pi^*$ = oracle solver program

Data profile construction: The initial database state is built as:

$$Db^0 = Db^0_{\text{core}} \cup Db^0_1 \cup \ldots \cup Db^0_{D-1}$$

where $Db^0_{\text{core}}$ is the minimal state for task feasibility and $Db^0_i$ are distractor data batches.

Filtering: Tasks are rejected unless $v(Db^0) = 0$ and $v(\text{Exec}(\pi^*, Db^0)) = 1$.

Controllable Data Composition

The task set is characterized by:

• A = number of apps (tool diversity)
• T = number of tasks per app (task diversity)
• S = number of initial databases per task instruction (database breadth)
• D = amount of distractor data per initial state (database depth)

Total tasks: $N = A \cdot T \cdot S$

Training Setup

• Expert: GLM-5.2 collects trajectories (up to 4 attempts per episode, first successful attempt retained)
• Students: Qwen3.5 family (2B–27B), trained via SFT with next-token prediction on reasoning and action tokens
• Data scale: 5.9B total tokens (721M supervision tokens)
• Evaluation: AppWorld, BFCLv3 Multi-Turn, ASTRA-Bench, and SynthAPI-Holdout

Normalized Score

For model $\theta^M$ at size $M$, the normalized score aggregates across $K$ diagnostic benchmarks:

$$P(\theta^M) = \frac{1}{K} \sum_{i=1}^{K} \frac{\text{Succ}_i(\theta^M) - \text{Succ}_i(\theta_0^M)}{\max_{\theta' \in \Theta_i} \text{Succ}_i(\theta') - \text{Succ}_i(\theta_0^M)}$$

Empirical Validation / Results

Task Property Analysis

Apps vs. Tasks Allocation

Fitted model (generalized logistic):

$$\hat{P}_{M,A}(N) = U \cdot \sigma\left(b + b_N \log_2 N\right)^\nu \tag{4.1}$$

Key findings (Figure 2a):

• At low budgets (64–512 tasks): $A = 8$ performs best
• At high budgets (≥4096 tasks): $A = 64$ is clearly best
• Performance ceiling increases with $A$; $A = 8$ saturates at substantially lower values
• Even at $T = 1024$, only 51% of tools are used per app (no task exhaustion)

Database Breadth (S)

Key findings (Figure 2b):

• 32–512 instances: $S = 1$ best
• 512–4096: $S = 2$ best
• Beyond 4096: $S = 4$ best
• Ceiling increases with $S$; learning same task across different states is valuable

Database Depth (D)

Key findings (Figure 2c):

• Large improvement from $D = 1, 2$ to $D = 4$ at high budgets
• Distractor data improves downstream scaling

Scaling Law

Data composition-dependent law:

$$\hat{P}_M^1(A, T, S, D) = \sigma(c_0 + c_A \log_2(A \cdot S \cdot D)) \cdot \sigma(b_0 + b_T \log_2 T)^\nu \tag{4.2}$$

Comparison to data-quantity-only law $\hat{P}_M^0(N) = U\sigma(b_0 + b_N \log_2 N)^\nu$:

The law reveals that app count, data profiles per task, and data complexity set the performance ceiling, while tasks per app control the rate of progress toward that ceiling.

Benchmark Results

Table 1: Unnormalized success rates

Table 2: Comparison to prior baselines

Key results:

• SynthAPI-Opt vs. SubOpt: Optimal recipe consistently outperforms suboptimal at fixed budget (especially large deltas in SynthAPI-Holdout and 4B model)
• SynthAPI-Big vs. Nemotron: +30.5 points average improvement; Nemotron underperforms zero-shot in 5/8 settings
• AppWorld: SynthAPI-Big exceeds SimSAT by 7.6 (4B) and 11.6 (9B) points on Test-C
• BFCL: Surpasses strongest baseline (EnvFactory at 4B, EnvScaler at 8B) by 2.0 and 3.7 points

Theoretical and Practical Implications

Theoretical Implications

1. Data composition matters as much as scale: The 6× improvement in extrapolation accuracy demonstrates that what data is generated matters as much as how much—a critical insight for the agent-training community.

2. Composition-optimal allocation shifts with budget: The finding that tool diversity and database diversity become increasingly valuable as budgets grow challenges the common practice of scaling task count alone.

3. Ceiling vs. rate decomposition: The scaling law's structure suggests a principled decomposition—diversity factors (A, S, D) determine the achievable ceiling, while task count (T) determines convergence rate toward that ceiling.

4. Generalization of scaling laws: Extends scaling law methodology from pre-training loss prediction to downstream task success prediction, connecting environment generation factors to agent generalization.

Practical Implications

1. Data generation guidance: Provides concrete guidance for practitioners: at small budgets, concentrate tasks within fewer tools; at large budgets, prioritize tool diversity and multiple database states per task.

2. Compute-efficient training: The scaling law enables predicting performance at 16× larger compute budgets, allowing researchers to select optimal data compositions before committing expensive training runs.

3. Executable environments over simulation: The strong results over LLM-simulation approaches (Nemotron, SimSAT) validate database-backed executable environments as a more reliable path to scalable, complex tool-use supervision.

Conclusion

Main Takeaways

1. SynthAPI enables controllable synthesis of executable tool-use environments, allowing systematic study of data composition effects.

2. Scaling law (Equation 4.2) accurately predicts downstream performance from data composition factors, with 16× extrapolation capability.

3. Optimal allocation shifts: As budgets grow, prioritize tool diversity (A), database breadth (S), and database depth (D) over task count (T).

4. Practical superiority: The optimal recipe outperforms prior large-scale datasets (Nemotron) and per-benchmark baselines (SimSAT, EnvScaler, EnvFactory) across all evaluated benchmarks.

Limitations

• No user-in-the-loop simulation (agents cannot ask clarifying questions)
• Limited information retrieval tasks with long, complex documents (lacks semantic depth of real user content)
• Scaling law validated on a single data pipeline; transferability to other pipelines unverified

Future Directions

• Multi-app tasks (current tasks involve tools from a single app)
• User-in-the-loop interaction synthesis
• Integration of real-world documents and data for richer semantic content
• Validation of scaling law transfer across different synthesis pipelines and training settings