# Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agents

> Trajectory2Gym reframes tool-use agent training as constructing learnable interactions, not collecting successes, boosting benchmark means by up to 29.8 points across all model scales with SFT only.

- **Source:** [arXiv](https://arxiv.org/abs/2610.33568)
- **Published:** 2026-10-28
- **Permalink:** https://picx.dev/p/4jy7v7
- **Whiteboard:** https://picx.dev/p/4jy7v7/image

## Summary

## Summary (Overview)

- **Core Contribution**: Trajectory2Gym reframes trajectory generation for tool-use agents as the deliberate construction of learnable interaction experience rather than the collection of successful trajectories, based on two coupled requirements: **compositional coverage** (observation–action dependencies must be represented) and **decision grounding** (each supervised action must be supported by learner-visible interaction history).
- **Three-Stage Framework**: (1) Compositional interaction synthesis targets under-supported capability paths in native environments; (2) Grounded supervision masks decisions by visibility, execution validity, and workflow-order certification; (3) Structural-support learning allocates repeated support per learner scale with parent-normalized SFT.
- **Key Empirical Result**: Across five Qwen3 scales (1.7B to 30B-A3B), Trajectory2Gym improves Tau1, Tau2, and BFCL at every scale, raising the three-benchmark equal-weight mean by **16.9–29.8 percentage points** over public initializations with SFT only (no RL).
- **Scale-Beating Performance**: The post-trained 4B model surpasses the public 30B-A3B initialization on Tau2 (69.17 vs. 28.33); the 8B model has the highest three-benchmark mean of any evaluated checkpoint.
- **Key Findings**: Gains concentrate in stateful, multi-turn interaction; parent-normalized SFT provides the largest isolated ablation gain (8.04 points); more structural support is not automatically better (non-monotonic depth sensitivity).

---

## Introduction and Theoretical Foundation

### Background and Motivation

The paper addresses a fundamental flaw in standard tool-use agent training: **successful trajectories are not necessarily useful supervision**. Even successful interactions may:

- Underrepresent multi-step dependencies (compositional coverage failure)
- Contain actions unjustified by the agent's decision-time visibility (decision grounding failure)

The authors identify two specific challenges from their candidate pool (34,772 certified candidates):
- **Challenge 1 (Compositional Coverage)**: 40.9% of trajectories contain at most two supervised decisions; only 10.0% contain a certified decision after an authentic error.
- **Challenge 2 (Decision Grounding)**: Successful API execution alone does not establish that an action was justified by visible information. A collector may use an unseen identifier, and native tool replay may accept actions that contradict later user confirmations.

### Theoretical Formulation

The policy acts from a decision context that includes only agent-visible information:

$$c_{t} = \big(x, \mathcal{T}_{t}, V(H_{<t})\big), \qquad a_{t} \sim \pi_{\theta}(\cdot | c_{t}), \tag{1}$$

where $H_{<t}$ is the history preceding decision $t$, $V$ projects the history onto the agent-visible subspace, and $\mathcal{T}_{t}$ is the tool schema.

The supervised dataset for learner scale $s$ is:

$$\mathcal{D}_{s} = \{(\tau_{i}, \mathbf{m}_{i})\}_{i=1}^{N_{s}}, \qquad \mathbf{m}_{i} \in \{0, 1\}^{T_{i}}, \tag{2}$$

where $T_i$ is the number of assistant decisions and $m_{it}=1$ marks decision $t$ as a supervised target.

**Two required properties:**
1. **Compositional coverage**: training data must provide sufficient support for observation–action dependencies and multi-step structures
2. **Decision grounding**: each supervised action must be supported by the realized interaction and decision-time context $c_t$

### Key Proposition (Collector-Private Uncertainty)

When a demonstrated action depends on collector-private information after conditioning on $c_t$, a policy restricted to $c_t$ faces additional uncertainty (characterized formally as Proposition 1 in the paper). This formalizes why execution validity alone is insufficient for supervision eligibility.

---

## Methodology

### Stage 1: Compositional Interaction Synthesis

A shared capability vocabulary $\mathcal{K}$ covers semantic action roles (lookup, error, clarification, retry, confirmation, cancellation, update, etc.). An ordered path $p = (\kappa_1, \ldots, \kappa_\ell)$ composes these requirements. A generator $G$ constructs request and initial state, and a teacher $\pi_{\mathrm{T}}$ executes:

$$(x, s_{0}) \sim G(\cdot | p, \mathcal{E}), \qquad \tau \sim P_{\mathcal{E}}(\cdot | x, s_{0}, \pi_{\mathrm{T}}). \tag{3}$$

The support count of a path in a selected set $\mathcal{S}$:

$$n_{\mathcal{S}}(p) = \sum_{\tau \in \mathcal{S}} \mathbf{1}[p(\tau) = p]. \tag{4}$$

**Key properties:**
- Paths are targeted when support falls below the scale's target (deficit-driven synthesis)
- Uniform random selection misses rare paths (Proposition 3: a path represented once is missed with probability $1 - B/N$)
- Paths are certified only when action/observation dependencies appear in the executed log
- Teacher hints and private events never enter the student trace

### Stage 2: Grounded Supervision

**Target Eligibility**: Distinguishes native execution validity $e_t \in \{0,1\}$ from decision-time grounding $v_t \in \{0,1\}$, with local eligibility:

$$g_{t} = e_{t} v_{t}. \tag{5}$$

- For a tool call: $e_t = 1$ requires authenticated non-error execution; $v_t = 1$ requires tool availability to the agent and identifier-like arguments supported by preceding visible information
- Rejected calls and authentic errors remain visible context (no imitation loss); the grounded recovery becomes a target

**Workflow Completion Mask**:

$$m_{t} = R_{\tau} \mathbf{1}[q_{\tau} \preceq A_{\tau}] g_{t}, \tag{6}$$

where $R_\tau$ is native terminal success, $q_\tau$ is the task's ordered requirements, $A_\tau$ is the canonicalized sequence of relevant operations, and $\preceq$ denotes ordered subsequence inclusion. Only trajectories with $\sum_t m_t > 0$ enter the eligible pool.

### Stage 3: Structural-Support Learning

**Structural Selection**: For scale $s$, target depth $d_s$ (distinct trajectories per canonical path) grows with learner scale (1, 2, 3, 4, 8 from 1.7B to 30B-A3B). The deficit:

$$\delta_{s}(p; \mathcal{S}) = \left[d_{s} - n_{\mathcal{S}}(p)\right]_{+}. \tag{7}$$

Selection starts from a certified core, adds completing exemplars, closes uncovered atoms/paths, then reduces deficits—exactly maximizing capped support $\sum_p \min(n_{\mathcal{S}}(p), d_s)$ under cardinality budget (Proposition 2).

**Parent-Normalized SFT**: Each trajectory's contribution is weighted at the parent level:

$$\ell_{\theta}(\tau) = -\frac{1}{N_{\tau}}\sum_{u=1}^{N_{\tau}} \log \pi_{\theta}(y_u \mid h_u), \qquad \mathcal{L}_{\mathrm{SFT}}(\theta) = \frac{1}{|\mathcal{D}_s|}\sum_{\tau \in \mathcal{D}_s} \ell_{\theta}(\tau). \tag{8}$$

This prevents long multi-turn interactions from dominating the loss via rendering multiplicity (Proposition 4 shows per-example averaging weights a parent proportionally to its rendered children count).

---

## Empirical Validation / Results

### Experimental Setup

- **Models**: Qwen3 scales (1.7B, 4B, 8B, 14B, 30B-A3B) from public initialization, one epoch of full-parameter parent-normalized SFT
- **Benchmarks**: Tau1, Tau2, BFCL (Task Success / Function Calling)
- **Data**: 11.6k–19.9k trajectories per scale
- **Evidence standard**: Paired comparison with own initialization, same task population, native interface, decoding, simulator, scorer

### Table 1: Tau1/Tau2 Results (Pass@1, %)

| Model | Tau1 Avg. | Tau2 Avg. | Mean | Δ |
|---|---|---|---|---|
| Qwen3-1.7B | 8.52 | 23.33 | 15.93 | — |
| + Trajectory2Gym | 34.43 | 37.50 | 35.97 | **+20.04** |
| Qwen3-4B | 24.22 | 19.17 | 21.70 | — |
| + Trajectory2Gym | 44.22 | 69.17 | 56.70 | **+35.00** |
| Qwen3-8B | 27.87 | 25.00 | 26.44 | — |
| + Trajectory2Gym | 51.13 | 74.17 | 62.65 | **+36.21** |
| Qwen3-14B | 32.04 | 25.00 | 28.52 | — |
| + Trajectory2Gym | 58.04 | 80.83 | 69.44 | **+40.92** |
| Qwen3-30B-A3B | 44.39 | 28.33 | 36.36 | — |
| + Trajectory2Gym | 66.78 | 84.17 | 75.47 | **+39.11** |

**Key comparisons**: Post-trained 4B (Tau2: 69.17) > public 30B-A3B (Tau2: 28.33); post-trained 30B-A3B (Tau2: 84.17) within 1.2 points of closed-source references (84.76, 85.31). 24 of 25 paired domain comparisons improve.

### Table 2: BFCL Results (Overall, %)

| Model | Overall | Δ |
|---|---|---|
| Qwen3-1.7B | 55.88 | — |
| + Trajectory2Gym | 66.48 | **+10.60** |
| Qwen3-4B | 62.92 | — |
| + Trajectory2Gym | 74.85 | **+11.93** |
| Qwen3-8B | 67.30 | — |
| + Trajectory2Gym | 78.62 | **+11.32** |
| Qwen3-14B | 72.01 | — |
| + Trajectory2Gym | 79.68 | **+7.67** |
| Qwen3-30B-A3B | 77.58 | — |
| + Trajectory2Gym | 80.93 | **+3.35** |

**Gains concentrate in Multi-turn Base** (rises 7–41 points). Every post-trained model from 4B exceeds GPT-5.5's reported Multi-turn Base (60.00). The 30B model's 80.93 Overall numerically exceeds Gemini-3.0-Pro's 79.97.

### Table 3: Ablation Study (Qwen3-8B)

| Setting | Tau1 | Tau2 | BFCL | Mean | Gain |
|---|---|---|---|---|---|
| Existing pool vs. + Targeted expansion | 51.00→58.61 | 75.83→81.67 | 80.39→77.61 | 69.08→72.63 | **+3.55** |
| Loose targets vs. + Decision grounding | 48.13→51.13 | 71.67→74.17 | 75.86→78.62 | 65.22→67.97 | **+2.75** |
| Random vs. + Structural selection | 51.13→57.00 | 74.17→75.83 | 79.40→78.81 | 68.23→70.55 | **+2.32** |
| Row averaging vs. + Parent normalization | 49.52→51.13 | 52.50→74.17 | 77.79→78.62 | 59.94→67.97 | **+8.04** |

### Table 4: Support-Depth Sensitivity (Qwen3-8B)

| Depth | Trajectories | Target tokens | Tau1 | Tau2 | BFCL | Mean |
|---|---|---|---|---|---|---|
| 2 | 13,986 | 18.02M | 55.61 | 82.50 | 78.74 | **72.28** |
| 3 | 14,819 | 19.42M | 57.00 | 75.83 | 78.81 | 70.55 |
| 4 | 15,805 | 20.61M | 54.00 | 75.00 | 79.31 | 69.44 |

**Finding**: Performance is non-monotonic in support depth—the shallowest setting wins despite fewer data. Structural support is an allocation problem, not a scaling rule.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Success gates, does not license, supervision**: The paper formalizes why outcome-filtered trajectory training is insufficient—success certifies episode completion, not what the episode teaches or whether each action was justified by visible evidence.
2. **Three independent levels of data design**: (a) which interaction structures are available, (b) which decisions are eligible for imitation, (c) how much optimization mass each interaction receives. These can vary independently even with similar trajectory counts.
3. **Decoupling execution validity from decision grounding**: Native execution validity ($e_t$) and decision-time grounding ($v_t$) are formally distinct ($g_t = e_t v_t$); both are required for supervision eligibility.

### Practical Implications

1. **Data construction beats data collection**: Targeted synthesis of under-supported paths yields gains that outcome filtering alone cannot provide, across all scales.
2. **Authentic errors are context, not targets**: Rejected calls with authentic errors remain visible context without imitation loss; the grounded recovery that follows becomes a target—preserving recovery reasoning without rewarding failure.
3. **Per-scale allocation matters**: Larger learners benefit from more realizations per path ($d_s$ growing from 1 to 8), but there is an optimal depth—more repetition is not automatically better.
4. **Rendering format should not affect optimization weight**: Parent-normalized training makes a semantic interaction the unit of training weight, independent of how many rendered children it produces.

---

## Conclusion

### Main Takeaways

The central claim is that **what a trajectory teaches is decided not by its outcome, but by the dependencies it realizes and the decisions its visible history can justify**. Trajectory2Gym operationalizes this through:

1. **Compositional interaction synthesis**: Builds the interactions a pool lacks (targeted expansion on deficit structures)
2. **Grounded supervision**: Certifies targets one decision at a time via visibility, execution, and workflow-order checks
3. **Structural-support learning**: Allocates repeated support per learner scale with parent-normalized training weight

### Key Results Summary

- Standard SFT (no RL) on constructed interactions improves all three benchmarks at all five scales (16.9–29.8 points on the three-benchmark mean)
- Gains are strongest in stateful and multi-turn tasks where compositional dependencies matter most
- Parent normalization provides the largest isolated gain (8.04 points)
- The post-trained 4B model exceeds the public 30B-A3B initialization on stateful tasks

### Future Directions

- Extending decision-level certification to **on-policy rollouts** (RL settings)
- Application to **web, software, and terminal agents**
- Tracking structural coverage and decision eligibility alongside data volume in future pipelines
- Keeping training weight of an interaction **independent of its rendering format**

### Broader Lesson

> "Scaling tool-use agents depends not only on collecting more successful trajectories, but on constructing interactions whose dependencies are represented and whose decisions are learnable from the agent's visible history."

---

_Markdown view of https://picx.dev/p/4jy7v7, served by PicX — AI-generated visual whiteboard summaries of research papers._