FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis

Summary (Overview)

• FACET is a framework for synthesizing complex, verifiable terminal tasks from heterogeneous agent skills, addressing two key challenges: information loss during multi-stage generation and cross-artifact inconsistencies (instruction, solution, verifier, environment).
• The pipeline has three stages: (1) skill collection and scenario–skill repository construction, (2) agentic scenario reconstruction with five-dimensional representation, and (3) executable-state-grounded task construction with targeted validation and repair.
• FACET generates 6,078 validated tasks with an average of 22.77 executable tests per task—the highest among compared datasets—and 1.2K training trajectories averaging 11.86 turns.
• Fine-tuning on just 1.2K successful trajectories yields consistent improvements across three model scales: Qwen3.5-4B (+7.12), Qwen3.5-9B (+8.24), and Qwen3.5-27B (+6.75) on Terminal-Bench 2.1.
• Analysis of generation schemes shows that sequential, environment-grounded construction (Forward: I → S → V) achieves the highest task yield (83/100) and significantly improves solution–verifier alignment compared to generating the verifier before the solution.

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses the challenge of training terminal agents—AI systems that interact with command-line environments to complete real-world tasks. While benchmarks like Terminal-Bench provide evaluation platforms, the bottleneck lies in acquiring scalable executable supervision for training.

A terminal task is formally defined as a tightly coupled bundle:

$$\mathcal{T} = (\mathcal{I}, \mathcal{E}, \mathcal{S}, \mathcal{V}, \mathcal{M})$$

where:

• $\mathcal{I}$ = user instruction
• $\mathcal{E}$ = environment specification
• $\mathcal{S}$ = reference solution
• $\mathcal{V}$ = executable verifier
• $\mathcal{M}$ = runtime metadata

Key Theoretical Challenges

The paper identifies two fundamental challenges in multi-stage terminal-task synthesis:

1. Information loss during generation: Rich source materials (skills with capabilities, dependencies, procedural constraints) get progressively compressed into simplified task descriptions, losing critical structure and complexity.

2. Cross-artifact drift: The instruction, solution, verifier, and environment can diverge during generation, leading to tasks that are unsolvable or incorrectly evaluated.

Formal Task Acceptance Criterion

A synthesized task is accepted when:

$$\mathcal{A}(\mathcal{T}) = B(\mathcal{E}) \wedge \neg \nu_{\mathcal{V}}(e_0) \wedge (e_T \neq \bot) \wedge \nu_{\mathcal{V}}(e_T)$$

where:

• $B(\mathcal{E})$ = environment builds successfully
• $\nu_{\mathcal{V}}(e)$ = verifier outcome on state $e$
• $e_0 = \text{Init}(\mathcal{E})$ = initial environment state
• $e_T = \text{Run}(S, e_0)$ = final state after executing solution $S$

This requires: a buildable environment, a non-trivial initial state (verifier fails), an executable reference solution, and a verifier-accepted final state.

Key Design Principles

The paper proposes two core principles:

1. Source-intent preservation: Reconstruct rich scenarios before generating task artifacts, rather than directly converting skills to tasks.
2. Shared executable-state grounding: Use the realized container state as a coordination channel across all artifact generation.

Methodology

Stage 1: Information Source Acquisition

Collection and filtering: Skills are collected from OpenClaw, ClawHub, and GitHub. Unsafe skills, those requiring private access, or depending on non-public resources are removed.

Skill understanding: Each skill is normalized into a structured record containing description, tools, inputs/outputs, procedural steps, and provenance—yielding 71K+ valid skills.

Scenario extraction: An extraction agent identifies possible application contexts, user goals, initial states, and desired final states for each skill.

Scenario–skill repository construction: Similar hypotheses are grouped into candidate pairs $p_c = (c, X_c)$, and a judge evaluates relevance, complementarity, non-redundancy, and executability:

$$\mathcal{P} = \{p_c \mid J(p_c) = 1\}$$

Stage 2: Scenario Reconstruction and Reference Building

The pipeline follows:

$$p_c \xrightarrow{\text{agentic reconstruction}} D_c \xrightarrow{\text{scenario synthesis}} C \xrightarrow{\text{reference building}} (R_S, R_I)$$

Five-dimensional scenario representation:

$$D_c = \{d_{\text{goal}}, d_{\text{context}}, d_{\text{capability}}, d_{\text{state}}, d_{\text{io-tool}}\}$$

• Goal: user objective and expected deliverables
• Context: application setting and motivation
• Capability: role of each skill and inter-skill relationships
• State: initial, intermediate, and desired final states
• Inputs/Outputs & Tools: files, formats, schemas, paths, services, dependencies

Reference generation:

$$R_S = f_S(C), \qquad R_I = f_I(C, R_S)$$

A consistency-alignment model verifies that both references share the same initial state and target outcome.

Stage 3: Executable-State-Grounded Task Construction

Environment construction and repair:

• Environment agent produces a manifest (directories, files, services, dependencies)
• Materializes assets using network access, shell/Python programs, or procedural generation
• Localizes resources for self-containment
• Allows at most 3 environment-repair iterations

Executable-state sharing: After successful build, the realized state $e_0$ is exposed to all downstream generators. Artifacts are generated sequentially:

1. Instruction I from $R_I$ and $e_0$
2. Solution S from I, $R_S$, and $e_0$
3. Execute S to obtain $e_T$
4. Verifier V from I, $R_S$, $e_0$, and $e_T$

Validation and targeted repair: Each candidate is validated for:

• (i) Environment builds and initializes
• (ii) Verifier does not pass in initial state
• (iii) Reference solution executes from clean state
• (iv) Verifier passes on final state

A constrained router identifies the responsible component from execution traces and invokes only the corresponding repair procedure, with at most 5 task-level iterations.

Empirical Validation / Results

Dataset Comparison

Table 1: Trajectory- and task-level comparison of terminal-agent datasets

Key observations:

• FACET has the highest average tests per task (22.77), indicating stricter completion criteria
• FACET's trajectories are among the longest (11.86 turns), reflecting complex multi-step workflows
• Lower pass rates (P@1=27.00) are consistent with denser verification requirements

Main Results

Table 2: Results on Terminal-Bench 2.1

Key findings:

• Consistent improvements across all scales (4B, 9B, 27B)
• Largest absolute gain: 9B model (+8.24)
• Largest relative improvement: 4B model (+40.5%)
• The 27B model (47.57) approaches the 397B model (49.06) despite being ~15× smaller

Task Difficulty Analysis

• Partial progress is common: 89.40% of individual checks are satisfied across teacher rollouts, but only 20.94% achieve full task success
• Failures cluster near the success boundary: 54.00% of unsuccessful rollouts fail only 1–2 verifier checks
• Difficulty emerges from composition: Structured-data tasks are solved more reliably than narrative-document tasks; longer instructions and broader verifier coverage decrease performance

Generation Scheme Comparison

Table: Initial validation outcomes by generation scheme

After repair: Forward reaches 83/100, Reverse 63/100, Joint 65/100.

Paired comparison over 88 shared scenario–skill pairs:

• Forward vs. Reverse: 29 vs. 9 successes (p = 0.0017, two-sided exact sign test)
• Forward vs. Joint: 27 vs. 18 (p = 0.233)

This provides strong evidence that generating the solution before the verifier improves cross-artifact alignment.

Theoretical and Practical Implications

Theoretical Contributions

1. Source-intent preservation principle: Demonstrates that reconstructing rich scenarios before task generation preserves information that would otherwise be lost in direct skill-to-task conversion.

2. Executable-state grounding principle: Shows that using the realized container state as a shared coordination channel prevents cross-artifact inconsistencies.

3. Sequential artifact generation: Provides empirical evidence that the order of artifact generation matters—specifically, generating the verifier after the solution (rather than before) significantly improves task validity.

Practical Implications

1. Data efficiency: 1.2K successful trajectories suffice for meaningful improvements, suggesting that quality-focused synthesis can complement quantity-focused approaches.

2. Scalability across model sizes: The consistent gains across 4B–27B models indicate the supervision transfers across capacity regimes.

3. Dense verification as training signal: The high test-per-task ratio (22.77) provides fine-grained execution feedback that distinguishes near-successful trajectories from failures, enabling more informative supervision.

4. Repair-based pipeline: The targeted repair mechanism (identifying and fixing only the responsible artifact) improves yield while preserving valid components.

Conclusion

Main Takeaways

FACET demonstrates that carefully coordinating scenario reconstruction, executable-state grounding, and artifact-level validation can provide data-efficient supervision for terminal agents without relying solely on large-scale task generation. The framework:

• Preserves source intent through five-dimensional scenario representation
• Ensures cross-artifact consistency through shared executable-state grounding
• Achieves high task yield through targeted validation and repair
• Produces effective training data that improves performance across multiple model scales

Future Directions

The paper outlines three planned extensions:

1. Broader knowledge sources: Extending FACET to more diverse sources of procedural knowledge
2. More interactive environments: Adapting the framework to broader interactive settings beyond terminals
3. Reinforcement learning: Investigating how generated tasks and execution feedback can support RL-based agent improvement

AI Use Statement

The authors disclose that generative AI tools were used for drafting, language editing, and LaTeX preparation, with the authors taking full responsibility for verifying all source attributions, experimental records, numerical claims, and generated text.