Summary (Overview)

• Problem Definition: The paper identifies the NL2Pipeline gap – the disconnect between users expressing data-processing workflows in natural language and the need for persistent, editable, platform-native pipeline artifacts (as opposed to disposable scripts generated by coding agents).
• Proposed System: DataFlow-Harness is introduced as a platform that guides an LLM agent (Claude Code) to construct platform-native directed acyclic graphs (DAGs) through typed, incremental mutations, rather than free-form code generation.
• Key Components: The system combines three decoupled components: DataFlow-Skills (procedural guidance), a Model Context Protocol (MCP) layer (live operator registry and pipeline state), and DataFlow-WebUI (synchronized conversational and visual DAG editor).
• Main Results: On a 12-task data-engineering benchmark, DataFlow-Harness achieves a 93.3% end-to-end pass rate. It reduces monetary cost by 72.5% and generation latency by 49.9% relative to Vanilla Claude Code, while achieving performance within 0.9 percentage points of the Context-Aware Claude Code baseline at 42.8% lower cost.
• Downstream Utility: Controlled case studies in math reasoning and general SFT pipeline construction show that pipelines authored with DataFlow-Harness produce training data that yields higher downstream model accuracy, especially on hard benchmarks (e.g., AIME24/25 gains of +10.8 and +12.9 percentage points at one epoch).

Introduction and Theoretical Foundation

Background and Motivation

Large language models (LLMs) are increasingly used to automate data-processing workflows for synthetic data generation, evaluation, retrieval augmentation, and model training. Recent coding agents can automatically translate natural-language requirements into executable implementations, significantly reducing effort. However, high task accuracy alone is often insufficient for production deployment:

"In industrial environments, workflow artifacts must remain visible, editable, reusable, and compatible with platform governance mechanisms throughout their lifecycle."

Direct code-generation agents frequently produce disposable scripts that exist only as source code – difficult to audit through graphical workflow interfaces and prone to hallucinating dependencies (unavailable operators, outdated platform assumptions, framework-specific behaviors).

Theoretical Foundation: The NL2Pipeline Gap

The paper formally defines the NL2Pipeline gap as:

"While users express workflow requirements in natural language, production environments require structured and persistent pipeline assets that can be visualized, edited, and reused."

Here:

• Workflow = the intended data-processing procedure
• Pipeline representation = its persistent platform object
• DAG = captures its execution dependencies

Closing this gap requires more than improving code-generation accuracy: construction must remain grounded in platform semantics and produce artifacts that integrate with the host platform.

Related Work

The paper situates itself in three research lines:

1. Agents for Code Generation: From Codex/StarCoder to Reflexion, Self-Debug, SWE-agent, and Claude Code. DataFlow-Harness differs by constraining a general-purpose agent within a domain-specific harness rather than improving the agent itself.
2. Data Engineering and LLM Pipelines: Systems like Data-Juicer, DCLM, DataFlow, and DSPy treat LLM operations as composable components. The paper builds on DataFlow but focuses on agent-assisted construction of the pipeline itself.
3. LLM-based Workflow Synthesis: Systems like AutoFlow and Balis et al. generate structured workflows. DataFlow-Harness focuses on interactive, stateful authoring inside a live data-engineering platform – the agent retrieves current pipeline state via MCP, applies typed mutations, validates each proposed state, and shares one persistent representation across interfaces.

Methodology

System Architecture

DataFlow-Harness organizes workflow synthesis around four components:

1. Data Pipeline Backend – The authoritative source of truth. A pipeline is represented as:

$$P = (D, O, E, S, R)$$

where:

• $D$ = set of data sources and their URIs
• $O$ = set of configured operator instances
• $E \subseteq O \times O$ = directed data-dependency edges
• $S$ = input and output field schemas
• $R$ = runtime state (e.g., model-serving endpoints)

Agents interact through typed mutations (add/remove operators, update parameters, connect edges). A mutation is committed only if the resulting graph is acyclic and adjacent operator schemas are compatible.

2. DataFlow-WebUI – Provides two synchronized modalities:

• Conversational Interface: Natural-language authoring; agent context is injected with current pipeline state and operator registry via MCP.
• Visual DAG Editor: Graphical editor for inspection and direct editing; manual edits are immediately committed to the backend.

3. MCP Tools Layer – Implements a Request-Validate-Commit protocol:

• State Retrieval: Agent fetches latest pipeline state at each turn.
• Mediated Mutation: Agent issues MCP tool calls expressing typed, structured mutations.
• Validation: System verifies DAG acyclicity and schema compatibility between adjacent operators.
• Validated Commitment: Changes written to backend; WebSocket broadcasts update to clients.

4. DataFlow-Skills – Encodes two classes of domain-specific knowledge:

• Procedural Blueprints: Recommended construction sequences (schema inference → operator selection → parameter configuration → serving verification).
• Compositional Constraints: Operator compatibility rules (modality matching, field-flow conventions for nested structures).

These guide agent reasoning while the MCP layer grounds execution against the live DataFlow environment.

Experimental Setup

• Benchmark: 12 tasks spanning six industrial scenarios: QA generation, review governance, long-document processing, multi-field scoring, schema normalization, and low-quality filtering.
• Agent Configurations: Four levels of platform grounding:
  1. Vanilla CC: Unconstrained Claude Code, generates disposable Python scripts.
  2. Context-Aware CC: Claude Code with raw DataFlow codebase in context, still produces scripts.
  3. MCP-only: Tool-augmented, synthesizes native DAGs but lacks procedural guidance.
  4. DataFlow-Harness: Full system with MCP + DataFlow-Skills.
• Model: Claude Opus 4.7 used for all experiments.
• Repetitions: Each task executed 10 times per setting (120 runs per method).
• Metrics: End-to-End (E2E) Pass (execution success + acceptance criteria), token consumption, monetary cost, generation latency.

Empirical Validation / Results

RQ1: Workflow Synthesis Effectiveness

Table 1: Main results on 12-task benchmark.

Key findings:

• Vanilla CC: 91.7% → Context-Aware CC: 94.2% (procedural context helps).
• MCP-only drops to 83.3%, revealing the NL2Pipeline gap.
• DataFlow-Harness recovers to 93.3% (within 0.9 pp of Context-Aware CC).

RQ2: Efficiency and System Cost

From Table 1:

• DataFlow-Harness vs. Vanilla CC: Cost reduced by 72.5% ($0.950 →$0.261); latency reduced by 49.9% (190.7s → 95.5s).
• DataFlow-Harness vs. Context-Aware CC: Cost reduced by 42.8%; latency reduced by 17.6%.
• Token consumption: DataFlow-Harness uses 74,958 input tokens vs. 185,626 for Context-Aware CC (59.6% reduction).

The improvement is driven by lower token consumption – workflow representations are far more compact than executable code.

Textbook-to-VQA Workflow Evaluation

Table 2: Extraction performance on challenging textbook-to-VQA task.

DataFlow-Harness achieves 97.2% precision and 87.3% coverage. The largest absolute improvement is in coverage, suggesting the framework constructs more complete workflows rather than merely filtering outputs. The advantage arises from enabling systematic reuse of mature platform assets (PDF parsing, layout recovery, OCR, figure extraction, multimodal understanding).

RQ3: Ablation and Micro-mechanisms

Table 3: Per-task end-to-end pass counts (out of 10 trials).

Three patterns emerge:

1. Procedural guidance most valuable on QA-generation and language-processing tasks (1a, 1b, 3b): aggregate success improves from 18/30 to 29/30.
2. Limited benefit for simple transformation/filtering tasks (5a, 5b, 6a, 6b): both achieve perfect success.
3. Cannot overcome non-synthesis bottlenecks: on multi-field scoring (4a, 4b), both methods show same failure rate due to downstream numerical constraints.

RQ4: Downstream Training Utility

Math Reasoning Pipeline (Table 4):

At matched epochs, DataFlow-Harness data yields higher average accuracy (51.6 vs. 49.9 at 1 epoch; 55.7 vs. 54.5 at 2 epochs). Gains concentrate on hardest benchmarks (AIME24/25), suggesting better verification, filtering, and deduplication.

General SFT Pipeline (Table 5):

Nearly identical knowledge performance (MMLU 74.2 vs. 74.4). Clear difference on code benchmarks: DataFlow-Harness pipeline is stronger across all four, with largest gap on MBPP (75.4 vs. 64.6). Overall 9-benchmark average improves by 2.3 points (63.8 vs. 61.5).

Theoretical and Practical Implications

Theoretical Implications

1. The NL2Pipeline gap is real and measurable: The 10-point drop from Context-Aware CC (94.2%) to MCP-only (83.3%) demonstrates that generating platform-native DAGs is substantially harder than generating executable scripts. This gap is not merely a code-generation problem but a reasoning-under-constraints problem.

2. Procedural guidance is more valuable than operator exposure alone: The ablation study shows that DataFlow-Skills provide the largest benefit on procedurally complex tasks (QA generation), where implicit domain knowledge cannot be recovered from operator specifications alone. This suggests that workflow synthesis requires both declarative (operator registry) and procedural (skills) knowledge.

3. Workflow representations are more efficient than code: The dramatic reduction in token consumption (59.6% fewer input tokens vs. Context-Aware CC) suggests that structured DAG representations are inherently more compact and easier for LLMs to manipulate than free-form code.

Practical Implications

1. Cost and latency reduction: DataFlow-Harness achieves 72.5% cost reduction and 49.9% latency reduction compared to Vanilla CC, making it practical for production deployment where budget and time constraints are critical.

2. Governance and editability: The platform produces persistent, editable DAG artifacts that can be inspected, modified, and reused – addressing a key industrial requirement that script-generation approaches fail to meet.

3. Downstream data quality: The controlled case studies provide preliminary evidence that grounded pipeline construction leads to higher-quality training data, especially for hard benchmarks (AIME) and code tasks. This suggests that platform grounding helps agents build more effective pipelines, not just more governable ones.

4. Reuse of platform assets: The textbook-to-VQA case study demonstrates that Data