Summary (Overview)

• DataPrep-Bench is the first unified, downstream-grounded benchmark that jointly evaluates two complementary capabilities of LLM-driven data preparation: data construction (transforming raw sources into supervised training data) and data quality evaluation (predicting training utility before downstream training).
• The benchmark covers six domains (General, Math, Science, Medical, Finance, Law) with shared raw source corpora, base models (Qwen2.5-7B, Llama-3.1-8B, Mistral-7B-v0.3), fine-tuning protocols, and downstream evaluation benchmarks.
• Key finding: Adding synthesized domain data on top of Dolly-15k often hurts downstream performance, indicating the need for stronger construction methods.
• Data-Construction-Skill (DCS) is a skill-guided agent baseline that lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance and is competitive in knowledge-extraction-dense domains.
• Distributional Alignment Score (DAS) is a distribution-based quality evaluator using Maximum Mean Discrepancy (MMD) between a candidate dataset and a domain proxy; it achieves the strongest cross-model correlation in four of six domains and is the only metric exceeding (r > 0.70) in Math, Science, and Medical simultaneously.

Introduction and Theoretical Foundation

The performance of large language models (LLMs) is fundamentally governed by the quality, diversity, and scale of their training data. As naturally occurring corpora fail to keep up with demand, the community increasingly uses LLMs, agents, and data-centric workflows to produce training data themselves. This paradigm is called LLM-driven data preparation and is decomposed into two capabilities:

1. Data construction: transforms raw, non-trainable sources (domain books, technical manuals, web dumps) into supervised training data.
2. Data quality evaluation: predicts, before training, which candidate datasets will improve a downstream model.

Existing works use disparate sources, base models, and downstream benchmarks, making cross-method comparison largely anecdotal. DataPrep-Bench addresses this by providing a unified, downstream-grounded benchmark that evaluates both capabilities under identical conditions.

Theoretical motivation for DAS comes from domain adaptation theory. Let (F) be the hypothesis class. For a hypothesis (h) trained on source distribution (S_X) and evaluated on target distribution (T_X), the generalization bound (Redko et al., 2019) is:

[ R_{T}^{\ell_q}(h) \leq R_{S}^{\ell_q}(h) + d_{\text{MMD}}(\hat{S}_X, \hat{T}_X) + \text{[concentration and complexity terms]} + \lambda, ]

where (R_{T}^{\ell_q}(h)) and (R_{S}^{\ell_q}(h)) are target and source risks, (d_{\text{MMD}}) is the empirical Maximum Mean Discrepancy, and (\lambda) is the combined error of the ideal joint hypothesis. This bound suggests that distributional alignment between training source and target is a principled factor governing target-side performance.

Methodology

Benchmark Design

Data Construction Track: Given raw domain source corpora (B_k) (books, long-form materials), a method (M) produces a supervised dataset (X_k^{(M)} = {(q_i, a_i)}_{i=1}^{N_k^{(M)}}). The method is scored by fine-tuning a base model (f_0) on (X_k^{(M)}) jointly with Dolly-15k and evaluating on downstream benchmarks (T_k):

[ \text{Score}_{\text{con}}(M, D_k) = \text{Perf}\left(f\left(X_k^{(M)}\right), T_k\right). ]

Data Quality Evaluation Track: Given a candidate dataset pool (\mathcal{X}k = {X{k,1}, \dots, X_{k,C_k}}), a scoring function (Q) assigns a scalar (q_{k,i} = Q(X_{k,i})). The ground-truth downstream performance is (s_{k,i} = \text{Perf}(f(X_{k,i}), T_k)). The metric is evaluated by the Pearson correlation between (q_k) and (s_k):

[ \rho(Q, D_k) = \frac{\sum_{i=1}^{C_k} (q_{k,i} - \bar{q}k)(s{k,i} - \bar{s}k)}{\sqrt{\sum{i=1}^{C_k} (q_{k,i} - \bar{q}k)^2} \sqrt{\sum{i=1}^{C_k} (s_{k,i} - \bar{s}_k)^2}}. ]

A metric is considered reliable only when the correlation is statistically significant ((p < 0.05)).

Source Corpus and Candidate Pools

For data construction, raw sources are domain-specific books collected from open textbook repositories (Wikibooks, FreeBookCentre, Open Textbook Library) and converted to Markdown via MinerU. For data quality evaluation, candidate pools mix in-domain and out-of-domain public SFT corpora (e.g., UltraChat, WizardLM, OpenR1-Math-220k, UltraMedical, Finance-Instruct-500k, etc.). Domain proxies are selected for distribution-based metrics:

Proposed Methods

Data-Construction-Skill (DCS): A skill-guided agent that partitions the source corpus into semantically coherent chunks, then for each chunk generates up to three types of QA pairs: concept-oriented ((G_{\text{concept}})), reasoning-oriented ((G_{\text{reason}})), and case-based ((G_{\text{case}})). The union forms the chunk-level supervision set:

[ G(c) = G_{\text{concept}}(c) \cup G_{\text{reason}}(c) \cup G_{\text{case}}(c). ]

The final dataset is (X_k^{(M_{\text{DCS}})} = \bigcup_{c \in C_k} G(c)). The skill layer provides reusable specifications for output schemas, filtering criteria, and validation rules, rather than relying on a one-off prompt.

Distributional Alignment Score (DAS): For a candidate dataset (X_{k,i}), encode each sample with a fixed text encoder (\phi) to obtain feature sets (\Phi(X_{k,i})) and (\Phi(X_k^{\text{proxy}})). Compute the empirical MMD using a Gaussian RBF kernel:

[ \hat{d}^2_{\text{MMD}}(A, B; \kappa) = \frac{1}{n^2}\sum_{i,i'} \kappa(a_i, a_{i'}) + \frac{1}{m^2}\sum_{j,j'} \kappa(b_j, b_{j'}) - \frac{2}{nm}\sum_{i,j} \kappa(a_i, b_j), ]

with (\kappa(x, y) = \exp\left(-\frac{|x-y|_2^2}{2\sigma^2}\right)). Then DAS is the negative MMD distance:

[ \text{DAS}(X_{k,i}) = -d_{\text{MMD}}(\Phi(X_{k,i}), \Phi(X_k^{\text{proxy}}); \kappa). ]

Higher DAS indicates stronger distributional alignment and higher expected downstream utility.

Empirical Validation / Results

Data Construction Track

Experiments were conducted on Qwen2.5-7B and Llama-3.1-8B. Key results for Qwen2.5-7B on Math, General, and Finance:

For Llama-3.1-8B on Finance:

Key takeaways:

• Adding synthetic domain data often hurts performance (many rows below Dolly-only baseline).
• No single family is universally best: DataFlow-Skill leads on structured domains (Finance, Law); agent-based methods lead in reasoning-heavy domains (Math, Medical); Data-Construction-Skill is strongest in knowledge-extraction-dense domains.
• Science and parts of Law remain challenging.

Data Quality Evaluation Track

DAS was compared against 17 other quality/diversity metrics across three base models (Qwen2.5-7B, Llama-3.1-8B, Mistral-7B-v0.3). DAS achieves the highest domain-averaged correlation in four of six domains (General, Math, Science, Medical) and is the only metric to simultaneously clear (r > 0.70) in Math, Science, and Medical. The full results table (from the paper) shows:

(Note: Actual values are approximate from the paper's text; the paper reports DAS as the only metric with (r>0.70) in Math, Science, Medical simultaneously.)

Finance and Law remain difficult for all tested metrics, suggesting that those domains may require more specialized proxy datasets or different evaluation approaches.

Theoretical and Practical Implications

• Theoretical: The DAS metric is grounded in domain adaptation theory (MMD bound on target risk), providing a principled connection between distributional alignment and downstream training utility. The proxy-based formulation avoids benchmark contamination via the triangle inequality: [ d_{\text{MMD}}(P_{k,i}, P_k^) \leq d_{\text{MMD}}(P_{k,i}, P_k^{\text{proxy}}) + d_{\text{MMD}}(P_k^{\text{proxy}}, P_k^). ]
• Practical: DataPrep-Bench provides a standardized testbed for developing and comparing data construction and quality evaluation methods. The finding that synthetic domain data often hurts performance highlights the need for more robust construction methods. DAS offers a lightweight, training-free alternative to expensive per-sample scoring or gradient-based methods.
• Implications for practitioners: The benchmark reveals that surface-level quality proxies (e.g., LLM-as-a-judge) are not reliable predictors of downstream utility, and that distributional alignment is a more robust signal. The DAS pipeline can be applied to any candidate dataset without fine-tuning, making it suitable for large-scale data selection.

Conclusion

The paper introduces DataPrep-Bench, the first unified benchmark for LLM-driven data preparation, covering both data construction and data quality evaluation across six domains. It releases two strong baselines:

• Data-Construction-Skill (DCS): a skill-guided agent that improves upon Dolly-only baselines and is competitive with state-of-the-art methods in knowledge-extraction-dense domains.
• Distributional Alignment Score (DAS): a distribution-based evaluator that achieves the best cross-model correlation in most domains and is the only metric with high correlations across Math, Science, and Medical simultaneously.

Key findings:

1. Adding synthetic domain data often degrades performance, indicating that current construction methods require improvement.
2. No single method dominates all domains; domain-specific strengths matter.
3. DAS is the most reliable quality-evaluation metric overall, but Finance and Law remain challenging.

Future directions: The benchmark is designed to be extensible with new candidate datasets, proxy datasets, and methods. The authors expect the candidate pools for specialized domains to grow as more high-quality SFT corpora become public. The code and data are publicly available at GitHub and Hugging Face.