Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training

Summary (Overview)

• Core contribution: Data-DPO is a target-model-aware data selection method for supervised fine-tuning (SFT) that treats data value as a dynamic property dependent on the compatibility between samples and the target model's capability distribution, rather than a static quality attribute.

• Key innovation: The method constructs pairwise data preferences from one-step training feedback of the target model (activation probing), trains a lightweight reward model using a DPO-style objective in data space, and combines target-model preference with external quality scores and marginal diversity for final subset selection.

• Empirical results: Data-DPO consistently outperforms existing baselines across Vision-Flan and LLaVA-CoT datasets under 5%, 10%, and 15% data budgets, achieving ARP (Average Relative Performance) of 100.76–102.70 on Vision-Flan and 101.31–103.93 on LLaVA-CoT, stably surpassing full-data training performance.

• Robustness: The method demonstrates stability across different target models (LLaVA-v1.5-7B/13B, Qwen2VL-2B-Instruct, Llama-3.2-11B-Vision-Instruct), quality scoring sources, and embedding sources, with ablation studies confirming the complementarity of the three selection signals.

Introduction and Theoretical Foundation

Background and Motivation

Supervised fine-tuning (SFT) is critical for improving large model capabilities, but as instruction data scales grow, fine-tuning on full datasets becomes computationally expensive and may not yield optimal performance. The paper addresses the problem of selecting a small yet effective training subset from large-scale candidate data.

Key Theoretical Insight

The authors challenge the prevailing assumption that data value is a static property determined by sample quality and representation space structure. Their central argument is:

"SFT typically does not require a model to learn knowledge from scratch; instead, it further activates, calibrates, and aligns the model's existing capability distribution."

Since models differ in architecture, parameter scale, and capability distribution, the same sample may play substantially different roles for different target models. The paper reframes the data selection question from "Which data is high-quality?" to "Which data is more suitable for the target model?"

Problem Formulation

Given an original training set $\mathcal{D}$, data selection aims to select a subset $\mathcal{D}' \subseteq \mathcal{D}$ under a fixed budget $K$:

$$\max_{\mathcal{D}' \subseteq \mathcal{D}} f(\mathcal{D}') \quad \text{s.t.} \quad |\mathcal{D}'| = K.$$

where $f(\cdot)$ denotes the target task performance of the model trained on a given dataset.

Related Work

The paper positions Data-DPO against two complementary perspectives in existing data selection methods:

• Sample importance: LESS, TIVE, ICONS, OPUS, ScalSelect, EL2N (gradient similarity, attention behavior, proxy model losses)
• Data diversity: PRISM, FisherSFT, SemDeDup, Self-Sup, D2 Pruning (embedding-based deduplication and coverage)
• Hybrid approaches: CoIDO, DataTailor, COINCIDE (combining quality and diversity)

Methodology

Dual View Encoding

Each sample $z_i = (c_i, y_i)$ where $c_i$ is the input condition and $y_i$ is the target response, is encoded with two complementary representations:

Condition embedding (for probe set construction and diversity):

$$e_i^c = E_{\mathrm{emb}}(c_i)$$

Supervision embedding (for preference learning and scoring):

$$e_i^s = E_{\mathrm{emb}}(c_i, y_i)$$

Both are extracted with a frozen embedding model, followed by centering and $L_2$-normalization.

Probe Set Construction

The candidate set $\mathcal{D}$ is clustered into $m$ clusters via spherical clustering based on condition embeddings:

$$\mathcal{D} = \bigcup_{k=1}^{m} \mathcal{C}_k$$

For each cluster $\mathcal{C}_k$, a greedy k-center strategy selects $n_k = \lceil \rho |\mathcal{C}_k| \rceil$ diverse samples:

$$z^* = \arg\max_{z_i \in \mathcal{C}_k \setminus S_k} \min_{z_j \in S_k} \left(1 - \cos(e_i^c, e_j^c)\right)$$

Activation Probing

For $T$ rounds of one-step probing on the probe set $\mathcal{D}_p$, with batch $B_t = \{z_1, \ldots, z_b\}$:

1. Compute SFT loss before update: $\ell_i^0 = \ell(M_0, z_i)$
2. Perform one temporary SFT update: $M_t' = \mathrm{OneStepUpdate}(M_0, B_t)$
3. Compute loss after update: $\ell_i^1 = \ell(M_t', z_i)$

The one-step activation gain is defined as:

$$g_i^{(t)} = \log \frac{\ell_i^0 + \epsilon}{\ell_i^1 + \epsilon}$$

with batch-level normalization:

$$s_i^{(t)} = \frac{g_i^{(t)} - \mu_t}{\sigma_t + \epsilon}$$

Preference Construction

For any two samples in a batch, the relative activation margin is:

$$d_{ij}^{(t)} = s_i^{(t)} - s_j^{(t)}$$

Positive $d_{ij}^{(t)}$ indicates $z_i \succ z_j$. Aggregated signed margins across multiple comparisons:

$$\bar{d}_{ij} = \frac{1}{n_{ij}} \sum_{t=1}^{n_{ij}} d_{ij}^{(t)}$$

producing the preference set $\mathcal{P} = \{(z_w, z_l)\}$.

Preference Learning (Data-DPO Objective)

A lightweight residual MLP reward model $f_\theta$ maps supervision embeddings to scalar rewards:

$$r_\theta(z_i) = f_\theta(e_i^s)$$

The policy over the probe set uses softmax parameterization:

$$\pi_\theta(z_i) = \frac{\exp(r_\theta(z_i))}{\sum_{z_j \in \mathcal{D}_p} \exp(r_\theta(z_j))}$$

The DPO-style objective uses a uniform empirical reference (since data selection has no natural model-induced reference policy):

$$\mathcal{L}_{\text{Data-DPO}} = -\mathbb{E}_{(z_w, z_l) \sim \mathcal{P}} \log \sigma\left(r_\theta(z_w) - r_\theta(z_l)\right)$$

This simplification arises because the uniform reference gives $\log \frac{\pi_{\mathrm{ref}}(z_w)}{\pi_{\mathrm{ref}}(z_l)} = 0$.

Sequential Selection

The final selection combines three signals with step reward:

$$R(S_t, z_i) = p_i + q_i + d_i(S_t)$$

where:

• $p_i = \sigma(\hat{r}_i)$ is the normalized target-model preference score
• $q_i \in [0,1]$ is the external quality score (1–5 rating normalized to [0,1])
• $d_i(S_t) = 1 - \max_{z_j \in S_t} \max(0, \cos(e_i^c, e_j^c))$ is the marginal diversity gain

At each step: $z_t = \arg\max_{z_i \in \mathcal{A}_t} R(S_t, z_i)$

Implementation Details

• Clustering: 2000 clusters for Vision-Flan, 1000 for LLaVA-CoT; probe ratio $\rho = 0.05$
• Probing: $T = 16$ rounds, batch size $b = 1024$
• Reward model: Residual MLP with input dim 2048, hidden dim 1024, 4 residual blocks with LayerNorm, GELU, Dropout(0.1); trained with AdamW, LR $1\times10^{-3}$, weight decay $1\times10^{-4}$, batch size 4096, 5 epochs

Empirical Validation / Results

Experimental Setup

• Datasets: Vision-Flan (general instruction tuning, ~186K samples) and LLaVA-CoT-100k (reasoning-oriented, ~99K samples)
• Target models: LLaVA-V1.5-7B (Vision-Flan), Llama-3.2-11B-Vision-Instruct (LLaVA-CoT)
• Data budgets: 5%, 10%, 15%
• Evaluation metric: Average Relative Performance (ARP) = $\frac{\text{Subset Data Performance}}{\text{Full Data Performance}} \times 100$
• Baselines: Random, XMAS, COINCIDE, SemDeDup, D2 Pruning, PRISM, ScalSelect, CLIP Score, EL2N

Main Results

On Vision-Flan (LLaVA-v1.5-7B target):

On LLaVA-CoT (Llama-3.2-11B-Vision-Instruct target):

Data-DPO outperforms full-data training in all settings except the 15% LLaVA-CoT budget where EL2N achieves slightly higher ARP.

Key Ablation Findings

Proxy model analysis: Using a proxy model (5% checkpoint) instead of the original target model for probing degrades ARP from 100.76→93.17 (5% budget), confirming that proxy models may provide misaligned selection signals.

Selection reward ablation (Vision-Flan, 10% budget):

Robustness Results

• Different target models: LLaVA-v1.5-13B (Vision-Flan) achieves ARP 93.76–94.39 (vs random 85.12–92.26); Qwen2VL-2B-Instruct (LLaVA-CoT) achieves ARP 97.50–100.13 (vs random 94.30–95.25)
• Different quality scorers: LLaVA-OneVision-1.5-4B-Instruct yields ARP 100.53–103.76, still outperforming random
• Different embedding sources: Qwen3-VL-Embedding-8B yields ARP 99.95–100.75, maintaining stable gains
• Simpler reward model: Plain MLP achieves ARP 95.36–101.75, confirming gains come from preference supervision rather than model capacity

Time Cost

Data-DPO requires ~19.0 GPU hours on LLaVA-CoT (NVIDIA A6000), slightly higher than baselines (6.70–15.99 hours) but acceptable compared to full SFT training.

Theoretical and Practical Implications

Theoretical Implications

1. Data value is target-model-conditional: The paper provides empirical evidence that data selection signals derived from proxy models can be misaligned with the original target model's needs, supporting the view that data value in SFT is intrinsically tied to the model's capability distribution.

2. DPO extends beyond model alignment: By applying DPO-style preference optimization in data space (rather than response space), the paper demonstrates that preference learning is a general framework applicable to data curation, not just policy alignment.

3. Signal complementarity: The ablation study reveals that preference signals alone can be harmful (92.33 ARP, below random), but become powerful when combined with quality and diversity constraints, suggesting that target-model feedback captures a distinct dimension of data value orthogonal to quality and coverage.

Practical Implications

1. Computational efficiency: Data-DPO achieves better-than-full-data performance with only 5–15% of training data, offering substantial cost savings in SFT.

2. Model-agnostic applicability: The method works across model families (LLaVA, Llama, Qwen) and scales (2B–13B), making it a practical tool for model-specific data curation.

3. Robustness to auxiliary signals: Practitioners can use different quality scorers or embedding models without significant performance degradation, lowering the barrier to adoption.

Conclusion

Main Takeaways

Data-DPO introduces a paradigm shift in SFT data selection by treating data value as a function of the target model's current capability distribution rather than a static property. The method's three-stage pipeline—activation probing on the target model, DPO-style preference learning, and joint preference-quality-diversity selection—consistently outperforms existing baselines and full-data training across multiple datasets and budgets.

Limitations

1. Bias sensitivity: Performance may degrade when external quality signals, embedding sources, or target model preferences are highly biased
2. Distribution shift limitations: The method assumes distribution-preserving settings; effectiveness may be limited under substantial distribution shifts in the full dataset
3. Computational overhead: Requires additional computation on the probe set (~19 GPU hours), leaving room for efficiency improvements

Future Directions

The paper suggests potential avenues including improving probing efficiency, addressing distribution shift scenarios, and further reducing the computational cost of preference construction while maintaining the benefits of target-model-aware data selection.