Summary (Overview)

• Core thesis: Individual-level signals (estimating each data sample's utility in isolation) are insufficient for curating synthetic data in LLM training; group-level signals (accounting for interactions among samples) are necessary for effective data curation.
• Key finding 1: Synthetic datasets that are indistinguishable under individual-level influence measures can differ significantly under group-level influence, and curating by group-level influence yields better downstream performance, particularly in generative capability.
• Key finding 2: As training pipelines become increasingly synthetic, only data curation methods that incorporate group-level signals improve over baseline; gains increase when relational weights capturing inter-sample relationships are amplified.
• Key finding 3: A cheap diagnostic based on within-group gradient diversity can identify which groups most need group-level estimation, recovering much of the benefit of full group-level scoring at a fraction of the compute cost.
• Practical contribution: The paper offers a theoretical bound (Proposition 5.1) connecting the oracle gap to gradient diversity, plus empirical validation across pre-training and post-training (RLVR) settings.

---

Introduction and Theoretical Foundation

Background and Motivation

Synthetic data is now essential to LLM training across all stages—from pre-training corpora to post-training reasoning induction. However, training on synthetic data at scale can degrade model generative capabilities (a phenomenon known as "model collapse") because synthetic data exhibits narrower coverage than organic web data and inherits generator biases. These consequences are costly for LLM developers who waste substantial compute recovering from collapsed training runs.

The Central Problem

Current data curation practices rely on individual-level signals—estimates of each data sample's training utility in isolation. The authors argue these are insufficient for synthetic data, which exhibits two key challenges:

Challenge 1: Synthetic data is homogeneous. Large synthetic corpora are often generated from few seed prompts, making them more homogeneous than organic corpora. This homogeneity makes it harder to distinguish which synthetic samples most improve model performance. The paper shows synthetic data has lower variance than organic data not only in token-level and information density metrics, but also in individual-level data influence.

Challenge 2: A dataset's value is non-additive. Interactions between samples can induce cancellation or amplification effects, making a dataset's value non-additive in its samples. These properties are invisible to an individual scorer.

Theoretical Foundation: Oracle Scoring

The paper formalizes data influence through oracle scoring, which compares a model trained with and without a sample (or group):

Definition 2.1 (Oracle Score: Group and Individual). The oracle score of a set of training examples $G = \{z_j\}_{j=1}^{|G|}$ measures the change in model performance when $G$ is included versus excluded from the training data, evaluated with respect to the model loss on a reference set $D_{ref}$:

$$I_{orcl}(G, \theta) = L(D_{ref} | \theta_{D_{train} \cup G}) - L(D_{ref} | \theta_{D_{train}})$$

For a single example $z$ ($|G| = 1$), we write $I_{orcl}(z, \theta) := I_{orcl}(\{z\}, \theta)$ and call it the individual oracle score.

Definition 2.2 (Individual Proxy Oracle). The group oracle is approximated by the mean of individual oracle scores:

$$I_{orcl}(G, \theta) \approx I_{proxy}(G, \theta) = \frac{1}{|G|} \sum_{z \in G} I_{orcl}(z, \theta)$$

The paper demonstrates that this proxy fails to capture group-level interactions, motivating the need for true group-level signals.

---

Methodology

Experimental Settings

The paper studies the gap between individual and group-level influence in two settings:

Phase 1: Candidate Pool Construction

To isolate interactions between data samples that the individual proxy fails to capture, the authors construct groups with:

• (i) Similar individual proxy scores (held fixed as a control)
• (ii) Different compositions (varying diversity and topics)

Groups are built by partitioning the corpus into 20 semantic clusters (using BGE embeddings with k-means) and drawing groups from between 1 (homogeneous) and 20 (diverse) clusters.

Phase 2: Data Selection

Each scoring method selects $k$ groups via Gumbel-Top-$k$ sampling:

$$D_{proxy} \leftarrow \text{Gumbel-Top-}k\{I_{proxy}(G, \theta)/\lambda | G \in \mathcal{G}\}$$ $$D_{orcl} \leftarrow \text{Gumbel-Top-}k\{I_{orcl}(G, \theta)/\lambda | G \in \mathcal{G}\}$$

where $\lambda \in (0, \infty)$ is the temperature.

Phase 3: Training and Evaluation

Three selection methods are compared under a matched budget: group oracle, individual proxy oracle, and random baseline. Training uses identical hyperparameters and step counts so differences come from selection alone.

Data Curation Methods (Section 4)

For the practical curation experiments, the paper compares:

The synthetic pools are constructed with varying rephrase factors $r \in \{1, 10, 50\}$, controlling the number of rephrases per organic source document.

---

Empirical Validation / Results

Finding 1: Group Oracle Recovers Signal the Individual Proxy Misses

Figure 3 shows that although all groups in the candidate pool have near-identical individual proxy scores, their group oracle scores are mismatched and span a much wider range. Table 1 shows that training on data selected by the group oracle outperforms both random selection and individual proxy oracle selection across all task settings and group sizes.

Table 1(a): Pre-training Results (Repro-400M, 400 steps)

Table 1(b): Post-training Results (Qwen2.5-1.5B-Instruct, 100 steps)

Finding 2: Gains Concentrate on Generative Capability

On free-form generation tasks (LAMBADA, SQuAD, CoQA), group oracle selection achieves +16.9% relative gain over the base model, far above individual proxy oracle selection (+6.3%) and random selection (+4.7%). On multiple-choice tasks, all methods are comparable (+3.8% to +4.8%).

Finding 3: Individual-Level Methods Collapse on Generative Tasks

As the rephrase factor $r$ increases (making data more synthetic), individual-level methods (GradSim, FineWeb-Edu, BM25) degrade on generative tasks, while group-level methods (GMRel, GREATS) maintain better performance.

Finding 4: Group-Level Methods Balance Value and Diversity

GradSim greedily selects highest-scoring samples, giving high group oracle value but least diverse groups. GMRel and GREATS form more diverse groups at a slight cost to group oracle value, suggesting that balancing value and diversity is important for effective synthetic data selection.

Finding 5: Stronger Relational Weighting Helps with Highly Synthetic Data

At $r = 50$, tripling GMRel's relational weight improves relative downstream performance by 4.7%, surpassing random selection and baseline.

Key Theoretical Result

Proposition 5.1 (Oracle gap). The individual proxy and true group oracles differ by at most:

$$I_{proxy}(G, \theta) - I_{orcl}(G, \theta) \leq \frac{\beta \eta^2}{2} \sigma^2_G$$

where $\beta \geq \|\nabla^2 L(\theta)\|$ bounds the curvature of the reference loss and $\sigma^2_G = \frac{1}{|G|} \sum_{z \in G} \|g_z - \bar{g}_G\|^2$ is the within-group gradient diversity.

The leading-order characterization is:

$$I_{proxy}(G, \theta) - I_{orcl}(G, \theta) = \frac{\eta^2}{2} \text{Tr}(H \Sigma_G) + O(\eta^3)$$

where $H = \nabla^2 L(\theta)$ is the Hessian of the reference loss and $\Sigma_G$ is the within-group gradient covariance.

---

Theoretical and Practical Implications

Theoretical Implications

1. The oracle gap is a second-order effect: The individual proxy and true group oracles agree to first order in $\eta$; their leading discrepancy is $O(\eta^2)$ and grows with both loss curvature $H$ and within-group gradient diversity $\Sigma_G$.

2. Gradient diversity as the key predictor: The gap vanishes when the group is perfectly coherent ($\Sigma_G = 0$) or the loss is locally flat ($H = 0$). Empirically, gradient diversity achieves Spearman correlation $\rho = 0.58\text{–}0.63$ with the oracle gap, outperforming alternatives like Vendi score (0.48) and compression ratio (0.57).

Practical Implications

Budget allocation strategy: Given a pool of groups $\mathcal{G} = \{G_1, \ldots, G_H\}$ and a budget of $B$ group oracle calls, the diagnostic $\nu: \mathcal{G} \rightarrow \mathbb{R}$ ranks groups by predicted oracle gap:

$$A_\nu = \underset{G \in \mathcal{G}}{\text{arg top } B} \nu(G)$$

Each group is then scored with the best oracle affordable:

$$\hat{I}_A(G, \theta) = \begin{cases} I_{orcl}(G, \theta), & G \in A_\nu \\ I_{proxy}(G, \theta), & \text{otherwise} \end{cases}$$

Key results:

• At a 20% budget, GradDiv recovers 75% of top-10% groups (vs. 20% under random allocation)
• Subsampling gradients at $q = 0.1$ gives similar benefit (recall 0.69 vs. 0.75)
• GradDiv at $q = 0.1$ recovers 74% of group oracle's recall using only 27% of its memory

The cumulative memory ratio for any allocation is:

$$\frac{\text{Mem}(B, q)}{\text{Mem}_{orcl}} = \frac{B}{H} + \kappa q \frac{3|G|}{3|G| + R} \approx \frac{B}{H} + \kappa q$$

where $\kappa = (2W + A)/(4W + A)$ is the memory discount of a gradient pass relative to an oracle call.

---

Conclusion

Main Takeaways

1. Group-level signals are necessary for effective synthetic data curation. Individual-level signals are blind to how samples jointly affect training, leading to suboptimal data selection.

2. The gap between individual and group-level signals widens as pipelines become more synthetic. Only data curation methods incorporating group-level signals improve over baseline, with gains concentrated in generative capability.

3. Gradient diversity provides a principled, cheap diagnostic for deciding where to allocate group-scoring budget under compute constraints.

Future Directions

The authors motivate future work on scalable group-level curation methods that model relational structure directly, rather than relying on expensive oracle scoring. Key open questions include:

• How to scale group-level influence estimation to frontier-scale models
• How to integrate group-level signals into existing data curation pipelines efficiently
• How to model higher-order interactions beyond pairwise relationships within groups