Summary (Overview)

• Core question: How does synthetic data affect the generalization of one-pass SGD in high-dimensional linear regression when real and synthetic data share covariates but differ in labeling functions (model shift)?
• Key contrast: Mixed training (interleaving real and synthetic samples) induces strong model collapse—a non-vanishing excess risk floor of order $p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$—while two-stage training (synthetic first, real second) avoids this floor entirely.
• Scaling laws: Under a random sketch model, larger model dimensions amplify synthetic-induced degradation in mixed training (drift floor $p^2(1-D^{1-b})$), but high-quality synthetic pretraining reduces bias in two-stage training.
• Exact condition: A necessary-and-sufficient finite-sample condition is established for two-stage training to strictly outperform real-only training, characterized by whether the synthetic initialization is closer to the target than the original initialization in directions that matter for real-data updates.
• Main message: Synthetic data is neither inherently harmful nor beneficial; its effect depends critically on both its quality (mismatch magnitude) and the training protocol used to incorporate it.

Introduction and Theoretical Foundation

Background and Motivation

Large language models (LLMs) have driven remarkable progress, but high-quality pre-training data is limited and rapidly diminishing. Synthetic data—generated by existing models—has emerged as a promising supplement. However, prior work (Shumailov et al., 2023) showed that training with synthetic data can induce model collapse, where iterative use of generated data leads to critical performance degradation.

Strong Model Collapse

Dohmatob et al. (2024b) introduced the concept of strong model collapse: even a small proportion of synthetic data prevents model performance from improving under data scaling, leaving a non-vanishing excess risk floor. This serves as a key motivation for the present analysis. However, most prior theoretical work focuses on:

• Static (batch) estimators rather than iterative optimization dynamics (e.g., SGD)
• Asymptotic regimes rather than finite-sample characterizations

Problem Setup

The paper considers a two-source setting with:

• Real distribution $\mathcal{P}_1$: $y = \langle \mathbf{x}, \mathbf{w}_1^*\rangle + \xi_1$
• Synthetic distribution $\mathcal{P}_2$: $y = \langle \mathbf{x}, \mathbf{w}_2^*\rangle + \xi_2$

Both share the same feature marginal $\mathbf{x} \sim \mathcal{D}$, differing only in labeling functions. The model shift is defined as:

$$\boldsymbol{\delta} := \mathbf{w}_2^* - \mathbf{w}_1^*$$

The goal is to minimize the population risk on the real distribution:

$$\mathcal{R}(\mathbf{w}) := \frac{1}{2}\mathbb{E}_{(\mathbf{x},y)\sim\mathcal{P}_1}(\langle\mathbf{x},\mathbf{w}\rangle - y)^2$$

Training Protocols

1. Mixed training: $M$ synthetic + $N$ real samples presented in random order with synthetic proportion $p = M/(M+N)$
2. Two-stage training: First train on $M$ synthetic samples, then on $N$ real samples

SGD with Geometric Decaying Stepsizes

$$\mathbf{w}_{t+1} = \mathbf{w}_t - \gamma_t(\langle\mathbf{x}_t,\mathbf{w}_t\rangle - y_t)\mathbf{x}_t$$

with stepsize schedule:

$$\gamma_t := \gamma/2^{\ell_t}, \quad \ell_t = \lfloor t/(T/\log(T))\rfloor$$

Key Assumptions

• Assumption 3.1: Data covariance $\mathbf{H} = \mathbb{E}[\mathbf{x}\mathbf{x}^\top]$ has finite trace
• Assumption 3.2: Fourth moment conditions with constants $\alpha, \beta > 0$
• Assumption 3.3: Well-specified linear models with Gaussian noise $\xi_\ell \sim N(0, \sigma_\ell^2)$

Methodology

Analytical Framework

The paper decomposes the excess risk into interpretable components:

For mixed training (Theorem 4.1), the expected excess risk decomposes as:

$$\mathbb{E}[\mathcal{E}_1(\mathbf{w}_{M+N})] \lesssim \underbrace{\text{BiasError}}_{\text{standard}} + \underbrace{\text{VarError}}_{\text{standard}} + \underbrace{\text{FlucError}}_{\text{new}} + \underbrace{\text{DriftError}}_{\text{new}}$$

where:

• BiasError: $\|\prod_{t=1}^{M+N}(\mathbf{I} - \gamma_t\mathbf{H})(\mathbf{w}_0 - \mathbf{w}_1^*)\|_{\mathbf{H}}^2$ — optimization error from initialization
• VarError: $\bar{\sigma}^2 \frac{\widetilde{D}_{\text{eff}}}{\widetilde{N}_{\text{eff}}}$ — label noise variance
• FlucError: $\alpha p\|\boldsymbol{\delta}\|_{\mathbf{H}}^2 \frac{\widetilde{D}_{\text{eff}}}{\widetilde{N}_{\text{eff}}}$ — stochastic mixing of sources
• DriftError: $p^2\|(\mathbf{I} - \prod_{t=1}^{M+N}(\mathbf{I} - \gamma_t\mathbf{H}))\boldsymbol{\delta}\|_{\mathbf{H}}^2$ — systematic drift toward synthetic model

Key quantities: $\widetilde{N}_{\text{eff}} = (M+N)/\log(M+N)$, $\bar{\sigma}^2 = (1-p)\sigma_1^2 + p\sigma_2^2$, and effective dimension $\widetilde{D}_{\text{eff}} = k^* + \gamma^2\widetilde{N}_{\text{eff}}^2\sum_{i>k^*}\lambda_i^2$ where $k^* = \max\{k: \lambda_k \geq 1/(\gamma\widetilde{N}_{\text{eff}})\}$.

Random Sketch Model

To study scaling laws with model size, the paper adopts a Gaussian random sketch operator $\mathbf{S}: \mathcal{H} \to \mathbb{R}^D$ with entries from $N(0, 1/D)$. The sketched model trains:

$$f_{\mathbf{v}}(\mathbf{x}) = \langle \mathbf{v}, \mathbf{S}\mathbf{x}\rangle, \quad \mathbf{v} \in \mathbb{R}^D$$

The sketch dimension $D$ plays the role of model size. Additional assumptions include:

• Assumption 4.6: Gaussian features, $\mathbb{E}[\mathbf{w}_1^*(\mathbf{w}_1^*)^\top] = \mathbf{I}$, $\mathbb{E}[\boldsymbol{\delta}\boldsymbol{\delta}^\top] = \boldsymbol{\Sigma}$
• Assumption 4.7: Power-law spectrum $\lambda_i \asymp i^{-a}$ for $a > 1$, and mismatch energy $\mathbb{E}[\lambda_i\langle\mathbf{v}_i, \boldsymbol{\delta}\rangle^2] \asymp i^{-b}$ for $b > 1$

The exponent $b$ characterizes how "hard" the mismatch is relative to the data spectrum.

Empirical Validation / Results

Key Theorems and Findings

Theorem 4.1 (Mixed training upper bound): The excess risk is bounded by four terms: bias, variance, fluctuation, and drift. As $M+N \to \infty$ with fixed $p$, all terms except drift vanish, giving a non-zero floor of order $p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$.

Theorem 4.2 (Mixed training lower bound): Shows this floor cannot be avoided in general, with the lower bound converging to the same limit $p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$.

Corollary 4.3 (Strong model collapse):

$$\lim_{M+N \to \infty}\mathbb{E}[\mathcal{E}_1(\mathbf{w}_{M+N})] \sim p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$$

This persists for both last-iterate and iterate-averaged SGD, showing collapse is induced by sample mixing, not the optimization scheme.

Theorem 4.4 (Two-stage training): Excess risk bounded by:

$$\mathbb{E}[\mathcal{E}_1(\mathbf{w}_{M+N})] \lesssim \underbrace{\mathbb{E}[\mathcal{E}_2(\mathbf{w}_M)] + \left\|\prod_{t=1}^{N}(\mathbf{I} - \gamma_t\mathbf{H})\boldsymbol{\delta}\right\|_{\mathbf{H}}^2}_{\text{EffectiveBias}} + \underbrace{(\alpha\mathbb{E}[\mathcal{E}_2(\mathbf{w}_M)] + \alpha\|\boldsymbol{\delta}\|_{\mathbf{H}}^2 + \sigma_1^2)\frac{D_{\text{eff}}}{N_{\text{eff}}}}_{\text{EffectiveVariance}}$$

Corollary 4.5 (No strong model collapse): $\lim_{M+N\to\infty}\mathbb{E}[\mathcal{E}_1(\mathbf{w}_{M+N})] = 0$ under two-stage training.

Scaling Laws

Theorem 4.8 (Mixed training scaling):

$$\mathbb{E}\mathcal{R}_D(\mathbf{v}_{M+N}) - \mathcal{R}(\mathbf{w}_1^*) \lesssim \underbrace{\frac{1}{D^{a-1}} + \frac{1}{(\widetilde{N}_{\text{eff}})^{1-1/a}}}_{\text{Approx + Bias}} + \underbrace{p^2(1 - \frac{1}{D^{b-1}})}_{\text{Drift Floor}}$$

In the optimization-saturated regime ($\gamma N_{\text{eff}} \gtrsim D^a$), this is tight:

$$\approx D^{1-a} + \widetilde{N}_{\text{eff}}^{1/a-1} + p^2(1 - D^{1-b})$$

Theorem 4.9 (Two-stage scaling upper bound):

$$\lesssim \underbrace{\frac{1}{D^{a-1}}}_{\text{Approx}} + \underbrace{\frac{\min\{D, N_{\text{eff}}^{1/a}\}}{N_{\text{eff}}}}_{\text{Variance}} + \underbrace{e^{-\Omega(N_{\text{eff}}/D^a)}\cdot\max\{D^{1-a\wedge b}, M_{\text{eff}}^{-(a\wedge b-1)/a}\} + \max\{D^{1-b}, N_{\text{eff}}^{-(b-1)/a}\}}_{\text{Bias}}$$

Exact Condition for Synthetic Benefit

Theorem 4.10: Two-stage training strictly outperforms real-only training if and only if:

$$\|\mathbb{E}[\mathbf{w}_M] - \mathbf{w}_1^*\|_{\mathbf{K}_N}^2 + \text{tr}(\mathbf{K}_N\text{Cov}(\mathbf{w}_M)) < \|\mathbf{w}_0 - \mathbf{w}_1^*\|_{\mathbf{K}_N}^2$$

where $\mathbf{K}_N = \mathbb{E}_{\mathbf{x}_1,\ldots,\mathbf{x}_N}[\Phi_N^\top\mathbf{H}\Phi_N]$ with $\Phi_N = \prod_{t=N}^{1}(\mathbf{I} - \gamma_t\mathbf{x}_t\mathbf{x}_t^\top)$.

Corollary 4.11 (One-dimensional illustration):

$$\mathbb{E}[\mathcal{E}_1(w_{M+N})] - \mathbb{E}[\mathcal{E}_1(w_N^{\text{real}})] = \frac{1}{2}\left(\prod_{t=1}^{N}(1-\gamma_t)\right)^2\left[\delta^2 + \frac{\sigma_2^2}{M} - (w_0-\theta)^2\right]$$

Synthetic pretraining strictly helps if and only if $\delta^2 + \sigma_2^2/M < (w_0-\theta)^2$, achievable when $\delta^2 < (w_0-\theta)^2$ with requirement $M > \sigma_2^2/((w_0-\theta)^2 - \delta^2)$.

Experimental Results

Figure 1: (a) Mixed training shows a clear p-dependent error floor as T scales; two-stage training continues improving. (b) Under mixing, excess risk scales approximately linearly with $p^2$, consistent with $p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$. (c) Degradation grows with model size D and saturates, consistent with $p^2(1-D^{1-b})$.

Figure 2: (a) Low-quality synthetic data induces larger error floor in mixed training. (b) Two-stage training improves with T for all synthetic qualities. (c) Synthetic data helps when quality is high (large b), but low-quality synthetic data hurts as M increases. (d) Benefit region shows synthetic data helps only when teacher mismatch is benign and label noise is not too large.

Theoretical and Practical Implications

Theoretical Contributions

1. Bridge between static estimators and optimization dynamics: The paper extends the strong model collapse phenomenon from batch estimators to SGD dynamics, showing collapse arises intrinsically from sample mixing in optimization.

2. Curriculum matters: The sharp contrast between mixed and two-stage training demonstrates that data ordering/scheduling is as important as the data itself—a simple curriculum avoids collapse.

3. Model size amplifies degradation: The scaling laws show larger models may amplify synthetic-induced degradation under mixing, revealing a fundamental trade-off between approximation power and exposure to source mismatch.

4. Bias reduction mechanism: Two-stage training benefits from synthetic data primarily through bias reduction (better initialization), not variance reduction.

Practical Implications

• Training protocol design: Practitioners should prefer two-stage (pretrain on synthetic, fine-tune on real) over mixed training when incorporating synthetic data.
• Quality thresholds: The exact condition provides a practical criterion for when synthetic pretraining helps, with explicit sample-size requirements.
• Model scaling caution: Larger models may be more susceptible to synthetic contamination under mixing, suggesting careful evaluation when scaling model size with mixed data.

Conclusion

The paper establishes that synthetic data's effect on SGD training depends critically on both its quality and the training protocol:

• Mixed training induces strong model collapse with a non-vanishing excess risk floor of order $p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$, which cannot be avoided even with infinite data.
• Two-stage training avoids collapse entirely, with synthetic data acting as a bias-reduction mechanism through improved initialization.
• Scaling laws reveal that larger models amplify synthetic-induced degradation under mixing, while high-quality synthetic pretraining reduces bias under two-stage training.
• An exact finite-sample condition characterizes when two-stage training strictly outperforms real-only training.

Future Directions

The paper identifies several open questions:

• Extension to joint covariate and model shift
• Multi-epoch SGD analysis
• Tight characterizations for data-limited regimes
• Matching lower bounds for two-stage training scaling laws