# Learning with Synthetic Data via SGD in High-Dimensional Linear Regression

> Mixed training with synthetic data causes strong model collapse with a non-vanishing error floor, while two-stage training avoids it entirely by using synthetic data as initialization.

- **Source:** [arXiv](https://arxiv.org/abs/2609.09572)
- **Published:** 2026-09-12
- **Permalink:** https://picx.dev/p/SurkGN
- **Whiteboard:** https://picx.dev/p/SurkGN/image

## Summary

## Summary (Overview)

- **Core question**: How does synthetic data affect the generalization of one-pass SGD in high-dimensional linear regression when real and synthetic data share covariates but differ in labeling functions (model shift)?
- **Key contrast**: Mixed training (interleaving real and synthetic samples) induces *strong model collapse*—a non-vanishing excess risk floor of order $p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$—while two-stage training (synthetic first, real second) avoids this floor entirely.
- **Scaling laws**: Under a random sketch model, larger model dimensions amplify synthetic-induced degradation in mixed training (drift floor $p^2(1-D^{1-b})$), but high-quality synthetic pretraining reduces bias in two-stage training.
- **Exact condition**: A necessary-and-sufficient finite-sample condition is established for two-stage training to strictly outperform real-only training, characterized by whether the synthetic initialization is closer to the target than the original initialization in directions that matter for real-data updates.
- **Main message**: Synthetic data is neither inherently harmful nor beneficial; its effect depends critically on both its quality (mismatch magnitude) and the training protocol used to incorporate it.

## Introduction and Theoretical Foundation

### Background and Motivation

Large language models (LLMs) have driven remarkable progress, but high-quality pre-training data is limited and rapidly diminishing. Synthetic data—generated by existing models—has emerged as a promising supplement. However, prior work (Shumailov et al., 2023) showed that training with synthetic data can induce *model collapse*, where iterative use of generated data leads to critical performance degradation.

### Strong Model Collapse

Dohmatob et al. (2024b) introduced the concept of **strong model collapse**: even a small proportion of synthetic data prevents model performance from improving under data scaling, leaving a non-vanishing excess risk floor. This serves as a key motivation for the present analysis. However, most prior theoretical work focuses on:

- Static (batch) estimators rather than iterative optimization dynamics (e.g., SGD)
- Asymptotic regimes rather than finite-sample characterizations

### Problem Setup

The paper considers a two-source setting with:
- **Real distribution** $\mathcal{P}_1$: $y = \langle \mathbf{x}, \mathbf{w}_1^*\rangle + \xi_1$
- **Synthetic distribution** $\mathcal{P}_2$: $y = \langle \mathbf{x}, \mathbf{w}_2^*\rangle + \xi_2$

Both share the same feature marginal $\mathbf{x} \sim \mathcal{D}$, differing only in labeling functions. The **model shift** is defined as:
$$\boldsymbol{\delta} := \mathbf{w}_2^* - \mathbf{w}_1^*$$

The goal is to minimize the population risk on the real distribution:
$$\mathcal{R}(\mathbf{w}) := \frac{1}{2}\mathbb{E}_{(\mathbf{x},y)\sim\mathcal{P}_1}(\langle\mathbf{x},\mathbf{w}\rangle - y)^2$$

### Training Protocols

1. **Mixed training**: $M$ synthetic + $N$ real samples presented in random order with synthetic proportion $p = M/(M+N)$
2. **Two-stage training**: First train on $M$ synthetic samples, then on $N$ real samples

### SGD with Geometric Decaying Stepsizes

$$\mathbf{w}_{t+1} = \mathbf{w}_t - \gamma_t(\langle\mathbf{x}_t,\mathbf{w}_t\rangle - y_t)\mathbf{x}_t$$

with stepsize schedule:
$$\gamma_t := \gamma/2^{\ell_t}, \quad \ell_t = \lfloor t/(T/\log(T))\rfloor$$

### Key Assumptions

- **Assumption 3.1**: Data covariance $\mathbf{H} = \mathbb{E}[\mathbf{x}\mathbf{x}^\top]$ has finite trace
- **Assumption 3.2**: Fourth moment conditions with constants $\alpha, \beta > 0$
- **Assumption 3.3**: Well-specified linear models with Gaussian noise $\xi_\ell \sim N(0, \sigma_\ell^2)$

## Methodology

### Analytical Framework

The paper decomposes the excess risk into interpretable components:

**For mixed training** (Theorem 4.1), the expected excess risk decomposes as:

$$\mathbb{E}[\mathcal{E}_1(\mathbf{w}_{M+N})] \lesssim \underbrace{\text{BiasError}}_{\text{standard}} + \underbrace{\text{VarError}}_{\text{standard}} + \underbrace{\text{FlucError}}_{\text{new}} + \underbrace{\text{DriftError}}_{\text{new}}$$

where:
- **BiasError**: $\|\prod_{t=1}^{M+N}(\mathbf{I} - \gamma_t\mathbf{H})(\mathbf{w}_0 - \mathbf{w}_1^*)\|_{\mathbf{H}}^2$ — optimization error from initialization
- **VarError**: $\bar{\sigma}^2 \frac{\widetilde{D}_{\text{eff}}}{\widetilde{N}_{\text{eff}}}$ — label noise variance
- **FlucError**: $\alpha p\|\boldsymbol{\delta}\|_{\mathbf{H}}^2 \frac{\widetilde{D}_{\text{eff}}}{\widetilde{N}_{\text{eff}}}$ — stochastic mixing of sources
- **DriftError**: $p^2\|(\mathbf{I} - \prod_{t=1}^{M+N}(\mathbf{I} - \gamma_t\mathbf{H}))\boldsymbol{\delta}\|_{\mathbf{H}}^2$ — systematic drift toward synthetic model

Key quantities: $\widetilde{N}_{\text{eff}} = (M+N)/\log(M+N)$, $\bar{\sigma}^2 = (1-p)\sigma_1^2 + p\sigma_2^2$, and effective dimension $\widetilde{D}_{\text{eff}} = k^* + \gamma^2\widetilde{N}_{\text{eff}}^2\sum_{i>k^*}\lambda_i^2$ where $k^* = \max\{k: \lambda_k \geq 1/(\gamma\widetilde{N}_{\text{eff}})\}$.

### Random Sketch Model

To study scaling laws with model size, the paper adopts a Gaussian random sketch operator $\mathbf{S}: \mathcal{H} \to \mathbb{R}^D$ with entries from $N(0, 1/D)$. The sketched model trains:
$$f_{\mathbf{v}}(\mathbf{x}) = \langle \mathbf{v}, \mathbf{S}\mathbf{x}\rangle, \quad \mathbf{v} \in \mathbb{R}^D$$

The sketch dimension $D$ plays the role of model size. Additional assumptions include:
- **Assumption 4.6**: Gaussian features, $\mathbb{E}[\mathbf{w}_1^*(\mathbf{w}_1^*)^\top] = \mathbf{I}$, $\mathbb{E}[\boldsymbol{\delta}\boldsymbol{\delta}^\top] = \boldsymbol{\Sigma}$
- **Assumption 4.7**: Power-law spectrum $\lambda_i \asymp i^{-a}$ for $a > 1$, and mismatch energy $\mathbb{E}[\lambda_i\langle\mathbf{v}_i, \boldsymbol{\delta}\rangle^2] \asymp i^{-b}$ for $b > 1$

The exponent $b$ characterizes how "hard" the mismatch is relative to the data spectrum.

## Empirical Validation / Results

### Key Theorems and Findings

**Theorem 4.1 (Mixed training upper bound)**: The excess risk is bounded by four terms: bias, variance, fluctuation, and drift. As $M+N \to \infty$ with fixed $p$, all terms except drift vanish, giving a non-zero floor of order $p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$.

**Theorem 4.2 (Mixed training lower bound)**: Shows this floor cannot be avoided in general, with the lower bound converging to the same limit $p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$.

**Corollary 4.3 (Strong model collapse)**:
$$\lim_{M+N \to \infty}\mathbb{E}[\mathcal{E}_1(\mathbf{w}_{M+N})] \sim p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$$

This persists for both last-iterate and iterate-averaged SGD, showing collapse is induced by sample mixing, not the optimization scheme.

**Theorem 4.4 (Two-stage training)**: Excess risk bounded by:

$$\mathbb{E}[\mathcal{E}_1(\mathbf{w}_{M+N})] \lesssim \underbrace{\mathbb{E}[\mathcal{E}_2(\mathbf{w}_M)] + \left\|\prod_{t=1}^{N}(\mathbf{I} - \gamma_t\mathbf{H})\boldsymbol{\delta}\right\|_{\mathbf{H}}^2}_{\text{EffectiveBias}} + \underbrace{(\alpha\mathbb{E}[\mathcal{E}_2(\mathbf{w}_M)] + \alpha\|\boldsymbol{\delta}\|_{\mathbf{H}}^2 + \sigma_1^2)\frac{D_{\text{eff}}}{N_{\text{eff}}}}_{\text{EffectiveVariance}}$$

**Corollary 4.5 (No strong model collapse)**: $\lim_{M+N\to\infty}\mathbb{E}[\mathcal{E}_1(\mathbf{w}_{M+N})] = 0$ under two-stage training.

### Scaling Laws

**Theorem 4.8 (Mixed training scaling)**:
$$\mathbb{E}\mathcal{R}_D(\mathbf{v}_{M+N}) - \mathcal{R}(\mathbf{w}_1^*) \lesssim \underbrace{\frac{1}{D^{a-1}} + \frac{1}{(\widetilde{N}_{\text{eff}})^{1-1/a}}}_{\text{Approx + Bias}} + \underbrace{p^2(1 - \frac{1}{D^{b-1}})}_{\text{Drift Floor}}$$

In the optimization-saturated regime ($\gamma N_{\text{eff}} \gtrsim D^a$), this is tight:
$$\approx D^{1-a} + \widetilde{N}_{\text{eff}}^{1/a-1} + p^2(1 - D^{1-b})$$

**Theorem 4.9 (Two-stage scaling upper bound)**:
$$\lesssim \underbrace{\frac{1}{D^{a-1}}}_{\text{Approx}} + \underbrace{\frac{\min\{D, N_{\text{eff}}^{1/a}\}}{N_{\text{eff}}}}_{\text{Variance}} + \underbrace{e^{-\Omega(N_{\text{eff}}/D^a)}\cdot\max\{D^{1-a\wedge b}, M_{\text{eff}}^{-(a\wedge b-1)/a}\} + \max\{D^{1-b}, N_{\text{eff}}^{-(b-1)/a}\}}_{\text{Bias}}$$

### Exact Condition for Synthetic Benefit

**Theorem 4.10**: Two-stage training strictly outperforms real-only training if and only if:
$$\|\mathbb{E}[\mathbf{w}_M] - \mathbf{w}_1^*\|_{\mathbf{K}_N}^2 + \text{tr}(\mathbf{K}_N\text{Cov}(\mathbf{w}_M)) < \|\mathbf{w}_0 - \mathbf{w}_1^*\|_{\mathbf{K}_N}^2$$

where $\mathbf{K}_N = \mathbb{E}_{\mathbf{x}_1,\ldots,\mathbf{x}_N}[\Phi_N^\top\mathbf{H}\Phi_N]$ with $\Phi_N = \prod_{t=N}^{1}(\mathbf{I} - \gamma_t\mathbf{x}_t\mathbf{x}_t^\top)$.

**Corollary 4.11 (One-dimensional illustration)**:
$$\mathbb{E}[\mathcal{E}_1(w_{M+N})] - \mathbb{E}[\mathcal{E}_1(w_N^{\text{real}})] = \frac{1}{2}\left(\prod_{t=1}^{N}(1-\gamma_t)\right)^2\left[\delta^2 + \frac{\sigma_2^2}{M} - (w_0-\theta)^2\right]$$

Synthetic pretraining strictly helps if and only if $\delta^2 + \sigma_2^2/M < (w_0-\theta)^2$, achievable when $\delta^2 < (w_0-\theta)^2$ with requirement $M > \sigma_2^2/((w_0-\theta)^2 - \delta^2)$.

### Experimental Results

**Figure 1**: (a) Mixed training shows a clear p-dependent error floor as T scales; two-stage training continues improving. (b) Under mixing, excess risk scales approximately linearly with $p^2$, consistent with $p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$. (c) Degradation grows with model size D and saturates, consistent with $p^2(1-D^{1-b})$.

**Figure 2**: (a) Low-quality synthetic data induces larger error floor in mixed training. (b) Two-stage training improves with T for all synthetic qualities. (c) Synthetic data helps when quality is high (large b), but low-quality synthetic data hurts as M increases. (d) Benefit region shows synthetic data helps only when teacher mismatch is benign and label noise is not too large.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Bridge between static estimators and optimization dynamics**: The paper extends the strong model collapse phenomenon from batch estimators to SGD dynamics, showing collapse arises intrinsically from sample mixing in optimization.

2. **Curriculum matters**: The sharp contrast between mixed and two-stage training demonstrates that data ordering/scheduling is as important as the data itself—a simple curriculum avoids collapse.

3. **Model size amplifies degradation**: The scaling laws show larger models may amplify synthetic-induced degradation under mixing, revealing a fundamental trade-off between approximation power and exposure to source mismatch.

4. **Bias reduction mechanism**: Two-stage training benefits from synthetic data primarily through bias reduction (better initialization), not variance reduction.

### Practical Implications

- **Training protocol design**: Practitioners should prefer two-stage (pretrain on synthetic, fine-tune on real) over mixed training when incorporating synthetic data.
- **Quality thresholds**: The exact condition provides a practical criterion for when synthetic pretraining helps, with explicit sample-size requirements.
- **Model scaling caution**: Larger models may be more susceptible to synthetic contamination under mixing, suggesting careful evaluation when scaling model size with mixed data.

## Conclusion

The paper establishes that synthetic data's effect on SGD training depends critically on both its quality and the training protocol:

- **Mixed training** induces strong model collapse with a non-vanishing excess risk floor of order $p^2\|\boldsymbol{\delta}\|_{\mathbf{H}}^2$, which cannot be avoided even with infinite data.
- **Two-stage training** avoids collapse entirely, with synthetic data acting as a bias-reduction mechanism through improved initialization.
- **Scaling laws** reveal that larger models amplify synthetic-induced degradation under mixing, while high-quality synthetic pretraining reduces bias under two-stage training.
- An **exact finite-sample condition** characterizes when two-stage training strictly outperforms real-only training.

### Future Directions

The paper identifies several open questions:
- Extension to joint covariate and model shift
- Multi-epoch SGD analysis
- Tight characterizations for data-limited regimes
- Matching lower bounds for two-stage training scaling laws

---

_Markdown view of https://picx.dev/p/SurkGN, served by PicX — AI-generated visual whiteboard summaries of research papers._