# Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning > KITE prevents model collapse in synthetic-data instruction tuning by targeting weak skills via DINA-based diagnosis and curating examples near the model's semantic knowledge boundary, achieving sustained monotonic improvement across benchmarks. - **Source:** [arXiv](https://arxiv.org/abs/2607.17043) - **Published:** 2026-09-05 - **Permalink:** https://picx.dev/p/6wb4Oh - **Whiteboard:** https://picx.dev/p/6wb4Oh/image ## Summary ## Summary (Overview) - **Core Problem**: Model collapse in iterative instruction tuning with synthetic data is not uniform degradation but a **polarization of competence** — synthetic training reinforces already-strong skills while further degrading weak ones. - **Key Contribution**: The paper proposes **KITE (Knowledge-boundary Instruction Tuning via Exploration)**, a two-stage framework combining failure-guided data generation with boundary-aware uncertainty curation. - **Diagnostic Finding**: Using the DINA cognitive assessment model, the authors show that fine-tuning on accumulated synthetic data strengthens skills like "Data" and "Money & Finance" while weakening "Algebra" and "Time & Scheduling" — revealing actionable granularity for data curation. - **Method**: Stage 1 generates candidate instructions targeting diagnosed weaknesses with rank-based noise injection; Stage 2 curates examples near the model's semantic knowledge boundary using a novel **Kernel Boundary Uncertainty (KBU)** score. - **Results**: KITE consistently outperforms strong baselines (Self-Instruct, Few-shot, CDS, ToEdit) across 4 benchmarks and 5 open-source LLMs, with stable long-horizon improvement over 9 generations. --- ## Introduction and Theoretical Foundation ### Background and Motivation Synthetic data has become essential in LLM post-training, but introduces a fundamental challenge: **model collapse** (Shumailov et al., 2024). As later-generation models train on data reflecting previous generations' outputs, the training distribution becomes distorted — losing diversity and fidelity to real-world data, reducing long-tail coverage, and harming held-out performance. ### Key Gap in Existing Work The paper identifies two critical gaps in prior research: 1. **Objective mismatch**: Existing work focuses on *bounding* degradation (keeping it finite), but in iterative model evolution, the meaningful goal is ensuring each successive model *improves* over its predecessor. 2. **Granularity problem**: Aggregate-level collapse analysis (overall accuracy drop, long-tail narrowing) is too coarse to guide synthetic data construction. Collapse could stem from data being too narrow/repetitive (needs diversification) or from specific skill weaknesses (needs targeted construction). ### Theoretical Foundation The paper leverages **DINA (Deterministic Inputs Noisy And gate)** model from cognitive assessment theory. DINA infers latent skill mastery from response patterns across many examples, treating each training instance as requiring a binary skill vector (from a Q-matrix). This provides structured weakness estimates beyond simple failure counting. ### The Polarization Finding Using GSM8K as an illustrative case, the authors estimate skill mastery for both base and fine-tuned models: > "Synthetic data improves several skills that were previously strong enough, including Data and Money & Finance. On the other hand, some weak skills not fully mastered by the base model become even weaker... such as Algebra and Time & Scheduling." This demonstrates that collapse in instruction tuning is **polarization of competence**, not uniform degradation. --- ## Methodology ### Notation and Formulation Let $M_t$ denote the model at evolution step $t$ with output distribution $p_{M_t}(y|x)$. The pipeline: 1. Start from verified training set $D_0^\star = \{(x_i, y_i^\star)\}_{i=1}^{n_0}$ 2. At step $t$: generate candidate bank $\mathcal{B}_t = \{x_i\}_{i=1}^{N}$ with $N \gg B$ 3. Select $D_t \subseteq \mathcal{B}_t$ with $|D_t| = B$ (labeling budget) 4. Label to $D_t^\star$, train next model on union of all verified data via SFT Each $M_t$ is fine-tuned from the **same base model** on accumulated data. ### Stage 1: Failure-Guided Candidate Bank Construction **1) Weakness profiling via DINA**: Each training instance has a binary skill requirement vector from a Q-matrix (constructed via LLM-assisted skill tagging). DINA estimates posterior mastery per skill; low-mastery skills become natural-language weakness descriptors $S_t = \{s_k\}_{k=1}^{K}$, used to instantiate question-generation prompts $\pi_Q(s)$. **2) Rank-based noise injection**: To avoid easy/templated instructions, the paper perturbs token probabilities by rank: $$ q_t^{(\alpha)}(v \mid c) = \operatorname{softmax}\left(\ell_t(c) + \alpha \log r_t(\cdot \mid c)\right)_v \propto q_t(v \mid c) \cdot r_t(v \mid c)^{\alpha} \tag{1} $$ where $r_t(v \mid c) = 1 + |\{u : \ell_{t,u}(c) > \ell_{t,v}(c)\}|$ is the rank of token $v$. This smoothly upweights lower-ranked tokens, encouraging exploration without hard truncation. ### Stage 2: Uncertainty-Based Data Curation (KBU) **Semantic uncertainty via kernel**: For each candidate $x$, sample $m$ answers $y^{(1)}, \ldots, y^{(m)} \sim M_t(\cdot \mid x)$, embed them as $h_i = \phi(y^{(i)})$, and define an RBF kernel: $$ K_{ij} = \exp\left(-\frac{\|h_i - h_j\|^2}{2\sigma^2}\right), \quad \sigma^2 = \operatorname{median}\{\|h_i - h_j\|^2 : i < j\} \tag{2} $$ **Likelihood weighting**: To suppress off-manifold low-probability generations, each sample is weighted by length-normalized log-likelihood: $$ \ell_i = \frac{1}{|y^{(i)}|} \sum_k \log p_{M_t}\left(y_k^{(i)} \mid x, y_{