# Conditional Transfer from Controlled Pretraining Mixtures to Code

> Loss-based data quality metrics mislead mixture selection: target-aligned sources dominate, with curated synthetic data optimal at a bounded 25% share, beating pure or adaptive loss-reduced mixtures.

- **Source:** [arXiv](https://arxiv.org/abs/2610.11548)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/ikojIV
- **Whiteboard:** https://picx.dev/p/ikojIV/image

## Summary

# Conditional Transfer from Controlled Pretraining Mixtures to Code

**Author:** Ohad Rubin

---

## Summary (Overview)

- **Three separable signals:** The paper distinguishes between *diagnosticity* (task loss tracks global pretraining progress), *teachability* (task loss responds to its own token budget), and *transfer* (a source's inclusion improves a downstream target) — signals that are usually conflated in loss-based data quality reasoning.

- **Teachability is common but family-dependent:** Across a per-task budget sweep, 14 of 27 tasks show detectable teachability, with a sharp asymmetry: 10/12 curated code-adjacent tasks versus only 4/15 literature-derived probes are teachable (FDR-adjusted: 6/12 vs. 2/15).

- **Transfer is conditional on mixture context:** On the curated–OpenCodeInstruct edge, HumanEval pass@20 is non-monotone in the curated share, peaking at **22.6** when OpenCodeInstruct holds 75% of the slice, while pure curated data falls to **15.9** — below the 16.5% zero-synthetic baseline.

- **Loss-based scheduling follows the wrong signal:** Ado (an adaptive loss-reduction scheduler) drives the OpenCodeInstruct share below 5% within the first 5k steps and to 1.2–1.4% by training end, underperforming matched fixed-mixture controls by **2.4–11.0 percentage points** of pass@20.

- **Practical rule:** For code generation targets, keep the target-aligned source dominant, add curated synthetic data as a bounded share, and choose proportions with a target-referenced signal rather than loss-reducibility alone.

---

## Introduction and Theoretical Foundation

### Motivation

A pretraining corpus is a mixture, and choosing the mixture is a first-class design problem. Two questions are typically asked of a candidate source:

1. **Diagnostic:** Does performance on it tell us something about the model?
2. **Economic:** Does buying more of it make the model better at what we care about?

The paper argues these questions are often collapsed into a single loss-based notion of data quality, which can be misleading. A synthetic task whose loss falls steadily during pretraining is compatible with at least three distinct situations:

- The loss falls because **general competence is rising** and the task reads it off (diagnostic value only).
- The loss falls because the model is **fitting the task's surface form** (responds to budget, but possibly locally).
- The loss falls for both reasons while the source **displaces something the downstream target needs** (actively harmful).

### Theoretical Foundation: Three Definitions

**Definition 1 (Diagnosticity):** Task $t$ is diagnostic when its held-out loss tracks global pretraining progress. Summarized by $R^2_{t,\text{step}}$ — the coefficient of determination from regressing logged validation loss on global pretraining step, pooled across the task's nine runs.

**Definition 2 (Teachability):** Task $t$ is teachable if its held-out loss responds to its own token budget across runs. This requires a cross-run comparison via a task budget sweep.

**Definition 3 (Transfer):** A family $f$ transfers at $w$ relative to $w'$ when increasing its share from the feasible mixture $w'$ to $w$ improves the downstream endpoint: $y(w) > y(w')$. Transfer is deliberately *local* because the simplex constrains every increase to displace another source.

**Exposure Advantage:** To compare diagnosticity and teachability explanations, the paper defines:
$$A_t = R^2_{t,\text{exposure}} - R^2_{t,\text{step}}$$
where the first term regresses logged validation loss on cumulative exposure. Seven tasks have $A_t > 0$, and five of the six largest values are curated tasks.

---

## Methodology

### Experimental Setup

**Base corpus and controlled slice:** Every pretraining run draws 70% of tokens from a fixed general Python corpus (CodeParrot) and 30% from a controlled slice whose composition is the only manipulated variable. A run is specified by a point $w = (w_O, w_C, w_L)$ on the 2-simplex $\Delta^2$:

$$P_r(x) = 0.70 P_{\text{code}}(x) + 0.30 \left[ w_{r,O} P_O(x) + w_{r,C} \sum_{t \in \mathcal{T}_C} w_{t|C} P_t(x) + w_{r,L} \sum_{t \in \mathcal{T}_L} w_{t|L} P_t(x) \right] \tag{1}$$

Within-source task weights sum to one and remain fixed across runs.

### Source Families

| Family | Description | Size |
|--------|-------------|------|
| **OpenCodeInstruct** | Target-aligned instruction-formatted code corpus (EER6/nvidia-OpenCodeInstruct-refined) | 444,611 examples |
| **Curated suite** | 12 procedurally generated code-adjacent tasks (execution, search, state mutation, logical inference, planning, ranking, binding) | 12 tasks |
| **Literature suite** | 15 machine-generated standard probes (copy, reverse sequence, majority, XOR parity, associative recall, Dyck completion, ListOps, passkey retrieval, variable tracing, SCAN, bAbI tasks, constant/random label) | 15 tasks |

### Model and Training

- **Architecture:** Sparse-expert decoder-only model with top-2 expert routing, 50,304-token vocabulary, 12 layers, 12 attention heads, width 768, SwiGLU width 2,048
- **Training:** Context length 1,024, effective batch size 256, 262,144 tokens per optimizer step
- **Optimizer:** AdamW, learning rate $10^{-3}$, $\beta_1 = 0.9$, weight decay $10^{-4}$, 500 warmup steps
- **Runs:** 60k or 120k steps with decay horizon matched to run length
- **Shadow sweep:** 15 fixed mixtures along every pairwise edge in 25-percentage-point steps

### Downstream Evaluation

Pretraining is followed by an identical fine-tuning stage (1,000 updates on refined OpenCodeInstruct, batch size 256, Adam with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-6}$, peak LR $10^{-5}$). HumanEval pass@k uses the unbiased estimator:

$$\widehat{\text{pass@}k} = \mathbb{E}_{\text{problems}} \left[ 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}} \right] \tag{2}$$

where $n = 20$ and $c$ is the number of correct samples. The **Best over steps** endpoint is the highest score among fine-tuning checkpoints 100, 200, ..., 1000.

### Teachability Detection

For task $t$ in run $r$, first fit:
$$\log L_{t,r}(s) = \alpha_{t,r} + \beta_{t,r} \log s + \epsilon_{t,r,s} \tag{3}$$
over all readouts in the window, evaluating the fitted log-loss fall $\Delta_{t,r}$ between endpoints. Then fit:
$$\Delta_{t,r} = a_t + b_t \log_2 e_{t,r} + \varepsilon_{t,r} \tag{4}$$

The slope $b_t$ reports learning per doubling of exposure. A one-sided permutation test (2,000 shuffles, p < 0.05) declares detection.

---

## Empirical Validation / Results

### Teachability Results

**Fourteen of 27 tasks** clear the permutation test: 10/12 curated and 4/15 literature-derived. Benjamini–Hochberg adjustment leaves 8 detections (6/12 curated, 2/15 literature-derived).

Curated tasks emit a median of 10.2 supervised tokens per example versus 1.0 for literature probes. However, length is only weakly related to explained variance (Pearson r = 0.34, p = 0.08; Spearman ρ = 0.27, p = 0.18), so emitted length does not fully explain the family difference.

### Mixture-Level Transfer: The Central Result

**Table 1: HumanEval pass@20 along the curated–OpenCodeInstruct edge** ($w_L = 0$)

| Slice composition | $w_O$ | $w_C$ | HumanEval pass@20 |
|---|---|---|---|
| Pure curated | 0.00 | 1.00 | **15.9** |
| 75% curated | 0.25 | 0.75 | 18.9 |
| 50/50 | 0.50 | 0.50 | 20.7 |
| 25% curated | 0.75 | 0.25 | **22.6** |
| Pure OpenCodeInstruct | 1.00 | 0.00 | 19.5 |

**Key findings:**
- The highest observed point is **interior** to the edge at 75% OpenCodeInstruct / 25% curated, exceeding pure OpenCodeInstruct by **3.1 percentage points**
- Pure curated data scores **3.6 points below** pure OpenCodeInstruct and **0.6 points below** the 16.5% zero-synthetic baseline
- The ranking is consistent at k = 5, 10, 20

### Adaptive Scheduling Results

**Table 2: 60k-step, four-expert mixture-profile and scheduler comparison**

| Run | pass@5 | pass@10 | pass@20 | Δ shadow | Final OpenCodeInstruct share |
|---|---|---|---|---|---|
| Zero-synthetic baseline | 11.4% | 13.8% | 16.5% | — | — |
| Curated-heavy shadow | 12.7% | 15.5% | 18.9% | +0.0 pp | 25.0% |
| + Ado preserved ratio | 11.0% | 13.9% | 17.1% | -1.8 pp | 25.0% |
| + Ado free ratio | 8.8% | 12.4% | 16.5% | **-2.4 pp** | 1.2% |
| OpenCodeInstruct-heavy shadow | 13.3% | 16.5% | 20.1% | +0.0 pp | 50.0% |
| + Ado preserved ratio | 13.3% | 16.0% | 18.9% | -1.2 pp | 50.0% |
| + Ado free ratio | 3.7% | 6.0% | 9.1% | **-11.0 pp** | 1.4% |
| Literature-heavy shadow | 13.6% | 16.8% | 20.1% | +0.0 pp | 25.0% |
| + Ado preserved ratio | 11.0% | 13.7% | 17.1% | -3.0 pp | 25.0% |
| + Ado free ratio | 10.2% | 12.7% | 15.2% | **-4.9 pp** | 1.2% |

**Critical observations:**
- All three 60k free-ratio runs drive OpenCodeInstruct below 5% of the controlled slice within the first 5k steps, finishing at 1.2–1.4%
- Free-ratio Ado trails matched shadows by 2.4, 11.0, and 4.9 percentage points
- In 120k branches, the scheduler only begins moving mass back around step 90k — revealing **temporal myopia** in the 1,000-step fitting horizon

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Diagnosticity ≠ Teachability ≠ Transfer.** The paper provides a rigorous demonstration that these three signals, usually read off the same loss curve, are logically independent. A task can combine strong progress tracking with weak budget response, or vice versa.

2. **Teachability does not order downstream transfer.** The curated suite has the highest teachability rate (10/12) yet its pure-source endpoint (15.9%) falls below both pure OpenCodeInstruct and the zero-synthetic baseline. Teachability is insufficient as the sole mixture objective.

3. **Loss-reducibility is a misleading objective for adaptive scheduling.** Ado maximizes predicted near-term loss reduction, which scores OpenCodeInstruct low early in training despite its high downstream value. The scheduler's 1,000-step horizon creates temporal myopia — the aligned source only becomes attractive under its own criterion after ~88k steps.

4. **Curated synthetic data has a bounded dose-response.** Its value is real but non-monotone: +3.1 points at 25% share, but -0.6 points below baseline at 100% share. The bound is set by how much target-aligned data the curated source displaces.

### Practical Implications

The paper's practical rule is narrow but concrete:

> "When the downstream target is code generation, keep the target-aligned source dominant in the mixture, add curated synthetic data as a bounded share, and choose proportions with a target-referenced signal."

Standard probes (the literature suite) remain useful as **instruments** — controlled readouts of retrieval, state tracking, composition, and transduction — even when their own training budget shows weak evidence of marginal learning value.

---

## Conclusion

The paper separates three signals usually conflated in loss-curve analysis. Under a controlled pretraining setup where only a 30% slice varies:

- **14 of 27 tasks** respond detectably to their own exposure
- Family-level teachability ranks differently from downstream code transfer
- The best observed HumanEval pass@20 (**22.6**) came from a mixture with 75% target-aligned source and 25% curated synthetic data
- The same curated suite alone fell **below the zero-synthetic baseline**
- A loss-based adaptive scheduler drove the aligned source below 5% and finished below all three mixture-matched fixed controls

### Future Directions

- **Denser simplex grids** to identify precise optima (the point (0.75, 0.25, 0) is the best observed, not proven optimal)
- **Task-isolated interventions** to decouple teachability estimates from mixture coupling
- **Transfer-aware schedulers** incorporating target-referenced marginal-value signals with multi-horizon evaluation
- **Other architectures, scales, and base corpora** to test generality
- **Additional downstream targets** beyond HumanEval (other code benchmarks, general reasoning)

---

_Markdown view of https://picx.dev/p/ikojIV, served by PicX — AI-generated visual whiteboard summaries of research papers._