Conditional Transfer from Controlled Pretraining Mixtures to Code

Author: Ohad Rubin

---

Summary (Overview)

• Three separable signals: The paper distinguishes between diagnosticity (task loss tracks global pretraining progress), teachability (task loss responds to its own token budget), and transfer (a source's inclusion improves a downstream target) — signals that are usually conflated in loss-based data quality reasoning.

• Teachability is common but family-dependent: Across a per-task budget sweep, 14 of 27 tasks show detectable teachability, with a sharp asymmetry: 10/12 curated code-adjacent tasks versus only 4/15 literature-derived probes are teachable (FDR-adjusted: 6/12 vs. 2/15).

• Transfer is conditional on mixture context: On the curated–OpenCodeInstruct edge, HumanEval pass@20 is non-monotone in the curated share, peaking at 22.6 when OpenCodeInstruct holds 75% of the slice, while pure curated data falls to 15.9 — below the 16.5% zero-synthetic baseline.

• Loss-based scheduling follows the wrong signal: Ado (an adaptive loss-reduction scheduler) drives the OpenCodeInstruct share below 5% within the first 5k steps and to 1.2–1.4% by training end, underperforming matched fixed-mixture controls by 2.4–11.0 percentage points of pass@20.

• Practical rule: For code generation targets, keep the target-aligned source dominant, add curated synthetic data as a bounded share, and choose proportions with a target-referenced signal rather than loss-reducibility alone.

---

Introduction and Theoretical Foundation

Motivation

A pretraining corpus is a mixture, and choosing the mixture is a first-class design problem. Two questions are typically asked of a candidate source:

1. Diagnostic: Does performance on it tell us something about the model?
2. Economic: Does buying more of it make the model better at what we care about?

The paper argues these questions are often collapsed into a single loss-based notion of data quality, which can be misleading. A synthetic task whose loss falls steadily during pretraining is compatible with at least three distinct situations:

• The loss falls because general competence is rising and the task reads it off (diagnostic value only).
• The loss falls because the model is fitting the task's surface form (responds to budget, but possibly locally).
• The loss falls for both reasons while the source displaces something the downstream target needs (actively harmful).

Theoretical Foundation: Three Definitions

Definition 1 (Diagnosticity): Task $t$ is diagnostic when its held-out loss tracks global pretraining progress. Summarized by $R^2_{t,\text{step}}$ — the coefficient of determination from regressing logged validation loss on global pretraining step, pooled across the task's nine runs.

Definition 2 (Teachability): Task $t$ is teachable if its held-out loss responds to its own token budget across runs. This requires a cross-run comparison via a task budget sweep.

Definition 3 (Transfer): A family $f$ transfers at $w$ relative to $w'$ when increasing its share from the feasible mixture $w'$ to $w$ improves the downstream endpoint: $y(w) > y(w')$. Transfer is deliberately local because the simplex constrains every increase to displace another source.

Exposure Advantage: To compare diagnosticity and teachability explanations, the paper defines:

$$A_t = R^2_{t,\text{exposure}} - R^2_{t,\text{step}}$$

where the first term regresses logged validation loss on cumulative exposure. Seven tasks have $A_t > 0$, and five of the six largest values are curated tasks.

---

Methodology

Experimental Setup

Base corpus and controlled slice: Every pretraining run draws 70% of tokens from a fixed general Python corpus (CodeParrot) and 30% from a controlled slice whose composition is the only manipulated variable. A run is specified by a point $w = (w_O, w_C, w_L)$ on the 2-simplex $\Delta^2$:

$$P_r(x) = 0.70 P_{\text{code}}(x) + 0.30 \left[ w_{r,O} P_O(x) + w_{r,C} \sum_{t \in \mathcal{T}_C} w_{t|C} P_t(x) + w_{r,L} \sum_{t \in \mathcal{T}_L} w_{t|L} P_t(x) \right] \tag{1}$$

Within-source task weights sum to one and remain fixed across runs.

Source Families

Model and Training

• Architecture: Sparse-expert decoder-only model with top-2 expert routing, 50,304-token vocabulary, 12 layers, 12 attention heads, width 768, SwiGLU width 2,048
• Training: Context length 1,024, effective batch size 256, 262,144 tokens per optimizer step
• Optimizer: AdamW, learning rate $10^{-3}$, $\beta_1 = 0.9$, weight decay $10^{-4}$, 500 warmup steps
• Runs: 60k or 120k steps with decay horizon matched to run length
• Shadow sweep: 15 fixed mixtures along every pairwise edge in 25-percentage-point steps

Downstream Evaluation

Pretraining is followed by an identical fine-tuning stage (1,000 updates on refined OpenCodeInstruct, batch size 256, Adam with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-6}$, peak LR $10^{-5}$). HumanEval pass@k uses the unbiased estimator:

$$\widehat{\text{pass@}k} = \mathbb{E}_{\text{problems}} \left[ 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}} \right] \tag{2}$$

where $n = 20$ and $c$ is the number of correct samples. The Best over steps endpoint is the highest score among fine-tuning checkpoints 100, 200, ..., 1000.

Teachability Detection

For task $t$ in run $r$, first fit:

$$\log L_{t,r}(s) = \alpha_{t,r} + \beta_{t,r} \log s + \epsilon_{t,r,s} \tag{3}$$

over all readouts in the window, evaluating the fitted log-loss fall $\Delta_{t,r}$ between endpoints. Then fit:

$$\Delta_{t,r} = a_t + b_t \log_2 e_{t,r} + \varepsilon_{t,r} \tag{4}$$

The slope $b_t$ reports learning per doubling of exposure. A one-sided permutation test (2,000 shuffles, p < 0.05) declares detection.

---

Empirical Validation / Results

Teachability Results

Fourteen of 27 tasks clear the permutation test: 10/12 curated and 4/15 literature-derived. Benjamini–Hochberg adjustment leaves 8 detections (6/12 curated, 2/15 literature-derived).

Curated tasks emit a median of 10.2 supervised tokens per example versus 1.0 for literature probes. However, length is only weakly related to explained variance (Pearson r = 0.34, p = 0.08; Spearman ρ = 0.27, p = 0.18), so emitted length does not fully explain the family difference.

Mixture-Level Transfer: The Central Result

Table 1: HumanEval pass@20 along the curated–OpenCodeInstruct edge ($w_L = 0$)

Key findings:

• The highest observed point is interior to the edge at 75% OpenCodeInstruct / 25% curated, exceeding pure OpenCodeInstruct by 3.1 percentage points
• Pure curated data scores 3.6 points below pure OpenCodeInstruct and 0.6 points below the 16.5% zero-synthetic baseline
• The ranking is consistent at k = 5, 10, 20

Adaptive Scheduling Results

Table 2: 60k-step, four-expert mixture-profile and scheduler comparison

Critical observations:

• All three 60k free-ratio runs drive OpenCodeInstruct below 5% of the controlled slice within the first 5k steps, finishing at 1.2–1.4%
• Free-ratio Ado trails matched shadows by 2.4, 11.0, and 4.9 percentage points
• In 120k branches, the scheduler only begins moving mass back around step 90k — revealing temporal myopia in the 1,000-step fitting horizon

---

Theoretical and Practical Implications

Theoretical Implications

1. Diagnosticity ≠ Teachability ≠ Transfer. The paper provides a rigorous demonstration that these three signals, usually read off the same loss curve, are logically independent. A task can combine strong progress tracking with weak budget response, or vice versa.

2. Teachability does not order downstream transfer. The curated suite has the highest teachability rate (10/12) yet its pure-source endpoint (15.9%) falls below both pure OpenCodeInstruct and the zero-synthetic baseline. Teachability is insufficient as the sole mixture objective.

3. Loss-reducibility is a misleading objective for adaptive scheduling. Ado maximizes predicted near-term loss reduction, which scores OpenCodeInstruct low early in training despite its high downstream value. The scheduler's 1,000-step horizon creates temporal myopia — the aligned source only becomes attractive under its own criterion after ~88k steps.

4. Curated synthetic data has a bounded dose-response. Its value is real but non-monotone: +3.1 points at 25% share, but -0.6 points below baseline at 100% share. The bound is set by how much target-aligned data the curated source displaces.

Practical Implications

The paper's practical rule is narrow but concrete:

"When the downstream target is code generation, keep the target-aligned source dominant in the mixture, add curated synthetic data as a bounded share, and choose proportions with a target-referenced signal."

Standard probes (the literature suite) remain useful as instruments — controlled readouts of retrieval, state tracking, composition, and transduction — even when their own training budget shows weak evidence of marginal learning value.

---

Conclusion

The paper separates three signals usually conflated in loss-curve analysis. Under a controlled pretraining setup where only a 30% slice varies:

• 14 of 27 tasks respond detectably to their own exposure
• Family-level teachability ranks differently from downstream code transfer
• The best observed HumanEval pass@20 (22.6) came from a mixture with 75% target-aligned source and 25% curated synthetic data
• The same curated suite alone fell below the zero-synthetic baseline
• A loss-based adaptive scheduler drove the aligned source below 5% and finished below all three mixture-matched fixed controls

Future Directions

• Denser simplex grids to identify precise optima (the point (0.75, 0.25, 0) is the best observed, not proven optimal)
• Task-isolated interventions to decouple teachability estimates from mixture coupling
• Transfer-aware schedulers incorporating target-referenced marginal-value signals with multi-horizon evaluation
• Other architectures, scales, and base corpora to test generality
• Additional downstream targets beyond HumanEval (other code benchmarks, general reasoning)