# Understanding the Role of Reasoning Data in Pretraining

> Optimal pretraining mixtures of atomic facts and reasoning traces favor more atomic data as reasoning chains lengthen, with interior mixtures outperforming pure extremes.

- **Source:** [arXiv](https://arxiv.org/abs/2611.19042)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/AWSyov
- **Whiteboard:** https://picx.dev/p/AWSyov/image

## Summary

## Summary (Overview)

- **Core Question**: How should a fixed pretraining budget be divided between atomic facts (individual transitions) and reasoning traces (multi-step chains)?
- **Key Finding**: The optimal mixture depends systematically on reasoning-chain length ($L$) and reasoning-trace coverage ($p$). Surprisingly, **longer reasoning chains favor allocating MORE data to atomic facts**, not less.
- **Theoretical Result**: For a linear-attention model, the authors derive an explicit formula for the optimal atomic weight: $\beta(z) = \frac{1}{2 - z + \sqrt{z^2 - z + 1}}$, where $z = L(1-q)$ combines chain length and held-out fraction.
- **Empirical Validation**: Trends persist in full transformer layers, controlled multi-hop reasoning, and realistic mathematical pretraining (Olmo-3 150M on FineMath/MathLake).
- **Practical Recommendation**: Mixing reasoning data from the beginning of training is never worse than delaying it; interior mixtures outperform either pure-source extreme.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Reasoning traces teach models **how to compose facts** across multiple steps, while atomic examples provide **broader coverage** of individual facts. Under a fixed training budget, these objectives compete:

- More reasoning data → better composition but less factual coverage
- More atomic data → broader coverage but less compositional practice

Recent work (Akter et al., 2026; Baek et al., 2026) has begun incorporating reasoning data into pretraining rather than post-training, but leaves open the fundamental question of optimal mixture allocation.

### Theoretical Setup: State Tracking

The paper constructs a **state-tracking world** (Definition 1) that cleanly separates factual coverage from reasoning composition:

- **Entities** $e \in [N]$ and **relations** $r \in [M]$ act through fixed permutations $\pi_r: [N] \to [N]$
- A length-$L$ chain $\chi = (r_1, \ldots, r_L, e_0)$ induces state sequence:
  $$e_{\ell} = \pi_{r_{\ell}}(e_{\ell-1}), \qquad \ell \in [L]$$
- **Held set** $\mathcal{H}$: a subset of $m$ facts excluded from reasoning traces (fraction $q = m/D$)
- **Reasoning coverage**: $p = 1 - q$ is the fraction of facts available to reasoning traces

**Key insight**: Every fact is observed atomically, but only a subset appears in reasoning traces. This allows measuring generalization to transitions observed atomically but **never used in training traces**.

### Learning Objective

The mixed loss combines atomic and reasoning supervision with atomic weight $\beta \in [0,1]$:

$$\mathcal{L}_{\beta}^{\text{CE}}(\theta) = \beta \, \mathbb{E}_{\chi \sim \mathcal{D}_{\text{at}}} \ell_{\text{CE},1}(\theta; \chi) + (1-\beta) \, \mathbb{E}_{\chi \sim \mathcal{D}_{\text{rea}}} \left[ \frac{1}{L} \sum_{\ell=1}^{L} \ell_{\text{CE},\ell}(\theta; \chi) \right]$$

where $\ell_{\text{CE},\ell}$ is the cross-entropy loss at role $\ell$.

---

## Methodology

### Model Architectures

**1. Full Transformer Layer** (Definition 5):
- Single causal self-attention head + position-wise MLP
- Token-content vectors: $c(r) = \binom{u_r}{0_N}$, $c(e) = \binom{0_M}{v_e}$
- Merged query-key matrix $\mathbf{A} \in \mathbb{R}^{d \times d}$ parameterizes attention scores directly
- Trained with cross-entropy loss via Equation (1)

**2. Scalar Linear-Attention Model** (Definition 6):
- Removes attention normalization, uses scalar linear readout
- Score structure: $(\mathbf{X}_t^{\top} \mathbf{B} \mathbf{X}_t) \odot \tilde{\mathbf{C}}_{1:t,1:t}$
- Trained by gradient flow: $\dot{\theta}_{\beta}(t) = -\nabla_{\theta} \mathcal{L}_{\beta}(\theta_{\beta}(t); \mathcal{S})$

### Key Assumptions

**Assumption 1 (Balanced state-tracking world)**: $N$ entities and $M = N$ relations such that for every pair of entities $e, e'$, exactly one relation maps $e$ to $e'$:
$$\sum_{r=1}^{N} \mathbf{1}\{\pi_r(e) = e'\} = 1$$

This ensures uniformly sampled relations map any entity to a uniformly distributed next entity, with:
$$\frac{1}{N} \sum_{r=1}^{N} \zeta_{\pi_r(e)} = 0, \quad \frac{1}{N} \sum_{r=1}^{N} \zeta_{\pi_r(e)}^2 = \frac{N^2 - 1}{3N^2}$$

### Evaluation Metric

**Held-transition error** measures whether facts learned atomically can be used in multi-step reasoning:
$$\mathcal{E}_H(\theta) := \frac{1}{2(L-1)} \sum_{\ell=2}^{L} \mathbb{E}_{\chi \sim \mathcal{D}_{H,\ell}} \left(f_{\ell}(\chi; \theta) - \zeta_{e_{\ell}}\right)^2$$

The **optimal atomic weight** is:
$$\beta_{\tau, S}^{\star} \in \arg\min_{\beta \in [0,1]} \mathcal{E}_H(\theta_{\beta}(\tau))$$

---

## Empirical Validation / Results

### Main Theoretical Result

**Theorem 1 (Length–coverage trend)**: For all sufficiently large $N$, with probability at least $1-\delta$, the finite-time MSE-optimal atomic weight $\beta_{\tau,S}^*$ is unique in $(1/3, 2/3)$ and satisfies:

$$\left| \beta_{\tau,S}^{*} - \beta(L(1-q)) \right| \leq \mathcal{R}_N, \quad \mathcal{R}_N = O_{L,p,\tau}\left(\frac{\sqrt{\log(2/\delta)}}{N}\right)$$

where the **leading-order optimal atomic weight** is:

$$\beta(z) = \frac{1}{2 - z + \sqrt{z^2 - z + 1}}, \qquad z > 0$$

**Key predictions**:
- $\beta$ **increases** with reasoning length $L$ (longer chains → more atomic data)
- $\beta$ **decreases** with held fraction $q$ (broader coverage → more atomic data)

### Synthetic State Tracking (Figure 1)

- World: 64 entities, 8 relations (random permutations)
- **Interior mixtures outperform both pure-source extremes** on OOD accuracy
- Optimal atomic ratio generally increases with $L$ and decreases with $q$
- Minor deviations at $L \in \{3,4\}$, $q = 30\%$ attributed to near-optimal plateaus

### Controlled Multi-Hop Reasoning (Figure 2)

- Task: Follow $H$ relationships between 32 people to identify the final person
- **Four-hop tasks favor larger atomic fractions** than two- and three-hop tasks
- Withholding more pairs from reasoning lowers peak OOD accuracy and shifts near-best region toward smaller atomic fractions
- Consistent with Theorem 1

### Mathematical Pretraining (FineMath–MathLake)

**Table 1: Mixture-ratio sweep (Olmo-3 150M, 10B tokens)**

| FM (%) | ML (%) | CE ↓ | PPL ↓ |
|--------|--------|------|-------|
| 100 | 0 | 1.785 | 5.96 |
| 90 | 10 | 1.613 | 5.02 |
| **80** | **20** | **1.608** | **4.99** |
| 70 | 30 | 1.674 | 5.33 |
| 50 | 50 | 1.768 | 5.86 |
| 0 | 100 | 7.513 | 1831.40 |

- **Both extremes are suboptimal**: pure FineMath CE = 1.785, pure MathLake degrades to 7.513 (overfitting to repeated corpus)
- **Optimal at 20% MathLake** (CE = 1.608)
- Small reasoning share helps significantly; beyond 20-30%, CE rises monotonically

### Quantity and Difficulty Effects (Figure 3)

- **Held-out ratio**: Optimal mixing ratio increases with pool availability $p$ (decreases with held-out fraction)
- **Difficulty**: Optimal mixing ratio increases with reasoning difficulty (harder problems → more atomic data)

### Timing of Reasoning Data (Table 2)

| Pure FM steps (K) | ML share while mixing | CE ↓ | PPL ↓ |
|-------------------|----------------------|------|-------|
| 0 | 20.0% | **1.744** | **5.72** |
| 2k | 25.0% | 1.754 | 5.78 |
| 4k | 33.3% | 1.771 | 5.88 |
| 8k | 100.0% | 2.411 | 11.14 |

**Delaying reasoning data brings no benefit** — GSM8K CE increases monotonically with delay. Recommendation: mix reasoning data from the beginning.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Novel mixture law**: The explicit formula $\beta(z) = \frac{1}{2 - z + \sqrt{z^2 - z + 1}}$ provides a principled characterization of how reasoning structure should shape data allocation.

2. **Counterintuitive insight**: Longer reasoning chains favor MORE atomic data, because each trace consumes more of the fixed budget, making broad factual coverage increasingly important.

3. **Separation of concerns**: The state-tracking framework cleanly separates factual coverage from compositional exposure, enabling precise analysis of how these interact.

### Practical Implications

1. **Pretraining recipe guidance**: Interior mixtures of atomic and reasoning data outperform pure-source extremes; optimal ratios depend on reasoning complexity and coverage.

2. **Data scheduling**: Reasoning data should be introduced from the start of training — delaying provides no benefit.

3. **Coverage monitoring**: The fraction of facts covered by reasoning traces directly affects the optimal mixture — broader coverage requires more atomic data.

---

## Conclusion

### Main Takeaways

- The optimal allocation between atomic facts and reasoning traces depends systematically on **reasoning-chain length** ($L$) and **reasoning-trace coverage** ($p$)
- Longer reasoning chains and broader coverage both favor **larger atomic allocations**
- Theoretical analysis (linear attention) explains these trends, validated empirically in full transformers and realistic mathematical pretraining
- Interior mixtures outperform extremes; reasoning data should be introduced early

### Future Directions

- Extend to heterogeneous data sources and varying data quality in large-scale pretraining
- Study how different data mixtures affect subsequent SFT and RL performance
- Investigate whether the derived mixture law holds for other reasoning paradigms beyond state tracking

### Limitations

- Controlled setting abstracts away real-world data heterogeneity and quality variation
- Focus on pretraining only; post-training interactions (SFT, RL) left for future work

---

_Markdown view of https://picx.dev/p/AWSyov, served by PicX — AI-generated visual whiteboard summaries of research papers._
