Summary (Overview)

  • Core Question: How should a fixed pretraining budget be divided between atomic facts (individual transitions) and reasoning traces (multi-step chains)?
  • Key Finding: The optimal mixture depends systematically on reasoning-chain length (LL) and reasoning-trace coverage (pp). Surprisingly, longer reasoning chains favor allocating MORE data to atomic facts, not less.
  • Theoretical Result: For a linear-attention model, the authors derive an explicit formula for the optimal atomic weight: β(z)=12−z+z2−z+1\beta(z) = \frac{1}{2 - z + \sqrt{z^2 - z + 1}}, where z=L(1−q)z = L(1-q) combines chain length and held-out fraction.
  • Empirical Validation: Trends persist in full transformer layers, controlled multi-hop reasoning, and realistic mathematical pretraining (Olmo-3 150M on FineMath/MathLake).
  • Practical Recommendation: Mixing reasoning data from the beginning of training is never worse than delaying it; interior mixtures outperform either pure-source extreme.

Introduction and Theoretical Foundation

Background and Motivation

Reasoning traces teach models how to compose facts across multiple steps, while atomic examples provide broader coverage of individual facts. Under a fixed training budget, these objectives compete:

  • More reasoning data → better composition but less factual coverage
  • More atomic data → broader coverage but less compositional practice

Recent work (Akter et al., 2026; Baek et al., 2026) has begun incorporating reasoning data into pretraining rather than post-training, but leaves open the fundamental question of optimal mixture allocation.

Theoretical Setup: State Tracking

The paper constructs a state-tracking world (Definition 1) that cleanly separates factual coverage from reasoning composition:

  • Entities e∈[N]e \in [N] and relations r∈[M]r \in [M] act through fixed permutations πr:[N]→[N]\pi_r: [N] \to [N]
  • A length-LL chain χ=(r1,…,rL,e0)\chi = (r_1, \ldots, r_L, e_0) induces state sequence: eℓ=πrℓ(eℓ−1),ℓ∈[L]e_{\ell} = \pi_{r_{\ell}}(e_{\ell-1}), \qquad \ell \in [L]
  • Held set H\mathcal{H}: a subset of mm facts excluded from reasoning traces (fraction q=m/Dq = m/D)
  • Reasoning coverage: p=1−qp = 1 - q is the fraction of facts available to reasoning traces

Key insight: Every fact is observed atomically, but only a subset appears in reasoning traces. This allows measuring generalization to transitions observed atomically but never used in training traces.

Learning Objective

The mixed loss combines atomic and reasoning supervision with atomic weight β∈[0,1]\beta \in [0,1]:

LβCE(θ)=β Eχ∼DatℓCE,1(θ;χ)+(1−β) Eχ∼Drea[1L∑ℓ=1LℓCE,ℓ(θ;χ)]\mathcal{L}_{\beta}^{\text{CE}}(\theta) = \beta \, \mathbb{E}_{\chi \sim \mathcal{D}_{\text{at}}} \ell_{\text{CE},1}(\theta; \chi) + (1-\beta) \, \mathbb{E}_{\chi \sim \mathcal{D}_{\text{rea}}} \left[ \frac{1}{L} \sum_{\ell=1}^{L} \ell_{\text{CE},\ell}(\theta; \chi) \right]

where ℓCE,ℓ\ell_{\text{CE},\ell} is the cross-entropy loss at role ℓ\ell.


Methodology

Model Architectures

1. Full Transformer Layer (Definition 5):

  • Single causal self-attention head + position-wise MLP
  • Token-content vectors: c(r)=(ur0N)c(r) = \binom{u_r}{0_N}, c(e)=(0Mve)c(e) = \binom{0_M}{v_e}
  • Merged query-key matrix A∈Rd×d\mathbf{A} \in \mathbb{R}^{d \times d} parameterizes attention scores directly
  • Trained with cross-entropy loss via Equation (1)

2. Scalar Linear-Attention Model (Definition 6):

  • Removes attention normalization, uses scalar linear readout
  • Score structure: (Xt⊤BXt)⊙C~1:t,1:t(\mathbf{X}_t^{\top} \mathbf{B} \mathbf{X}_t) \odot \tilde{\mathbf{C}}_{1:t,1:t}
  • Trained by gradient flow: θ˙β(t)=−∇θLβ(θβ(t);S)\dot{\theta}_{\beta}(t) = -\nabla_{\theta} \mathcal{L}_{\beta}(\theta_{\beta}(t); \mathcal{S})

Key Assumptions

Assumption 1 (Balanced state-tracking world): NN entities and M=NM = N relations such that for every pair of entities e,e′e, e', exactly one relation maps ee to e′e':

∑r=1N1{πr(e)=e′}=1\sum_{r=1}^{N} \mathbf{1}\{\pi_r(e) = e'\} = 1

This ensures uniformly sampled relations map any entity to a uniformly distributed next entity, with:

1N∑r=1Nζπr(e)=0,1N∑r=1Nζπr(e)2=N2−13N2\frac{1}{N} \sum_{r=1}^{N} \zeta_{\pi_r(e)} = 0, \quad \frac{1}{N} \sum_{r=1}^{N} \zeta_{\pi_r(e)}^2 = \frac{N^2 - 1}{3N^2}

Evaluation Metric

Held-transition error measures whether facts learned atomically can be used in multi-step reasoning:

EH(θ):=12(L−1)∑ℓ=2LEχ∼DH,ℓ(fℓ(χ;θ)−ζeℓ)2\mathcal{E}_H(\theta) := \frac{1}{2(L-1)} \sum_{\ell=2}^{L} \mathbb{E}_{\chi \sim \mathcal{D}_{H,\ell}} \left(f_{\ell}(\chi; \theta) - \zeta_{e_{\ell}}\right)^2

The optimal atomic weight is:

βτ,S⋆∈arg⁡min⁡β∈[0,1]EH(θβ(τ))\beta_{\tau, S}^{\star} \in \arg\min_{\beta \in [0,1]} \mathcal{E}_H(\theta_{\beta}(\tau))

Empirical Validation / Results

Main Theoretical Result

Theorem 1 (Length–coverage trend): For all sufficiently large NN, with probability at least 1−δ1-\delta, the finite-time MSE-optimal atomic weight βτ,S∗\beta_{\tau,S}^* is unique in (1/3,2/3)(1/3, 2/3) and satisfies:

∣βτ,S∗−β(L(1−q))∣≤RN,RN=OL,p,τ(log⁡(2/δ)N)\left| \beta_{\tau,S}^{*} - \beta(L(1-q)) \right| \leq \mathcal{R}_N, \quad \mathcal{R}_N = O_{L,p,\tau}\left(\frac{\sqrt{\log(2/\delta)}}{N}\right)

where the leading-order optimal atomic weight is:

β(z)=12−z+z2−z+1,z>0\beta(z) = \frac{1}{2 - z + \sqrt{z^2 - z + 1}}, \qquad z > 0

Key predictions:

  • β\beta increases with reasoning length LL (longer chains → more atomic data)
  • β\beta decreases with held fraction qq (broader coverage → more atomic data)

Synthetic State Tracking (Figure 1)

  • World: 64 entities, 8 relations (random permutations)
  • Interior mixtures outperform both pure-source extremes on OOD accuracy
  • Optimal atomic ratio generally increases with LL and decreases with qq
  • Minor deviations at L∈{3,4}L \in \{3,4\}, q=30%q = 30\% attributed to near-optimal plateaus

Controlled Multi-Hop Reasoning (Figure 2)

  • Task: Follow HH relationships between 32 people to identify the final person
  • Four-hop tasks favor larger atomic fractions than two- and three-hop tasks
  • Withholding more pairs from reasoning lowers peak OOD accuracy and shifts near-best region toward smaller atomic fractions
  • Consistent with Theorem 1

Mathematical Pretraining (FineMath–MathLake)

Table 1: Mixture-ratio sweep (Olmo-3 150M, 10B tokens)

FM (%)ML (%)CE ↓PPL ↓
10001.7855.96
90101.6135.02
80201.6084.99
70301.6745.33
50501.7685.86
01007.5131831.40
  • Both extremes are suboptimal: pure FineMath CE = 1.785, pure MathLake degrades to 7.513 (overfitting to repeated corpus)
  • Optimal at 20% MathLake (CE = 1.608)
  • Small reasoning share helps significantly; beyond 20-30%, CE rises monotonically

Quantity and Difficulty Effects (Figure 3)

  • Held-out ratio: Optimal mixing ratio increases with pool availability pp (decreases with held-out fraction)
  • Difficulty: Optimal mixing ratio increases with reasoning difficulty (harder problems → more atomic data)

Timing of Reasoning Data (Table 2)

Pure FM steps (K)ML share while mixingCE ↓PPL ↓
020.0%1.7445.72
2k25.0%1.7545.78
4k33.3%1.7715.88
8k100.0%2.41111.14

Delaying reasoning data brings no benefit — GSM8K CE increases monotonically with delay. Recommendation: mix reasoning data from the beginning.


Theoretical and Practical Implications

Theoretical Implications

  1. Novel mixture law: The explicit formula β(z)=12−z+z2−z+1\beta(z) = \frac{1}{2 - z + \sqrt{z^2 - z + 1}} provides a principled characterization of how reasoning structure should shape data allocation.

  2. Counterintuitive insight: Longer reasoning chains favor MORE atomic data, because each trace consumes more of the fixed budget, making broad factual coverage increasingly important.

  3. Separation of concerns: The state-tracking framework cleanly separates factual coverage from compositional exposure, enabling precise analysis of how these interact.

Practical Implications

  1. Pretraining recipe guidance: Interior mixtures of atomic and reasoning data outperform pure-source extremes; optimal ratios depend on reasoning complexity and coverage.

  2. Data scheduling: Reasoning data should be introduced from the start of training — delaying provides no benefit.

  3. Coverage monitoring: The fraction of facts covered by reasoning traces directly affects the optimal mixture — broader coverage requires more atomic data.


Conclusion

Main Takeaways

  • The optimal allocation between atomic facts and reasoning traces depends systematically on reasoning-chain length (LL) and reasoning-trace coverage (pp)
  • Longer reasoning chains and broader coverage both favor larger atomic allocations
  • Theoretical analysis (linear attention) explains these trends, validated empirically in full transformers and realistic mathematical pretraining
  • Interior mixtures outperform extremes; reasoning data should be introduced early

Future Directions

  • Extend to heterogeneous data sources and varying data quality in large-scale pretraining
  • Study how different data mixtures affect subsequent SFT and RL performance
  • Investigate whether the derived mixture law holds for other reasoning paradigms beyond state tracking

Limitations

  • Controlled setting abstracts away real-world data heterogeneity and quality variation
  • Focus on pretraining only; post-training interactions (SFT, RL) left for future work

Related papers