Summary (Overview)
- Core Question: How should a fixed pretraining budget be divided between atomic facts (individual transitions) and reasoning traces (multi-step chains)?
- Key Finding: The optimal mixture depends systematically on reasoning-chain length () and reasoning-trace coverage (). Surprisingly, longer reasoning chains favor allocating MORE data to atomic facts, not less.
- Theoretical Result: For a linear-attention model, the authors derive an explicit formula for the optimal atomic weight: , where combines chain length and held-out fraction.
- Empirical Validation: Trends persist in full transformer layers, controlled multi-hop reasoning, and realistic mathematical pretraining (Olmo-3 150M on FineMath/MathLake).
- Practical Recommendation: Mixing reasoning data from the beginning of training is never worse than delaying it; interior mixtures outperform either pure-source extreme.
Introduction and Theoretical Foundation
Background and Motivation
Reasoning traces teach models how to compose facts across multiple steps, while atomic examples provide broader coverage of individual facts. Under a fixed training budget, these objectives compete:
- More reasoning data → better composition but less factual coverage
- More atomic data → broader coverage but less compositional practice
Recent work (Akter et al., 2026; Baek et al., 2026) has begun incorporating reasoning data into pretraining rather than post-training, but leaves open the fundamental question of optimal mixture allocation.
Theoretical Setup: State Tracking
The paper constructs a state-tracking world (Definition 1) that cleanly separates factual coverage from reasoning composition:
- Entities and relations act through fixed permutations
- A length- chain induces state sequence:
- Held set : a subset of facts excluded from reasoning traces (fraction )
- Reasoning coverage: is the fraction of facts available to reasoning traces
Key insight: Every fact is observed atomically, but only a subset appears in reasoning traces. This allows measuring generalization to transitions observed atomically but never used in training traces.
Learning Objective
The mixed loss combines atomic and reasoning supervision with atomic weight :
where is the cross-entropy loss at role .
Methodology
Model Architectures
1. Full Transformer Layer (Definition 5):
- Single causal self-attention head + position-wise MLP
- Token-content vectors: ,
- Merged query-key matrix parameterizes attention scores directly
- Trained with cross-entropy loss via Equation (1)
2. Scalar Linear-Attention Model (Definition 6):
- Removes attention normalization, uses scalar linear readout
- Score structure:
- Trained by gradient flow:
Key Assumptions
Assumption 1 (Balanced state-tracking world): entities and relations such that for every pair of entities , exactly one relation maps to :
This ensures uniformly sampled relations map any entity to a uniformly distributed next entity, with:
Evaluation Metric
Held-transition error measures whether facts learned atomically can be used in multi-step reasoning:
The optimal atomic weight is:
Empirical Validation / Results
Main Theoretical Result
Theorem 1 (Length–coverage trend): For all sufficiently large , with probability at least , the finite-time MSE-optimal atomic weight is unique in and satisfies:
where the leading-order optimal atomic weight is:
Key predictions:
- increases with reasoning length (longer chains → more atomic data)
- decreases with held fraction (broader coverage → more atomic data)
Synthetic State Tracking (Figure 1)
- World: 64 entities, 8 relations (random permutations)
- Interior mixtures outperform both pure-source extremes on OOD accuracy
- Optimal atomic ratio generally increases with and decreases with
- Minor deviations at , attributed to near-optimal plateaus
Controlled Multi-Hop Reasoning (Figure 2)
- Task: Follow relationships between 32 people to identify the final person
- Four-hop tasks favor larger atomic fractions than two- and three-hop tasks
- Withholding more pairs from reasoning lowers peak OOD accuracy and shifts near-best region toward smaller atomic fractions
- Consistent with Theorem 1
Mathematical Pretraining (FineMath–MathLake)
Table 1: Mixture-ratio sweep (Olmo-3 150M, 10B tokens)
| FM (%) | ML (%) | CE ↓ | PPL ↓ |
|---|---|---|---|
| 100 | 0 | 1.785 | 5.96 |
| 90 | 10 | 1.613 | 5.02 |
| 80 | 20 | 1.608 | 4.99 |
| 70 | 30 | 1.674 | 5.33 |
| 50 | 50 | 1.768 | 5.86 |
| 0 | 100 | 7.513 | 1831.40 |
- Both extremes are suboptimal: pure FineMath CE = 1.785, pure MathLake degrades to 7.513 (overfitting to repeated corpus)
- Optimal at 20% MathLake (CE = 1.608)
- Small reasoning share helps significantly; beyond 20-30%, CE rises monotonically
Quantity and Difficulty Effects (Figure 3)
- Held-out ratio: Optimal mixing ratio increases with pool availability (decreases with held-out fraction)
- Difficulty: Optimal mixing ratio increases with reasoning difficulty (harder problems → more atomic data)
Timing of Reasoning Data (Table 2)
| Pure FM steps (K) | ML share while mixing | CE ↓ | PPL ↓ |
|---|---|---|---|
| 0 | 20.0% | 1.744 | 5.72 |
| 2k | 25.0% | 1.754 | 5.78 |
| 4k | 33.3% | 1.771 | 5.88 |
| 8k | 100.0% | 2.411 | 11.14 |
Delaying reasoning data brings no benefit — GSM8K CE increases monotonically with delay. Recommendation: mix reasoning data from the beginning.
Theoretical and Practical Implications
Theoretical Implications
-
Novel mixture law: The explicit formula provides a principled characterization of how reasoning structure should shape data allocation.
-
Counterintuitive insight: Longer reasoning chains favor MORE atomic data, because each trace consumes more of the fixed budget, making broad factual coverage increasingly important.
-
Separation of concerns: The state-tracking framework cleanly separates factual coverage from compositional exposure, enabling precise analysis of how these interact.
Practical Implications
-
Pretraining recipe guidance: Interior mixtures of atomic and reasoning data outperform pure-source extremes; optimal ratios depend on reasoning complexity and coverage.
-
Data scheduling: Reasoning data should be introduced from the start of training — delaying provides no benefit.
-
Coverage monitoring: The fraction of facts covered by reasoning traces directly affects the optimal mixture — broader coverage requires more atomic data.
Conclusion
Main Takeaways
- The optimal allocation between atomic facts and reasoning traces depends systematically on reasoning-chain length () and reasoning-trace coverage ()
- Longer reasoning chains and broader coverage both favor larger atomic allocations
- Theoretical analysis (linear attention) explains these trends, validated empirically in full transformers and realistic mathematical pretraining
- Interior mixtures outperform extremes; reasoning data should be introduced early
Future Directions
- Extend to heterogeneous data sources and varying data quality in large-scale pretraining
- Study how different data mixtures affect subsequent SFT and RL performance
- Investigate whether the derived mixture law holds for other reasoning paradigms beyond state tracking
Limitations
- Controlled setting abstracts away real-world data heterogeneity and quality variation
- Focus on pretraining only; post-training interactions (SFT, RL) left for future work
Related papers
- The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search
A naive LLM critic can drive AutoML search below the no-critic random-search floor on 7 of 8 tabular datasets, with losses up to 0.245 AUC.
- Rethinking Fixed-Horizon Latent Reasoning
Adaptive latent inference that stops reasoning when progress stalls improves accuracy by up to 7.2 points while cutting computation by 29%, as completing fixed latent horizons can degrade performance.
- When Can Self-Play Fix a False Belief? Answer Keys, Pass Bands and Replacement Rules in an Execution-Grounded Loop
Self-play with execution-grounded keys cannot correct false beliefs because the generator's own code inherits its misreadings, and decorrelated keys are necessary but insufficient for correction.