Summary of "Understanding Reasoning from Pretraining to Post-Training"

Summary (Overview)

• Controlled chess testbed: The authors develop a chess-based framework that mirrors the standard LLM training pipeline (pretraining → SFT → RL), enabling systematic study of how pretraining choices affect RL post-training outcomes.
• Joint pretraining–RL scaling law: They establish that pretraining validation loss strongly predicts post-RL performance (pass@1), while the slope of RL improvement grows approximately linearly with log pretraining tokens, enabling compute-optimal allocation predictions across pretraining and RL.
• Mechanistic policy analysis: RL does not uniformly sharpen the SFT policy—it amplifies correct moves on easy puzzles but surfaces previously absent correct moves and reinforces incorrect ones on hard puzzles, explaining why RL improves pass@1 without consistently improving pass@k.
• Transfer beyond chess: The same predictive pattern (pretraining loss → post-RL performance, pretraining tokens → RL slope) holds for a 1B parameter language model trained on math-domain text (10B–200B tokens).

Introduction and Theoretical Foundation

The paper addresses a fundamental tension in LLM training: how to allocate compute between pretraining (building a strong base model from human text) and RL post-training (learning from environmental interaction with verifiable rewards). While scaling laws exist for pretraining alone (Kaplan et al., 2020; Hoffmann et al., 2022) and RL scaling has been studied separately (Khatri et al., 2025), no quantitative framework connects the two stages.

Two competing views motivate the work:

1. RL mostly sharpens existing capabilities (Yue et al., 2025)—suggesting more compute should go to pretraining.
2. RL can genuinely discover new behaviors (Yuan et al., 2025; Sun et al., 2025)—suggesting more compute should go to RL.

Studying these questions in natural language is prohibitive due to:

• Massive, uncontrolled pretraining corpora
• Prohibitively expensive compute sweeps at frontier scale
• Opaque token-level actions and lack of step-level supervision

Chess provides an ideal testbed: a compact action space, exact move verification via engines, plentiful controllable human data (filterable by Elo), and small models that reach nontrivial performance, making systematic compute sweeps affordable.

Methodology

Chess Representation

Each chess game is serialized as alternating player moves, with each move encoded as 4 tokens: $\langle piece\rangle, \langle source\rangle, \langle destination\rangle, \langle flag\rangle$, giving a vocabulary of size $|V| = 81$.

Three-Stage Pipeline

1. Pretraining: Autoregressive next-token prediction on 54B tokens of Lichess human games (Blitz and Rapid, 2022). Models trained at 10 scales: {5M, 10M, 20M, 32M, 50M, 100M, 200M, 410M, 680M, 1B} parameters, with 11 pretraining compute budgets from $6.5 \times 10^{16}$ to $6.5 \times 10^{19}$ FLOPs.

2. Supervised Fine-Tuning (SFT): Training on synthetic reasoning traces followed by ground-truth solution moves. Reasoning traces are constructed by sampling $K$ continuations from a proposal policy, merging them into a prefix tree, and serializing in depth-first order:

$$r = \langle T \rangle \tilde{\tau}_1 \langle sep \rangle \tilde{\tau}_2 \langle sep \rangle \cdots \tilde{\tau}_m \langle /T \rangle$$

3. Reinforcement Learning: GRPO optimization on 156K chess puzzles with binary outcome reward:

$$R(\zeta, s_0) = \mathbf{1}[a_1 = a_1^{\star}, \ldots, a_H = a_H^{\star}]$$

Scaling Law Analysis

For RL scaling, the authors use a first-order Taylor expansion of the sigmoid RL scaling law in the non-saturated regime:

$$R_{N,T}(C) = R_{N,T}^{\text{ref}} + B_{N,T}(\log_{10} C - \log_{10} C_{\text{ref}}) \tag{1}$$

where $B_{N,T}$ is the local slope (performance gain per decade of RL compute) and $R_{N,T}^{\text{ref}}$ is the fitted reward at reference compute level $C_{\text{ref}}$.

Empirical Validation / Results

Key Scaling Law Findings

The joint pretraining–RL scaling law takes the form:

$$R(C_{RL}, N, T) = f(L_{pt}(N, T)) + g(N, T)\left(\log_{10} C_{RL} - \log_{10} C_{\text{ref}}\right)$$

with fitted parameterizations:

$$f(L) = 0.0314 + \exp(4.87 - 12.85L)$$ $$g(N,T) = -0.216 + 0.0172\log_{10}T + 0.0098\log_{10}N$$ $$L(N,T) = 0.414 + 95.4N^{-0.404} + 1.62 \times 10^{5}T^{-0.677}$$

Key results:

• Post-RL performance at reference compute is strongly predicted by pretraining loss (Spearman correlation $|\rho|$ increases from 0.93 to 0.99 as $\log_{10} C_{\text{ref}}$ increases from 16 to 20)
• Slope $B_{N,T}$ correlates positively with pretraining tokens (Pearson $r = +0.84$)
• The joint model using both $\log_{10} T$ and $\log_{10} N$ achieves the lowest RMSE and highest $R^2$
• Pretraining tokens receive a larger coefficient than model size in predicting RL slope

Compute-Optimal Frontier

The optimal RL fraction increases with total compute, while pretraining token allocation does not significantly deviate from Chinchilla scaling.

Mechanism Analysis

Policy change categorization (with $k=3$, $\epsilon_{tail} = 0.05$):

• Ground-truth amplification: Correct move already in top-k, further reinforced
• Tail discovery: Correct move promoted from probability below $\epsilon_{tail}$ into top-k
• Wrong-mode amplification: Correct move remains outside top-k while wrong preferred move is reinforced

Difficulty-stratified results (B1 = easiest, B5 = hardest):

RL improves pass@1 but not pass@16 consistently, explained by this mixed redistribution of probability mass.

Math Domain Transfer

For a 1B OLMo-2 model pretrained on 10B–200B tokens of math text (70% Nemotron-CC-Math-v1, 30% Dolma3), SFT on NuminaMath-CoT, then RL on GSM8K/MATH/DeepScaler problems:

• Lower pretraining loss consistently predicts higher post-RL performance
• Slope $B_T$ increases nearly linearly with $\log_{10} T$
• Pattern mirrors chess results qualitatively

Theoretical and Practical Implications

1. Compute allocation guidance: The results suggest RL is strongly initialization-dependent—starting RL from weakly pretrained checkpoints gives limited gains. However, as total compute grows, RL should take an increasingly large share. This provides a quantitative framework for deciding when to switch from pretraining to RL.

2. Understanding RL's mechanism: The finding that RL both amplifies existing correct moves and surfaces buried ones (while sometimes reinforcing errors) explains the pass@1 vs. pass@k discrepancy and suggests that improving RL beyond pass@1 requires methods targeting wrong-mode amplification reduction.

3. Unified scaling framework: The joint law connects Chinchilla-style pretraining scaling with RL scaling, enabling evaluation of hypothetical recipes $(N, T, C_{RL})$ without training them—a practical tool for compute allocation decisions.

4. Beyond chess: The transfer to math suggests the findings may generalize to natural language reasoning, providing a foundation for studying pretraining-to-post-training interactions in more complex domains.

Conclusion

The paper provides the first quantitative account of the pretraining-to-RL interface using a controlled chess testbed. Key takeaways:

1. Pretraining loss predicts post-RL performance level, while pretraining data scale predicts the rate of RL improvement
2. RL's effect is heterogeneous: it amplifies correct moves on easy puzzles but surfaces buried moves and reinforces wrong ones on hard puzzles
3. Optimal compute allocation shifts toward RL as total budget grows, but RL remains initialization-dependent

Future directions include:

• Studying optimal switching points from pretraining to RL
• Developing methods to reduce wrong-mode amplification
• Exploring interleaved pretraining/RL strategies
• Using the testbed for synthetic data design, self-play, transcendence, and weak-to-strong generalization studies