Summary (Overview)

  • This paper investigates how pretraining data statistics causally shape language model learning dynamics through a case study on entity comparison tasks (e.g., "Between France and Brazil, which country is larger?").
  • The authors identify three distinct phases of learning: (1) a frequency heuristic where the model picks the more frequently seen entity, (2) a positional bias where the model picks the first- or last-mentioned entity, and (3) the correct algorithm based on true scalar values.
  • Controlled experiments with 163M-parameter GPT-2 models trained on mixtures of natural text and synthetic task data show that the emergence of these phases is causally determined by data properties: the amount of task-relevant synthetic data and the entity frequency distribution (Zipfian vs. uniform).
  • The same three-phase structure appears in the open-source OLMo-2 family (1B–32B parameters) trained on natural text alone, demonstrating that small-scale synthetic experiments can predict training dynamics at scale.
  • Flattening heavy-tailed Zipfian entity distributions (making them uniform) weakens the frequency heuristic phase and accelerates learning of the correct algorithm.

Introduction and Theoretical Foundation

The paper addresses a fundamental question in interpretability: how do statistical properties of pretraining data drive the strategies language models adopt during training?

The authors choose entity comparison as a testbed because it naturally admits multiple strategies:

  • The correct algorithm requires retrieving scalar attributes vd,ev_{d,e} for each entity and comparing them: A(d,e1,e2)=max⁡(vd,e1,vd,e2)\mathcal{A}(d, e_1, e_2) = \max(v_{d,e_1}, v_{d,e_2}).
  • Heuristics such as positional bias (Zheng et al., 2023) or frequency-based selection (Gottesman et al., 2025; Yang et al., 2025) provide shortcuts that may be easier to learn.

A key observation motivates the study: natural pretraining corpora exhibit Zipfian entity frequency distributions (Figure 2a), where a few entities (e.g., China, appearing 441M times) dominate while most are rare (e.g., Luxembourg, 6.8M times). Moreover, entity frequency correlates with scalar attributes (e.g., country area, log-log Pearson's r=0.62r = 0.62, Figure 2b), making the frequency heuristic a genuinely viable shortcut.

The theoretical framing builds on prior work showing that:

  • LMs rely on heuristics rather than intended algorithms (McCoy et al., 2019; Razeghi et al., 2022).
  • Factual recall emerges in distinct phases (Zucchet et al., 2025; Chang et al., 2024).
  • Data composition causally affects model behavior (Chan et al., 2022; Allen-Zhu and Li, 2024).

Methodology

Task Formulation

Entity comparison tasks are defined by triples (d,e1,e2)(d, e_1, e_2) where e1,e2e_1, e_2 are entities and dd is a numerical domain. The study covers nine comparison tasks across five entity types (countries, U.S. states, U.S. universities, movies, notable people), each with two directions (larger/smaller magnitude), yielding 18 task variants.

Controlled Pretraining Setup

  • Models: GPT-2 architecture (12 layers, 12 heads, 768-dim embeddings) with 163M parameters, trained from scratch.
  • Data: Mixtures of natural text from the OLMo-mix-1124 corpus and synthetic task data (e.g., "By total area, among the two countries A and B, the larger country is ___").
  • Budget: 1.1B tokens total, trained for 2 epochs with 3 random seeds.
  • Tokenization: allenai/dolma2-tokenizer, matching OLMo-2.

Feature Extraction

Three features are used to characterize model behavior at each checkpoint:

  1. Frequency feature: The log ratio of cumulative occurrences:
log⁡(countt(e1)countt(e2))\log \left( \frac{\text{count}_t(e_1)}{\text{count}_t(e_2)} \right)

where countt(e)\text{count}_t(e) counts entity mentions (aggregated across aliases) up to training step tt.

  1. Positional bias feature: Captured via the regression intercept, since each pair is evaluated in both orderings, isolating any systematic first/last-position preference.

  2. Value feature: The log ratio of true scalar values:

log⁡(vd,e1vd,e2)\log \left( \frac{v_{d,e_1}}{v_{d,e_2}} \right)

Regression Analysis

A logistic regression model predicts the LM's choice:

p^=σ(w⊤ϕ(d,e1,e2))\hat{p} = \sigma \left( \mathbf{w}^\top \phi(d, e_1, e_2) \right)

with L1 regularization and 5-fold cross-validation. The drop-one accuracy—the decrease in regression accuracy when a feature is removed—measures each feature's importance at each checkpoint. The dominant phase is the feature with the largest drop-one accuracy.

OLMo-2 Analysis

The same regression analysis is applied to OLMo-2 checkpoints (1B, 7B, 13B, 32B parameters) at 26 checkpoints across stage-1 pretraining, with 3 prompt templates per checkpoint.


Empirical Validation / Results

1. Controlled Experiments: Ratio of Synthetic Data

Key findings (Figure 3):

  • The frequency phase is invariant to the amount of synthetic task data—it peaks at step 200 and decays by step 1K in all conditions.
  • The correct algorithm emerges only above a critical density of task-specific data:
    • At 0% and 0.01% synthetic data: the value feature stays near zero; the model never learns the task.
    • At 0.1%: the correct algorithm emerges around 2.5K steps and rises steadily.
    • At 1%: it emerges within the first 1K steps and reaches a drop-one accuracy of 18 by end of training.
  • The position phase shrinks and disappears as synthetic data increases. Without sufficient task data, the model converges to a persistent positional bias.

2. Controlled Experiments: Entity Frequency Distribution

Key findings (Figure 4):

  • When natural text is Zipfian, a strong early frequency phase appears regardless of the synthetic distribution.
  • When natural text is uniform, the frequency phase is much weaker and nearly disappears when both natural and synthetic data are uniform.
  • The correct algorithm shows the opposite pattern: it emerges earliest and reaches its largest drop-one accuracy when both distributions are uniform.
  • The frequency bias persists but diminishes throughout training: accuracy is consistently higher when the correct entity is more frequent (Figure 5).

3. OLMo-2 Models

Across model scales (Figure 6):

  • All four OLMo-2 models (1B–32B) exhibit the frequency and position heuristics in the same order as the controlled experiments.
  • Only the 13B and 32B models learn the correct algorithm: at ~1T tokens for 13B and ~400B tokens for 32B.
  • The 1B and 7B models remain stuck in the positional heuristic phase, never learning the task—paralleling the critical-density threshold in synthetic experiments.

Across nine tasks (Figure 7):

  • At the earliest checkpoint (9B tokens), frequency is the dominant phase in 15 of 18 task variants.
  • Phases can be skipped for some tasks, but the relative ordering (frequency → position → value) is largely preserved whenever multiple phases appear.

Theoretical and Practical Implications

Theoretical Implications

  • The consistent ordering of phases across very different settings (163M to 32B parameters, synthetic to natural data) suggests a structural property of task learning: each phase corresponds to the easiest-to-learn predictive feature given the model's current state.
    • Frequency is directly available from token-level statistics.
    • Position depends on syntactic patterns.
    • Value-based comparison requires manipulating factual knowledge.
  • Each subsequent strategy is likely more expensive to learn but eventually becomes more loss-reducing than its predecessor.

Practical Implications

  • The long tail of natural pretraining corpora may impose a real cost on downstream task acquisition. Flattening heavy-tailed distributions can make learning more efficient.
  • Below a critical density of task-relevant signal (or critical model scale), models never escape heuristics—suggesting a threshold phenomenon for capability emergence.
  • The results demonstrate that small-scale synthetic experiments can predict training dynamics at much larger scales, providing a cost-effective methodology for studying data effects.

Conclusion

The paper identifies three phases of learning in entity comparison—frequency heuristic, positional bias, and correct algorithm—and demonstrates through controlled interventions that pretraining data composition causally determines which phases emerge and when. The same phase structure appears in OLMo-2 models up to 32B parameters trained on natural text alone, suggesting these dynamics are a general feature of how LMs learn this task.

Future directions include:

  • Understanding the underlying mechanisms for each heuristic (e.g., whether positional bias relies on induction-head-style retrieval).
  • Tracing the origins of the three-phase ordering.
  • Investigating whether latent representations track entity frequency.

Limitations acknowledged by the authors:

  • Results may not generalize beyond entity comparison tasks.
  • Controlled experiments are at 163M scale due to compute constraints.
  • Causal evidence at OLMo scale is not established.
  • Modifying entity distributions may introduce domain shifts and confounders.

Related papers