# Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models

> Pretraining data statistics causally determine three sequential learning phases—frequency heuristic, positional bias, then correct algorithm—with critical data density thresholds governing whether models ever escape heuristics.

- **Source:** [arXiv](https://arxiv.org/abs/2611.16882)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/drq5nQ
- **Whiteboard:** https://picx.dev/p/drq5nQ/image

## Summary

## Summary (Overview)

- This paper investigates how pretraining data statistics causally shape language model learning dynamics through a case study on entity comparison tasks (e.g., *"Between France and Brazil, which country is larger?"*).
- The authors identify **three distinct phases of learning**: (1) a **frequency heuristic** where the model picks the more frequently seen entity, (2) a **positional bias** where the model picks the first- or last-mentioned entity, and (3) the **correct algorithm** based on true scalar values.
- Controlled experiments with 163M-parameter GPT-2 models trained on mixtures of natural text and synthetic task data show that the emergence of these phases is **causally determined by data properties**: the amount of task-relevant synthetic data and the entity frequency distribution (Zipfian vs. uniform).
- The same three-phase structure appears in the open-source **OLMo-2 family (1B–32B parameters)** trained on natural text alone, demonstrating that small-scale synthetic experiments can predict training dynamics at scale.
- Flattening heavy-tailed Zipfian entity distributions (making them uniform) weakens the frequency heuristic phase and accelerates learning of the correct algorithm.

---

## Introduction and Theoretical Foundation

The paper addresses a fundamental question in interpretability: **how do statistical properties of pretraining data drive the strategies language models adopt during training?**

The authors choose **entity comparison** as a testbed because it naturally admits multiple strategies:

- The **correct algorithm** requires retrieving scalar attributes $v_{d,e}$ for each entity and comparing them: $\mathcal{A}(d, e_1, e_2) = \max(v_{d,e_1}, v_{d,e_2})$.
- **Heuristics** such as positional bias (Zheng et al., 2023) or frequency-based selection (Gottesman et al., 2025; Yang et al., 2025) provide shortcuts that may be easier to learn.

A key observation motivates the study: natural pretraining corpora exhibit **Zipfian entity frequency distributions** (Figure 2a), where a few entities (e.g., China, appearing 441M times) dominate while most are rare (e.g., Luxembourg, 6.8M times). Moreover, entity frequency correlates with scalar attributes (e.g., country area, log-log Pearson's $r = 0.62$, Figure 2b), making the frequency heuristic a genuinely viable shortcut.

The theoretical framing builds on prior work showing that:
- LMs rely on heuristics rather than intended algorithms (McCoy et al., 2019; Razeghi et al., 2022).
- Factual recall emerges in distinct phases (Zucchet et al., 2025; Chang et al., 2024).
- Data composition causally affects model behavior (Chan et al., 2022; Allen-Zhu and Li, 2024).

---

## Methodology

### Task Formulation
Entity comparison tasks are defined by triples $(d, e_1, e_2)$ where $e_1, e_2$ are entities and $d$ is a numerical domain. The study covers **nine comparison tasks across five entity types** (countries, U.S. states, U.S. universities, movies, notable people), each with two directions (larger/smaller magnitude), yielding 18 task variants.

### Controlled Pretraining Setup
- **Models**: GPT-2 architecture (12 layers, 12 heads, 768-dim embeddings) with 163M parameters, trained from scratch.
- **Data**: Mixtures of natural text from the OLMo-mix-1124 corpus and synthetic task data (e.g., *"By total area, among the two countries A and B, the larger country is ___"*).
- **Budget**: 1.1B tokens total, trained for 2 epochs with 3 random seeds.
- **Tokenization**: allenai/dolma2-tokenizer, matching OLMo-2.

### Feature Extraction
Three features are used to characterize model behavior at each checkpoint:

1. **Frequency feature**: The log ratio of cumulative occurrences:
$$\log \left( \frac{\text{count}_t(e_1)}{\text{count}_t(e_2)} \right)$$
where $\text{count}_t(e)$ counts entity mentions (aggregated across aliases) up to training step $t$.

2. **Positional bias feature**: Captured via the regression intercept, since each pair is evaluated in both orderings, isolating any systematic first/last-position preference.

3. **Value feature**: The log ratio of true scalar values:
$$\log \left( \frac{v_{d,e_1}}{v_{d,e_2}} \right)$$

### Regression Analysis
A logistic regression model predicts the LM's choice:
$$\hat{p} = \sigma \left( \mathbf{w}^\top \phi(d, e_1, e_2) \right)$$

with L1 regularization and 5-fold cross-validation. The **drop-one accuracy**—the decrease in regression accuracy when a feature is removed—measures each feature's importance at each checkpoint. The **dominant phase** is the feature with the largest drop-one accuracy.

### OLMo-2 Analysis
The same regression analysis is applied to OLMo-2 checkpoints (1B, 7B, 13B, 32B parameters) at 26 checkpoints across stage-1 pretraining, with 3 prompt templates per checkpoint.

---

## Empirical Validation / Results

### 1. Controlled Experiments: Ratio of Synthetic Data

**Key findings (Figure 3):**
- The **frequency phase is invariant** to the amount of synthetic task data—it peaks at step 200 and decays by step 1K in all conditions.
- The **correct algorithm emerges only above a critical density** of task-specific data:
  - At 0% and 0.01% synthetic data: the value feature stays near zero; the model never learns the task.
  - At 0.1%: the correct algorithm emerges around 2.5K steps and rises steadily.
  - At 1%: it emerges within the first 1K steps and reaches a drop-one accuracy of 18 by end of training.
- The **position phase shrinks and disappears** as synthetic data increases. Without sufficient task data, the model converges to a persistent positional bias.

### 2. Controlled Experiments: Entity Frequency Distribution

**Key findings (Figure 4):**
- When **natural text is Zipfian**, a strong early frequency phase appears regardless of the synthetic distribution.
- When **natural text is uniform**, the frequency phase is much weaker and nearly disappears when both natural and synthetic data are uniform.
- The **correct algorithm shows the opposite pattern**: it emerges earliest and reaches its largest drop-one accuracy when both distributions are uniform.
- The frequency bias **persists but diminishes** throughout training: accuracy is consistently higher when the correct entity is more frequent (Figure 5).

### 3. OLMo-2 Models

**Across model scales (Figure 6):**
- All four OLMo-2 models (1B–32B) exhibit the frequency and position heuristics **in the same order** as the controlled experiments.
- Only the **13B and 32B models learn the correct algorithm**: at ~1T tokens for 13B and ~400B tokens for 32B.
- The 1B and 7B models remain stuck in the positional heuristic phase, never learning the task—paralleling the critical-density threshold in synthetic experiments.

**Across nine tasks (Figure 7):**
- At the earliest checkpoint (9B tokens), **frequency is the dominant phase in 15 of 18 task variants**.
- Phases can be skipped for some tasks, but **the relative ordering (frequency → position → value) is largely preserved** whenever multiple phases appear.

---

## Theoretical and Practical Implications

### Theoretical Implications
- The consistent ordering of phases across very different settings (163M to 32B parameters, synthetic to natural data) suggests a **structural property of task learning**: each phase corresponds to the easiest-to-learn predictive feature given the model's current state.
  - Frequency is directly available from token-level statistics.
  - Position depends on syntactic patterns.
  - Value-based comparison requires manipulating factual knowledge.
- Each subsequent strategy is likely **more expensive to learn** but eventually becomes more loss-reducing than its predecessor.

### Practical Implications
- The **long tail of natural pretraining corpora may impose a real cost** on downstream task acquisition. Flattening heavy-tailed distributions can make learning more efficient.
- Below a **critical density of task-relevant signal** (or critical model scale), models never escape heuristics—suggesting a threshold phenomenon for capability emergence.
- The results demonstrate that **small-scale synthetic experiments can predict training dynamics at much larger scales**, providing a cost-effective methodology for studying data effects.

---

## Conclusion

The paper identifies three phases of learning in entity comparison—**frequency heuristic, positional bias, and correct algorithm**—and demonstrates through controlled interventions that pretraining data composition causally determines which phases emerge and when. The same phase structure appears in OLMo-2 models up to 32B parameters trained on natural text alone, suggesting these dynamics are a general feature of how LMs learn this task.

**Future directions** include:
- Understanding the underlying mechanisms for each heuristic (e.g., whether positional bias relies on induction-head-style retrieval).
- Tracing the origins of the three-phase ordering.
- Investigating whether latent representations track entity frequency.

**Limitations** acknowledged by the authors:
- Results may not generalize beyond entity comparison tasks.
- Controlled experiments are at 163M scale due to compute constraints.
- Causal evidence at OLMo scale is not established.
- Modifying entity distributions may introduce domain shifts and confounders.

---

_Markdown view of https://picx.dev/p/drq5nQ, served by PicX — AI-generated visual whiteboard summaries of research papers._
