# Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining

> Optimal repetition count for scarce data decreases with model size, so smaller-model sweeps must prune candidates rather than pick a single count for larger-scale training.

- **Source:** [arXiv](https://arxiv.org/abs/2610.05126)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/DJWooP
- **Whiteboard:** https://picx.dev/p/DJWooP/image

## Summary

## Summary (Overview)

- **Core finding**: The optimal repetition count for a scarce target corpus in mixture pretraining changes with model size. A count preferred by a small model does not necessarily remain best at a larger scale.
- **Key evidence**: In Proof-Pile-2 experiments, reducing repetition from 16 to 8 at 520M parameters improved loss by ~0.5 nat while *halving* the training token budget (2B to 1B tokens). This was confirmed in two independent six-seed experiments with all 12 paired differences favoring the lower count.
- **Method**: A candidate-pruning rule uses loss curves from smaller models (35M, 50M, 80M) to retain a short list of 3 promising repetition counts for evaluation at larger scales. On PubMed and Caselaw corpora, these retained sets always contained the lowest-loss count at both 200M and 520M scales.
- **Theoretical framework**: The paper proposes an empirical two-effect model where a decreasing loss term and an increasing repetition-penalty term compete, with the balance shifting as model size grows. A first-order expansion in log model size yields an affine regression form that motivates the candidate-selection procedure.
- **Empirical validation**: The simple point-prediction model, fitted only on smaller models (35M–80M), selects repetition counts with excess losses of only 0.003–0.008 nats above the measured minima on refined 12-count grids at 200M.

## Introduction and Theoretical Foundation

### Background and Motivation

The paper addresses a critical practical question in large language model (LLM) pretraining: **when valuable target data is scarce, how many times should it be repeated?** This choice involves a tension:
- Repeating target tokens improves learning from that data
- But repeated exposure may cause overfitting or diminishing returns
- Each repetition also lengthens training and adds generic tokens (since the target fraction is fixed)

The key insight is that **repetition preferences are model-size-dependent**. A repetition count that works well for a small model may be suboptimal at larger scales, because the relative balance between learning from repetition and the penalty of over-exposure shifts with model capacity.

### Theoretical Foundation

The paper builds on several lines of prior work:
- **Scaling laws** (Kaplan et al., 2020; Hoffmann et al., 2022) relate loss to model size and training tokens
- **Repetition-aware scaling laws** (Muennighoff et al., 2025; Sedova et al., 2026; Lovelace et al., 2026) account for finite data and repeated exposures
- **Data selection methods** use smaller proxy models to guide larger runs (Liu et al., 2025; Magnusson et al., 2025)

The central theoretical contribution is the **two-effect model**:

$$L_N(r) = A(N) + N^\eta U(r) + N^{\eta+q} V(r), \quad U'(r) < 0, \; V'(r) > 0, \; q > 0$$

where:
- $L_N(r)$ is the held-out target loss at model size $N$ with repetition count $r$
- $A(N)$ is a common offset independent of repetition
- $N^\eta U(r)$ is a decreasing term (benefit of repetition and generic data)
- $N^{\eta+q} V(r)$ is an increasing term (penalty from over-repetition) that grows relatively faster with scale

The model predicts that for any pair of counts $a < b$, the ordering can reverse **at most once** as $N$ increases, since:

$$G_{b \to a}(N) = N^\eta (DN^q - C)$$

where $C = U(a) - U(b) > 0$ and $D = V(b) - V(a) > 0$.

### Key Definitions

- **Target pool**: $S_t = 100$M tokens of scarce data
- **Target fraction**: $h = 0.8$ of training tokens
- **Total training tokens**: $D_{\text{total}}(r) = rS_t/h = 125M \cdot r$
- **Observed minimum**: $r^{\text{obs}}_N \in \arg\min_{r \in R_N} L_N(r)$
- **Excess loss**: $E_N(r) = L_N(r) - \min_{u \in R_N} L_N(u)$

## Methodology

### Experimental Design

Each repetition count $r$ is trained **independently to completion** with its own cosine learning-rate schedule over its prescribed token count. This differs from checkpoint-based approaches. The losses are endpoints of separate runs, not intermediate states of one longer run.

**Model sizes tested**: 35M, 50M, 80M (smaller models); 200M, 520M (target models). All are decoder-only Transformers with varying layer counts and widths, using AdamW optimizer with learning rate $3 \times 10^{-4}$, bf16 training, and a global batch of 524,288 tokens.

**Corpora**:
- Wikipedia-derived (initial)
- Proof-Pile-2 (initial + 520M confirmation)
- PubMed and Caselaw (main evaluation corpora)

### The Candidate Selection Rule

The candidate-pruning rule (Section 5.2) uses the first-order expansion of the two-effect model. With $x = \log(N/N_0)$:

$$E_N(r) = C_r e^{\eta x} + D_r e^{(\eta+q)x} = c_r + d_r x + O(x^2)$$

This motivates a **separate linear regression for each count** of excess loss against log model size. The full rule:
1. Resample whole seed curves (27 ordered resamples per size)
2. Average them and subtract each curve's minimum
3. Regress excess loss on $\log N$ for each count
4. Predict at target size, clip negative predictions to zero
5. Average across resamples, retain the **3 lowest-scoring counts**

The selected sets were: PubMed $\{4, 8, 12\}$, Caselaw $\{8, 12, 16\}$.

### Point Prediction Model

A simpler specialization sets $\eta = 0$, $q = 1$:

$$\hat{L}_N(r) = A_N + B_{\text{dec}} r^{-1/2} + P_{\text{inc}} \left(\frac{N}{N_{50}}\right) r$$

where $A_N$ are per-size offsets, $B_{\text{dec}}$, $P_{\text{inc}}$ are shared coefficients fit by least squares to smaller-model means, and $N_{50}$ is the 50M model size.

### Structural Model for Loss Curves

For completed curves, a richer model is fit:

$$w(n) = \frac{n^q}{1 + (n/n_s)^q}, \quad F_N(r) = n^\eta [Bx^{-\alpha} + P w(n) x^\nu]$$

$$\hat{L}_N(r) = A_N + \frac{K F_N(r)}{K + F_N(r)}$$

with $n = N/(50M)$, $x = r/8$. This yields the unique model minimum:

$$r^*(N) = 8 \left(\frac{\alpha B}{\nu P}(n^{-q} + n_s^{-q})\right)^{1/(\alpha + \nu)}$$

## Empirical Validation / Results

### Ranking Reversals

- **Wikipedia-derived**: At 35M, $r=36$ beats $r=24$; at 80M, the ordering reverses with 24 lower by 0.139 nat
- **Proof-Pile-2**: At 50M, $r=16$ beats $r=8$; at 80M, 8 is best (all 3 paired differences favor 8)
- **520M confirmation**: Two independent 6-seed experiments show loss reductions of $0.524 \pm 0.007$ and $0.509 \pm 0.005$ nats when switching from 16 to 8 repetitions. All 12 paired differences favor 8.

The decomposition of the Proof-Pile-2 reversal (50M → 80M, 16 → 8): $\delta(16) - \delta(8) = 0.110$, $m_s = 0.007$, giving $G = 0.104$ nats.

### Candidate Retention Results

| Corpus | 520M Retained Set | Best Count | Loss vs. 80M Winner Reuse |
|--------|------------------|------------|---------------------------|
| PubMed | {4, 8, 12} | 8 | 0.854 nat lower |
| Caselaw | {8, 12, 16} | 12 | 2.492 nat lower |

At 200M, frozen sets {8, 12, 16} (PubMed) and {12, 16, 24} (Caselaw) both contained count 12, which had the lowest loss in all seeds. On the refined 12-count grid, the sets incurred 0.006 and 0 nats excess loss respectively.

### Point Predictions at 200M

The simple model (Eq. 8), fitted only on 35M/50M/80M data, made predications frozen before 200M training:

| | PubMed | Caselaw |
|---|---|---|
| Continuous minimum | 9.659 | 11.431 |
| Original grid choice | r = 8 (0.038 nat excess) | r = 12 (0 excess) |
| Integer choice on 12-count grid | r = 10 (0.003 nat excess) | r = 11 (0.008 nat excess) |

### Model Comparison on Held-out Counts

**Table 2: Matched fits to seven counts at 35M/50M/80M/200M/520M**

| Model | New-count RMSE (PubMed) | New-count RMSE (Caselaw) | Selection loss (PubMed) | Selection loss (Caselaw) |
|-------|-------------------------|--------------------------|------------------------|--------------------------|
| Adapted Lovelace | 0.238 | 0.224 | 0.003 | 0.008 |
| Same, free η | 0.159 | 0.184 | 0.003 | 0.008 |
| Saturated extension (Eq. 9) | 0.108 | 0.074 | 0.006 | 0.018 |

### Cost Reduction

Evaluating only the retained counts at 520M (plus smaller-model scans) would reduce compute by **65.6%** compared to running all seven counts at 520M, while achieving the same lowest loss.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Scale-dependent repetition preferences**: The paper provides systematic evidence that the optimal repetition count decreases with model size. The two-effect model explains this via the relative growth of the increasing penalty term ($q > 0$ was confirmed across all four corpora).

2. **One-time reversal guarantee**: For any pair of repetition counts, the ordering can reverse at most once as model size increases, provided the two-effect model holds.

3. **Local affine approximation**: The first-order expansion in $\log N$ provides a theoretical justification for linear regression-based candidate selection.

4. **Comparison with prior scaling laws**: The adapted Lovelace form (Eq. 19) is a special case ($\eta = 0$) of the proposed model. Freeing $\eta$ to allow both terms to vary in overall amplitude improves held-out loss prediction by 33% (PubMed) and 18% (Caselaw).

### Practical Implications

1. **Candidate pruning is more reliable than point prediction**: Retained sets consistently contained the best count across all validation scenarios (262 retrospective checks at 520M and 80M grids), while point predictions were sometimes off by 2+ counts with substantial loss penalties.

2. **Cost-effective validation**: The pruning rule reduces training cost by ~65% while achieving equivalent final loss, compared to exhaustive sweeps.

3. **Practical guidance**: When more unique target data is unavailable, the empirical models can estimate:
   - A promising repetition count
   - The loss associated with nearby choices
   - Low-loss intervals for refinement

## Conclusion

The paper demonstrates that repetition count selection cannot be naively transferred across model scales in data-constrained pretraining. Key takeaways:

1. **Ranking reversals are real and consequential**: Observed reversals ranged from 0.104 to 0.524 nats in loss penalty when reusing smaller-model choices.

2. **Candidate pruning works**: The simple rule of selecting 3 counts from smaller-model curves, validated via bootstrapping and local regression in $\log N$, reliably retains the best count at larger scales.

3. **Point prediction is possible but riskier**: The simple additive model made good predictions at 200M (0–0.008 nat excess), but extrapolating to 520M from 35M/50M sometimes selected substantially suboptimal counts (1.435 nat penalty on PubMed).

4. **The decreasing-then-increasing loss shape is consistent**: All corpora show a minimum in the repetition valley, with the optimum shifting downward as model size increases.

**Future directions**: The paper notes limitations including:
- Fixed target-pool size and mixture fraction (h = 0.8)
- Single model family and training setup
- Remaining amplitude errors in shared cross-size fits
- The need to test broader data budgets, mixtures, and architectures

The practical recommendation is to use smaller-model curves for **candidate retention** (robust) rather than exact point prediction, and to use local loss estimates for guided refinement around the identified valley.

---

_Markdown view of https://picx.dev/p/DJWooP, served by PicX — AI-generated visual whiteboard summaries of research papers._