Summary (Overview)

• Core finding: The optimal repetition count for a scarce target corpus in mixture pretraining changes with model size. A count preferred by a small model does not necessarily remain best at a larger scale.
• Key evidence: In Proof-Pile-2 experiments, reducing repetition from 16 to 8 at 520M parameters improved loss by ~0.5 nat while halving the training token budget (2B to 1B tokens). This was confirmed in two independent six-seed experiments with all 12 paired differences favoring the lower count.
• Method: A candidate-pruning rule uses loss curves from smaller models (35M, 50M, 80M) to retain a short list of 3 promising repetition counts for evaluation at larger scales. On PubMed and Caselaw corpora, these retained sets always contained the lowest-loss count at both 200M and 520M scales.
• Theoretical framework: The paper proposes an empirical two-effect model where a decreasing loss term and an increasing repetition-penalty term compete, with the balance shifting as model size grows. A first-order expansion in log model size yields an affine regression form that motivates the candidate-selection procedure.
• Empirical validation: The simple point-prediction model, fitted only on smaller models (35M–80M), selects repetition counts with excess losses of only 0.003–0.008 nats above the measured minima on refined 12-count grids at 200M.

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a critical practical question in large language model (LLM) pretraining: when valuable target data is scarce, how many times should it be repeated? This choice involves a tension:

• Repeating target tokens improves learning from that data
• But repeated exposure may cause overfitting or diminishing returns
• Each repetition also lengthens training and adds generic tokens (since the target fraction is fixed)

The key insight is that repetition preferences are model-size-dependent. A repetition count that works well for a small model may be suboptimal at larger scales, because the relative balance between learning from repetition and the penalty of over-exposure shifts with model capacity.

Theoretical Foundation

The paper builds on several lines of prior work:

• Scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) relate loss to model size and training tokens
• Repetition-aware scaling laws (Muennighoff et al., 2025; Sedova et al., 2026; Lovelace et al., 2026) account for finite data and repeated exposures
• Data selection methods use smaller proxy models to guide larger runs (Liu et al., 2025; Magnusson et al., 2025)

The central theoretical contribution is the two-effect model:

$$L_N(r) = A(N) + N^\eta U(r) + N^{\eta+q} V(r), \quad U'(r) < 0, \; V'(r) > 0, \; q > 0$$

where:

• $L_N(r)$ is the held-out target loss at model size $N$ with repetition count $r$
• $A(N)$ is a common offset independent of repetition
• $N^\eta U(r)$ is a decreasing term (benefit of repetition and generic data)
• $N^{\eta+q} V(r)$ is an increasing term (penalty from over-repetition) that grows relatively faster with scale

The model predicts that for any pair of counts $a < b$, the ordering can reverse at most once as $N$ increases, since:

$$G_{b \to a}(N) = N^\eta (DN^q - C)$$

where $C = U(a) - U(b) > 0$ and $D = V(b) - V(a) > 0$.

Key Definitions

• Target pool: $S_t = 100$M tokens of scarce data
• Target fraction: $h = 0.8$ of training tokens
• Total training tokens: $D_{\text{total}}(r) = rS_t/h = 125M \cdot r$
• Observed minimum: $r^{\text{obs}}_N \in \arg\min_{r \in R_N} L_N(r)$
• Excess loss: $E_N(r) = L_N(r) - \min_{u \in R_N} L_N(u)$

Methodology

Experimental Design

Each repetition count $r$ is trained independently to completion with its own cosine learning-rate schedule over its prescribed token count. This differs from checkpoint-based approaches. The losses are endpoints of separate runs, not intermediate states of one longer run.

Model sizes tested: 35M, 50M, 80M (smaller models); 200M, 520M (target models). All are decoder-only Transformers with varying layer counts and widths, using AdamW optimizer with learning rate $3 \times 10^{-4}$, bf16 training, and a global batch of 524,288 tokens.

Corpora:

• Wikipedia-derived (initial)
• Proof-Pile-2 (initial + 520M confirmation)
• PubMed and Caselaw (main evaluation corpora)

The Candidate Selection Rule

The candidate-pruning rule (Section 5.2) uses the first-order expansion of the two-effect model. With $x = \log(N/N_0)$:

$$E_N(r) = C_r e^{\eta x} + D_r e^{(\eta+q)x} = c_r + d_r x + O(x^2)$$

This motivates a separate linear regression for each count of excess loss against log model size. The full rule:

1. Resample whole seed curves (27 ordered resamples per size)
2. Average them and subtract each curve's minimum
3. Regress excess loss on $\log N$ for each count
4. Predict at target size, clip negative predictions to zero
5. Average across resamples, retain the 3 lowest-scoring counts

The selected sets were: PubMed $\{4, 8, 12\}$, Caselaw $\{8, 12, 16\}$.

Point Prediction Model

A simpler specialization sets $\eta = 0$, $q = 1$:

$$\hat{L}_N(r) = A_N + B_{\text{dec}} r^{-1/2} + P_{\text{inc}} \left(\frac{N}{N_{50}}\right) r$$

where $A_N$ are per-size offsets, $B_{\text{dec}}$, $P_{\text{inc}}$ are shared coefficients fit by least squares to smaller-model means, and $N_{50}$ is the 50M model size.

Structural Model for Loss Curves

For completed curves, a richer model is fit:

$$w(n) = \frac{n^q}{1 + (n/n_s)^q}, \quad F_N(r) = n^\eta [Bx^{-\alpha} + P w(n) x^\nu]$$ $$\hat{L}_N(r) = A_N + \frac{K F_N(r)}{K + F_N(r)}$$

with $n = N/(50M)$, $x = r/8$. This yields the unique model minimum:

$$r^*(N) = 8 \left(\frac{\alpha B}{\nu P}(n^{-q} + n_s^{-q})\right)^{1/(\alpha + \nu)}$$

Empirical Validation / Results

Ranking Reversals

• Wikipedia-derived: At 35M, $r=36$ beats $r=24$; at 80M, the ordering reverses with 24 lower by 0.139 nat
• Proof-Pile-2: At 50M, $r=16$ beats $r=8$; at 80M, 8 is best (all 3 paired differences favor 8)
• 520M confirmation: Two independent 6-seed experiments show loss reductions of $0.524 \pm 0.007$ and $0.509 \pm 0.005$ nats when switching from 16 to 8 repetitions. All 12 paired differences favor 8.

The decomposition of the Proof-Pile-2 reversal (50M → 80M, 16 → 8): $\delta(16) - \delta(8) = 0.110$, $m_s = 0.007$, giving $G = 0.104$ nats.

Candidate Retention Results

At 200M, frozen sets {8, 12, 16} (PubMed) and {12, 16, 24} (Caselaw) both contained count 12, which had the lowest loss in all seeds. On the refined 12-count grid, the sets incurred 0.006 and 0 nats excess loss respectively.

Point Predictions at 200M

The simple model (Eq. 8), fitted only on 35M/50M/80M data, made predications frozen before 200M training:

Model Comparison on Held-out Counts

Table 2: Matched fits to seven counts at 35M/50M/80M/200M/520M

Cost Reduction

Evaluating only the retained counts at 520M (plus smaller-model scans) would reduce compute by 65.6% compared to running all seven counts at 520M, while achieving the same lowest loss.

Theoretical and Practical Implications

Theoretical Contributions

1. Scale-dependent repetition preferences: The paper provides systematic evidence that the optimal repetition count decreases with model size. The two-effect model explains this via the relative growth of the increasing penalty term ($q > 0$ was confirmed across all four corpora).

2. One-time reversal guarantee: For any pair of repetition counts, the ordering can reverse at most once as model size increases, provided the two-effect model holds.

3. Local affine approximation: The first-order expansion in $\log N$ provides a theoretical justification for linear regression-based candidate selection.

4. Comparison with prior scaling laws: The adapted Lovelace form (Eq. 19) is a special case ($\eta = 0$) of the proposed model. Freeing $\eta$ to allow both terms to vary in overall amplitude improves held-out loss prediction by 33% (PubMed) and 18% (Caselaw).

Practical Implications

1. Candidate pruning is more reliable than point prediction: Retained sets consistently contained the best count across all validation scenarios (262 retrospective checks at 520M and 80M grids), while point predictions were sometimes off by 2+ counts with substantial loss penalties.

2. Cost-effective validation: The pruning rule reduces training cost by ~65% while achieving equivalent final loss, compared to exhaustive sweeps.

3. Practical guidance: When more unique target data is unavailable, the empirical models can estimate:

   • A promising repetition count
   • The loss associated with nearby choices
   • Low-loss intervals for refinement

Conclusion

The paper demonstrates that repetition count selection cannot be naively transferred across model scales in data-constrained pretraining. Key takeaways:

1. Ranking reversals are real and consequential: Observed reversals ranged from 0.104 to 0.524 nats in loss penalty when reusing smaller-model choices.

2. Candidate pruning works: The simple rule of selecting 3 counts from smaller-model curves, validated via bootstrapping and local regression in $\log N$, reliably retains the best count at larger scales.

3. Point prediction is possible but riskier: The simple additive model made good predictions at 200M (0–0.008 nat excess), but extrapolating to 520M from 35M/50M sometimes selected substantially suboptimal counts (1.435 nat penalty on PubMed).

4. The decreasing-then-increasing loss shape is consistent: All corpora show a minimum in the repetition valley, with the optimum shifting downward as model size increases.

Future directions: The paper notes limitations including:

• Fixed target-pool size and mixture fraction (h = 0.8)
• Single model family and training setup
• Remaining amplitude errors in shared cross-size fits
• The need to test broader data budgets, mixtures, and architectures

The practical recommendation is to use smaller-model curves for candidate retention (robust) rather than exact point prediction, and to use local loss estimates for guided refinement around the identified valley.