# Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs

> Small batches act as implicit regularizers in repeated-data pretraining, yielding a U-shaped loss curve with optima far below standard recipes, plus three recipe tweaks that further improve performance.

- **Source:** [arXiv](https://arxiv.org/abs/2610.15965)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/SmdqdF
- **Whiteboard:** https://picx.dev/p/SmdqdF/image

## Summary

## Summary (Overview)

- **Key finding**: This paper revisits repeated-data pretraining (training on finite data multiple times) and shows that batch size is a critical, previously underexplored factor. Jointly optimizing batch size with learning rate, weight decay, and epoch count reveals a **U-shaped relationship** between validation loss and batch size, with the optimum at smaller batches than conventional pretraining.

- **Mechanism**: Small-batch gradient noise provides implicit regularization against repeated-data overfitting. The U-curve reflects a balance between optimization (favoring more steps) and generalization (favoring stochastic regularization).

- **Recipe improvements**: Three simple training modifications—REPACK (repacking documents into new sequences each epoch), tied input/output embeddings, and positive embedding weight decay—substantially improve attainable loss and extend beneficial repetitions (from 20 to 32 epochs for 1.5B model; from 112 to 240 for 153M model).

- **Resource decomposition**: The paper separates GPU availability from training time as distinct resources, identifying four compute regimes. Under fixed deadlines, larger batches are preferred for short budgets, but smaller batches with fewer GPUs can achieve lower loss even under the same time constraint—challenging the common practice of maximizing data parallelism.

- **Practical guidance**: Batch size must account for repeated-data generalization; training recipes must be adapted for repeated exposure; and compute allocation must distinguish GPUs from time.

---

## Introduction and Theoretical Foundation

### Background

Most language-model pretraining assumes a **single pass** over a large, diverse web corpus, with compute as the binding resource and data treated as effectively unlimited (Kaplan et al., 2020; Hoffmann et al., 2022). However, the supply of high-quality human-written data is finite and grows at a physically limited rate (Villalobos et al., 2024), so continued scaling increasingly requires training repeatedly on the same data (Muennighoff et al., 2023).

### Prior Work

- **Kim et al. (2026)**: Fix the unique data, remove the compute constraint, and show that stronger regularization (especially weight decay tuning) substantially delays overfitting. They select a small batch using a single-pass ablation and hold it fixed.
- **Xu et al. (2026)**: Build on this work, proposing additional regularization methods and an interaction-aware scaling law, also using a fixed batch throughout.

### Key Theoretical Insight

The paper identifies that batch size controls both statistical behavior and available parallelism. In single-pass training, batch size optimization balances statistical efficiency and parallelism, with a **critical batch size** below which jointly increasing batch size and learning rate preserves token efficiency (McCandlish et al., 2018; Kaplan et al., 2020). However, for repeated-data training, the authors hypothesize that **smaller batches provide useful implicit regularization** against overfitting, fundamentally changing the optimal tradeoff.

---

## Methodology

### Experimental Setup

- **Models**: Dense decoder-only Transformers with 153M, 474M, and 1.5B parameters, using the olmo-core codebase.
- **Data**: DCLM-Baseline corpus with 4,096-token context and document-disjoint splits. Three unique-data pools: 0.333B, 1B, and 3B tokens. The measured grid covers $D_u/N$ from 0.22 to 19.61.
- **Optimizer**: AdamW with warmup-stable-decay (WSD) learning-rate schedule, with a 100M-token warmup and 10% terminal decay.
- **Batch sizes**: Global batches of 16–512 sequences for the 1.5B model and 32–512 for smaller models.

### Hyperparameter Tuning Methodology

For each $(N, D_u, B)$ combination, the authors tune peak learning rate and weight decay through **gradual adaptive coordinate search**:

> "We first select the pair for an earlier epoch, then proceed to the next epoch by starting from the pair selected for the previous epoch and including neighboring learning rates $(1/2 \times$ and 2×) and a weight-decay value one level higher (3×)."

### Key Recipe Variations Tested

| Method | Description |
|--------|-------------|
| **FIXED** | Keeping packed sequences, minibatch composition, and order fixed across epochs (as in Kim et al., 2026) |
| **SHUFFLE** | Keeping sequence boundaries fixed while reshuffling sequences across minibatches |
| **REPACK** | Reshuffling documents and dynamically repacking them into fixed-length sequences each epoch |

### Evaluation

Primary metric is **next-token cross-entropy (CE)** on a fixed 100M-token validation split (larger than the 4M tokens in Kim et al. (2026) or 10M in Xu et al. (2026)).

---

## Empirical Validation / Results

### 1. Batch Size Has an Interior Optimum

**Figure 2** shows that batch-size tuning has a substantial effect. Relative to the best $B=512$ run:

| Model | Best B=512 CE | Best B=64 CE | Improvement |
|-------|---------------|--------------|-------------|
| 1.5B | 3.124 | 2.977 | 0.147 |
| 474M | 3.136 | 3.093 | 0.043 |
| 153M | 3.416 | 3.332 | 0.084 |

For all three model sizes, the complete sweep is **U-shaped**: validation CE is minimized near $B=64$, with both larger and smaller batches performing worse. This is substantially smaller than typical public pretraining recipes (GPT-2, Cerebras-GPT, Pythia, OLMo use global batches of 512–2,048 sequences).

### 2. Comparison with Single-Pass Batch Size Predictions

The paper compares against single-pass scaling laws:

$$B_{opt} = 0.2920 \dot{C}_F^{0.3271} \quad \text{(Bi et al., 2024)}$$

$$B_{opt} = 6.42 \times 10^3 C_F^{0.102} \quad \text{(Shuai et al., 2024)}$$

**Table 1**: Batch-size comparison for the 1.5B model on the 1B-token pool ($C_F \approx 6NED_u$):

| Method | $C_F$ (equiv. $E$) | Batch size (million tokens) |
|--------|-------------------|----------------------------|
| Bi et al. | $0.90 \times 10^{19}$ ($E=1$) | 0.463 |
| Bi et al. | $7.20 \times 10^{19}$ ($E=8$) | 0.914 |
| Bi et al. | $18.0 \times 10^{19}$ ($E=20$) | 1.233 |
| Shuai et al. | $0.90 \times 10^{19}$ ($E=1$) | 0.551 |
| Shuai et al. | $7.20 \times 10^{19}$ ($E=8$) | 0.681 |
| Shuai et al. | $18.0 \times 10^{19}$ ($E=20$) | 0.747 |
| **Ours** | $\infty$ ($C_F$: $E=20$) | **0.262** |

The single-pass laws predict batches 2.9–4.7× larger than the repeated-data optimum.

### 3. Data Loader Ablations

**Figure 4(a)** — Best validation CE by loader:

| Loader | B = 64 | B = 512 |
|--------|--------|---------|
| FIXED | 3.011 (E16) | 3.205 (E8) |
| SHUFFLE | 2.977 (E20) | 3.124 (E8) |
| REPACK | **2.971 (E24)** | **3.064 (E24)** |

REPACK improves on SHUFFLE at both batch sizes and reaches its best validation loss later in training.

### 4. Weight Tying and Embedding Decay

**Figure 5(a)** — With REPACK dataloader:

| Method | B = 64 | B = 512 |
|--------|--------|---------|
| Untied | 2.971 (E24) | 3.064 (E24) |
| Tied, $WD_{Emb}=0$ | 2.971 (E24) | 3.061 (E20) |
| Untied, $WD_{Emb}>0$ | 2.951 (E32) | 3.050 (E28) |
| **Tied, $WD_{Emb}>0$** | **2.946 (E32)** | **3.030 (E28)** |

Weight tying alone yields little improvement, but combining it with positive embedding decay gives the best results.

### 5. Complete Repetition-Tuned Recipe Results

The repetition-aware recipe (REPACK + tied embeddings + positive embedding decay) improves validation loss and extends the validation-selected training horizon:

| Model | Original Epoch | Repetition-tuned Epoch | Batch Optimum Shift |
|-------|----------------|----------------------|---------------------|
| 1.5B | 20 | 32 | 64 → 64 |
| 474M | 32 | 80 | 64 → 128 |
| 153M | 112 | 240 | 64 → 128 |

### 6. Time-Constrained Scaling

**Figure 8** shows that under short deadlines, larger batches are preferred (processing more data per optimizer step). As the deadline increases, progressively smaller batches become preferable until reaching the unconstrained optimum $B^\star$.

**Key finding (Figure 1(c))**: Under a short deadline (8K steps), a larger batch using 64 GPUs achieves the best loss. As the deadline increases to 32K steps, a small batch using only **16 GPUs** achieves lower loss despite using strictly fewer GPUs.

### 7. Extrapolated 70B Example

For a hypothetical 70B model on 10T unique tokens, the fitted optimum gives:

$$E^\star \approx 214.2, \quad B^\star \approx 1{,}536$$

$$G^\star \approx 6{,}144 \text{ H100 GPUs}, \quad T^\star \approx 22.6 \text{ years}$$

This motivates separating GPU and time constraints, as the infinite-compute recipe may be practically infeasible.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Repeated-data training changes the batch-size tradeoff**: Small-batch gradient noise, which can be undesirable in single-pass training, becomes a valuable implicit regularizer against repeated-data overfitting. The U-curve reflects a balance between optimization (more steps) and generalization (stochastic regularization).

2. **Recipe choices matter more than previously recognized**: Data presentation across epochs (FIXED vs. SHUFFLE vs. REPACK), weight tying, and embedding decay are not neutral choices—they materially affect the attainable frontier when data are repeated.

3. **Four compute regimes**: Separating GPU availability from training time yields a framework for understanding when different strategies are optimal:
   - Infinite GPUs + infinite time → use infinite-compute recipe directly
   - Finite GPUs + infinite time → gradient accumulation trades parallelism for sequential execution
   - Infinite GPUs + finite time → larger batches or latency-oriented parallelism
   - Finite GPUs + finite time → joint optimization of all choices

### Practical Implications

1. **Batch size guidance**: For repeated-data training, batch sizes substantially smaller than typical pretraining recipes (e.g., $B=64$ vs. $B=512$+) can yield significant validation loss improvements.

2. **Recipe recommendations**: Use REPACK (reshuffling documents each epoch), tie input/output embeddings, and apply positive weight decay to the shared embeddings.

3. **Resource allocation**: Maximizing GPU utilization through larger batches under a fixed deadline can be suboptimal. Even with the same time budget, smaller batches using fewer GPUs can achieve lower validation loss.

4. **Open-source training stacks**: The survey of nine representative open-source stacks shows inconsistent combinations of these ingredients, suggesting that many existing recipes are leaving performance on the table for data-constrained settings.

---

## Conclusion

This work demonstrates that when unique training data become the binding constraint, both pretraining recipes and compute allocation must change:

1. **Batch size** must account for repeated-data generalization—favoring smaller batches than conventional pretraining.
2. **Training recipes** must be adapted to repeated exposure—repacking documents, weight tying, and embedding regularization improve attainable loss.
3. **Compute allocation** must distinguish GPUs from time—maximizing GPU utilization can be suboptimal.

### Limitations

- Observations are limited to 153M–1.5B-parameter models and 0.333B–3B-token data pools; the 70B/10T example requires significant extrapolation.
- Findings may depend on the specific architecture (dense models), optimizer (AdamW with WSD), context length, and data (DCLM-Baseline).
- Extending results to mixture-of-experts (MoE) models is especially likely to affect conclusions and remains an important future direction.

---

_Markdown view of https://picx.dev/p/SmdqdF, served by PicX — AI-generated visual whiteboard summaries of research papers._