Summary of "When Data Is Scarce: Scaling Sparse Language Models with Repeated Training"

Summary (Overview)

• This paper presents the first systematic study of Dynamic Sparse Training (DST) in data-constrained regimes, where limited unique tokens require multi-epoch training.
• The authors introduce a novel sparsity-aware scaling law that jointly models active parameters, unique tokens, data repetition, and sparsity, achieving validation $R^2 = 0.914$ on unseen configurations.
• Key finding: sparsity delays data saturation — the data saturation scale $R_d^*(S)$ rises from 4.4 at $S=0$ to about 6.90 at $S=0.5$, making multi-epoch training more effective for sparse models.
• Resource trade-offs: Loss-optimal sparsity remains moderate ($S \approx 0.45$–$0.5$), while compute-optimal sparsity increases with data scale ($S \approx 0.6$–$0.75$).
• Compute savings: At compute-optimal sparsity, sparse models match dense-model accuracy while using 8×–10× fewer training FLOPs, with up to 3× smaller parameter-to-token ratios.

Introduction and Theoretical Foundation

Background

The paper addresses a critical challenge in LLM pre-training: the growing mismatch between data availability and computation. While high-quality human data grows at ~1.03× per year, pretraining data usage grows at ~4× per year, leading to a data-constrained pre-training regime.

Two Orthogonal Scaling Perspectives

The authors identify two largely independent lines of scaling law research:

1. Sparse scaling laws (Frantar et al., 2024; Jin et al., 2025): Extend dense formulations by accounting for active parameters during training but assume data-sufficient regimes.

2. Data-constrained scaling laws (Muennighoff et al., 2023): Model diminishing returns from repeated data via effective dataset size but assume dense optimization.

Key Insight: Why Sparsity and Data Reuse Interact

The central motivation is that sparse training alters not just model capacity but how parameters are optimized:

• Dense training repeatedly updates the same parameters on limited data → accelerates saturation
• Sparse optimization distributes updates across different subsets of parameters → potentially changes how models benefit from repeated data

Theoretical Foundation: Chinchilla Scaling Law

The starting point is the dense scaling law:

$$L(N, D) = \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} + E, \tag{1}$$

with compute modeled as $C \approx 6ND$.

The data-constrained extension introduces effective dataset size:

$$D' = U_d + U_d R_d^* \left(1 - e^{-R_d/R_d^*}\right), \tag{2}$$

where $U_d$ = unique tokens, $R_d$ = repetition ratio, $R_d^*$ = saturation rate.

Methodology

Experimental Setup

• Models: LLaMA-2 family trained on C4 dataset
• Scale: Up to 1.92B dense-equivalent parameters; validation on held-out models up to 7.68B
• Data budgets: 520M, 1.3B, and 2.6B unique tokens
• Epochs: 1, 2, 4, 8, 16 (total tokens up to 41.6B)
• Active parameter sizes: 15M to 960M (2× increments)
• Sparsity levels: {0%, 50%, 75%, 87.5%, 93.75%}
• Total runs: 500 training runs
• Training details: Batch size 512, Adam optimizer, bfloat16, $\mu$P learning rate scaling, magnitude pruning with random regrowth

Key Empirical Observations

Observation 1 (Sparsity–capacity trade-off): For fixed non-zero parameters $N$, increasing sparsity $S$ initially improves loss but saturates/diminishes at extreme levels.

Modeling implication — modify the parameter-scaling term:

$$\frac{A}{N^{\alpha}} \longrightarrow \frac{A\left((1-S)^{\epsilon} + PS^{\mu}\right)}{N^{\alpha}} \tag{3}$$

Observation 2 (Sparsity delays data saturation): Sparsity delays repeated-data saturation, with largest gains at moderate sparsity.

Modeling implication — make saturation threshold sparsity-dependent:

$$R_d^*(S) = R_d^*\left(1 + \lambda_1 S + \sigma_1 S^2\right) \tag{7}$$

Final Proposed Scaling Law

The complete sparse data-constrained scaling law:

$$L(N', D', S) = \frac{A\left((1-S)^{\epsilon} + PS^{\mu}\right)}{N'^{\alpha}} + \frac{B}{D'^{\beta}} + E \tag{9}$$

with effective sizes:

$$N' = U_n + U_n R_n^*(S)\left(1 - e^{-R_n/R_n^*(S)}\right) \tag{10}$$ $$D' = U_d + U_d R_d^*(S)\left(1 - e^{-R_d/R_d^*(S)}\right) \tag{11}$$

and sparsity-dependent saturation:

$$R_d^*(S) = R_d^*\left(1 + \lambda_1 S + \sigma_1 S^2\right) \tag{12}$$ $$R_n^*(S) = R_n^*\left(1 + \lambda_2 S + \sigma_2 S^2\right) \tag{13}$$

All coefficients fitted via L-BFGS optimization.

Empirical Validation / Results

Scaling Law Fit Quality

Fitted Coefficient Insights

• $P < 0$: Sparsity reduces the effective parameter-scaling penalty
• $\mu = 0.847 < 1$: Benefits grow sublinearly (diminishing marginal returns)
• $\epsilon \approx 0$ (slightly negative): Weak penalty near extreme sparsity
• $\sigma_1 < 0$, $\sigma_2 < 0$: Both saturation functions are concave in sparsity

Key Saturation Dynamics

• $R_d^*(S)$ peaks at $S \approx 0.66$, rising from 4.4 ($S=0$) to ~6.9 ($S=0.5$)
• $R_n^*(S)$ peaks earlier at $S \approx 0.34$, then decreases sharply

Resource Allocation Results

Iso-Loss (fixed data = 1.3B tokens):

• Dense: frontier shifts toward larger models, fewer epochs
• Sparse (50%): frontier favors more epochs, fewer parameters
• Parameter-to-token ratio: dense ≈ 0.19 vs. sparse ≈ 0.064 (~3× reduction)

Iso-FLOPs: Moderate sparsity achieves best balance between effective capacity and data reuse under both sparse ($C = 6ND$) and dense FLOPs cost models.

Optimal Sparsity Analysis

Theoretical and Practical Implications

Theoretical Contributions

1. Unified framework: First scaling law to jointly model sparsity, data constraints, and repetition — bridging two previously disjoint research lines.

2. Mechanistic understanding: Reveals that sparsity operates through two distinct channels:

   • Parameter efficiency: $(1-S)^{\epsilon} + PS^{\mu}$ factor improves capacity utilization
   • Data efficiency: Increased $R_d^*(S)$ delays saturation of repeated data

3. Optimal allocation shift: Sparsity changes the compute-optimal trade-off between parameters and tokens, favoring longer training over larger models.

Practical Implications

1. Data-scarce domains: Moderate sparsity (~50%) can significantly improve performance when data is limited.

2. Compute efficiency: 8–10× FLOPs reduction at compute-optimal sparsity translates to substantial energy and cost savings.

3. Inference efficiency: Smaller parameter-to-token ratios mean more efficient deployment.

4. Scaling guidance: As data budgets grow, higher sparsity becomes increasingly compute-optimal ($S \approx 0.6$–$0.75$).

Conclusion

This work demonstrates that sparsity is not just an efficiency tool but a mechanism for improving scaling trade-offs under data scarcity. The key takeaways are:

1. Sparse training improves data utilization by delaying repeated-data saturation
2. Moderate sparsity ($S \approx 0.5$) is loss-optimal across data budgets
3. Compute-optimal sparsity grows with data scale, reaching $S \approx 0.6$–$0.75$ at larger budgets
4. Sparse models achieve dense-equivalent performance with 8–10× less compute

The authors suggest that dynamic sparse training provides a principled approach for training large language models when data availability—not compute—is the primary bottleneck. Future work could extend these findings to architectural sparsity (e.g., MoE), other data modalities, and practical sparse training implementations that achieve ideal hardware-level speedups.