# Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

> Mixture-of-Experts models overfit repeated data faster than dense Transformers, losing their advantage by 32x repetition, but dropout and output masking can restore it even at 64x.

- **Source:** [arXiv](https://arxiv.org/abs/2609.11917)
- **Published:** 2026-09-12
- **Permalink:** https://picx.dev/p/nVF6Ym
- **Whiteboard:** https://picx.dev/p/nVF6Ym/image

## Summary

## Summary (Overview)

- **Core finding**: Mixture-of-Experts (MoE) architectures overfit to repeated training data significantly more than densely-activated Transformers, with performance degradation scaling with total (rather than active) parameters.
- **Key result**: While 80M dense models tolerate up to 8× data repetition with minimal degradation, MoEs begin to suffer at 4× repetition and cede their performance advantage over dense models by 32× repetition.
- **Mechanism**: MoE routing stabilizes early in training, causing each expert to see a nearly fixed, reduced shard of data — under repetition, this leads to rapid memorization and over-specialization.
- **Remediation**: Dropout and output masking-based regularization can substantially mitigate repetition-driven overfitting; with strong masking, MoEs can outperform dense models even at 64× data repetition.
- **Scope**: Findings hold across model scales (80M, 200M, 1B active parameters), data domains (web crawl, code, academic, encyclopedic), and MoE configurations (expert count and granularity variations).

---

## Introduction and Theoretical Foundation

The paper addresses two converging trends in large language model (LLM) training:

1. **Data exhaustion**: The supply of unique human-written text is being depleted, making data repetition standard practice despite known overfitting risks (Muennighoff et al., 2023).
2. **Sparse architecture adoption**: Mixture-of-Experts (MoE) models (Shazeer et al., 2017) have become dominant due to their compute efficiency, achieved via sparsity — the ratio of total to active parameters.

The central tension is that:

- **Sparsity** decouples total parameters from active parameters.
- **Data repetition** decouples total tokens from unique tokens.

Both introduce axes of variation that complicate standard scaling laws, which prescribe simple token-to-parameter ratios without distinguishing unique vs. total tokens or active vs. total parameters. Prior work (e.g., Muennighoff et al., 2023) studied data repetition only for dense Transformers, while work on MoEs (e.g., a single 16-expert T5 configuration) suggested that multi-epoch degradation increases with total parameters — but a systematic investigation was lacking.

The paper hypothesizes that MoEs may overfit more because each expert's routed token set becomes fixed early in training, exposing each expert's feed-forward network to a significantly reduced set of unique tokens compared to dense FFNs.

---

## Methodology

### Model Configurations

- **Scales**: Models with 80M, 200M, and 1B active parameters (denoted 80M, 200M, 1B).
- **MoE variants**: Total experts $n \in \{8, 16, 32, 64, 128\}$ with granularity $g \in \{\frac{1}{2}, \frac{1}{4}, \frac{1}{8}, \frac{1}{16}\}$.
- **Notation**: Models denoted as MoE(n × g), e.g., MoE(64 × 1/4) = 4 active experts out of 64 total with granularity 1/4.

### Training Setup

- **Compute matching**: For each active-parameter scale $N_a$, total training tokens are fixed at $T \approx 20 \cdot N_a$.
- **Repetition rate**: Under fixed total data $T$, the number of unique tokens $U$ varies such that $R \cdot U = T$ for repetition rate $R$.
- **Data domains**: Single-domain and multi-domain mixes including DCLM (web crawl), peS2o (academic), StarCoder (code), and Wikipedia (encyclopedic text).
- **Evaluation**: Cross-Entropy (CE) Loss on held-out data across domains.

### Experimental Variations

1. **Sparsity effects**: Fixed granularity with varying total expert count; fixed expert count with varying granularity (Figure 4).
2. **Domain-specific repetition**: Varying repetition rates for different domains within data mixes (e.g., repeated StarCoder mixed with all-unique DCLM).
3. **Data quality**: Training on filtered (DCLM-BASELINE) vs. unfiltered data proportions.
4. **Regularization**: Dropout and output masking applied under high repetition regimes.

### Mechanistic Analyses

- **Routing stability**: Measuring consecutive checkpoint agreement of expert routing decisions.
- **Expert specialization**: Measuring "knockout cost" — the increase in held-out CE loss when an expert's output is zeroed at inference time.

---

## Empirical Validation / Results

### Data Repetition Effects Across Architectures

**Key finding (Figure 1)**: Across all scales, MoEs degrade more rapidly than dense models as repetition rate $R$ increases:

- Dense models: noticeable degradation at $R = 8$, sharp degradation at $R > 64$.
- MoEs: noticeable impact at $R = 4$, sharper increase at higher $R$.
- **Reversal point**: MoEs outperform dense models at $R \leq 16$, but dense models outperform MoEs at $R = 32$.

**Training loss behavior (Figure 2)**: At high repetition rates, training loss falls to near 0 (below 1E-2 at $R = 512$ for 80M dense, and $R = 160$ for 200M dense), indicating memorization, while validation loss rises.

### Sparsity and Total Parameters

**Figure 4** shows that overfitting increases with total parameters via two paths:

- (a) Fixed granularity, more total experts → more overfitting.
- (b) Fixed total expert count, larger experts → more overfitting.

**Total vs. active parameters**: Comparing 200M dense models to 80M MoE(32 × 1/4) and MoE(64 × 1/4) models (158M and 244M total parameters respectively), the 200M dense model's repetition response lies between the two MoE configurations. This suggests repetition effects depend on **total parameters**, not active parameters.

**Total data budget**: Training 80M active models with 4× more data ($T/N_a = 80$) shows that repetition response changes only slowly with total token budget — total parameters are the dominant factor.

### Domain Robustness

**Figure 3**: Patterns are consistent across all four domains (web crawl, code, academic, encyclopedic). MoEs begin to underperform dense models at $R \in [16, 32]$ regardless of domain, despite the semantic diversity of the data.

### Domain Mixing Effects

- **Figure 7**: Mixing repeated data with non-repeated, semantically similar data can have a regularizing effect.
- DCLM + StarCoder mixes degrade similarly regardless of mixing percentage.
- DCLM + peS2o mixes: higher DCLM proportion slows degradation at $R = 32$ — academic text (peS2o) is more semantically similar to web text (DCLM) than code (StarCoder) is.

### Expert Specialization (Mechanistic Analysis)

**Routing stability**: For 80M MoE(64 × 1/4) models, consecutive checkpoint agreement is near chance (1/64) at the start of training but exceeds 0.9 in the second half — routing becomes nearly fixed.

**Knockout cost (Figure 10a)**: As $R$ increases, the cost of removing an expert grows, more rapidly at higher expert counts:

- 80M models, increasing $R$ from 1 to 32 increases knockout effect by 1.1× for 16 experts.
- Same increase: **2.3×** for 128 experts of 1/4 granularity.

This supports the hypothesis that each expert sees a nearly stationary, reduced shard of data, and under repetition, this leads to over-specialization and memorization.

### Regularization Results

- **Dropout and output masking** dramatically decrease the response to data repetition.
- With sufficiently high dropout probability, MoEs can outperform dense models even at 64× repetition.
- Some other regularization methods were found ineffective (not detailed in abstract).

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Scaling law refinement**: The results demonstrate that scaling laws must disambiguate between unique vs. total tokens and active vs. total parameters. The interaction between sparsity and data repetition creates a conditional benefit: sparsity helps on unique data but hurts under repetition.

2. **Mechanistic understanding**: The routing stability finding provides a concrete mechanism for MoE overfitting — early fixation of routing decisions means each expert's effective unique data exposure is far smaller than the global unique token count, amplifying memorization.

3. **Regularization theory**: The success of dropout and masking suggests that disrupting parameter specialization is key to mitigating repetition-driven overfitting, pointing toward a "specialization-regularization" axis for future architecture design.

### Practical Implications

1. **Data curation**: When repeated data must be used, mixing it with non-repeated, semantically similar data (e.g., web text with academic text) can reduce repetition effects.

2. **Architecture design**: Practitioners should consider total parameter count when anticipating data repetition sensitivity — sparser models with more total parameters will overfit faster.

3. **Training strategies**: Applying dropout or output masking is an effective, practical intervention for high-repetition training regimes, potentially enabling MoEs to retain their compute-efficiency advantages even under severe data constraints.

---

## Conclusion

The paper provides comprehensive evidence that MoE models overfit more severely to repeated training data than dense Transformers, with degradation scaling with total parameters and worsening with sparsity. The core mechanism involves early routing stabilization that confines each expert to a reduced data shard, leading to over-specialization and memorization. 

Key takeaways:

1. MoEs cede their performance advantage over dense models at 32× repetition.
2. Repetition effects depend on **total** parameters, not active parameters.
3. Dropout and output masking can substantially mitigate these effects, allowing MoEs to outperform dense models even at 64× repetition.
4. Mixing repeated data with semantically similar non-repeated data has a regularizing effect.

**Future directions**: The authors recommend further exploration of masking-based methods to minimize repetition-driven overfitting through decreased parameter specialization. The work opens avenues for designing architectures and training strategies that are robust to data scarcity while retaining the compute-efficiency benefits of sparsity.

---

_Markdown view of https://picx.dev/p/nVF6Ym, served by PicX — AI-generated visual whiteboard summaries of research papers._