Summary (Overview)

• First systematic study of MoE design choices: This paper presents over 2,000 pretraining runs (models from 10M to 6.6B total parameters) exhaustively varying expert count, granularity, heterogeneous experts, shared experts, load balancing, and dropless routing—far beyond previous one-or-two-factor studies.

• Performance improves monotonically with total MoE parameters: Even at extreme activation sparsity ratios (up to 128), increasing inactive expert parameters consistently improves performance in FLOP-matched settings.

• Optimal expert size depends primarily on active parameter count, not total parameter count. The authors recommend scaling sparsity primarily by increasing total expert count while keeping expert granularity nearly fixed.

• Heterogeneous experts and generalists (shared experts) provide no benefit: Heterogeneous expert pools perform no better than well-configured homogeneous MoEs, and generalists consistently degrade performance.

• Simplified recipe: Focus optimization on expert count and granularity; load balancing hyperparameters have minimal impact within reasonable ranges, while dropless routing yields a small but consistent gain.

---

Introduction and Theoretical Foundation

Background

Mixture-of-Experts (MoE) architectures (Shazeer et al., 2017a) decouple computational overhead from model capacity by conditionally activating expert submodules via a learned router. Recent MoE LLMs (Fedus et al., 2022; DeepSeek-AI et al., 2025; Du et al., 2022) demonstrate improved efficiency at massive scale, yet the core design choices controlling sparse activation remain poorly understood.

Key Design Choices Under Investigation

1. Granularity and topology of experts: Total expert count (Clark et al., 2022), shared FFN components (Rajbhandari et al., 2022), and fine-grained experts (Dai et al., 2024).
2. Routing algorithms: Token choice, load balancing losses, loss-free mechanisms, and dropless routing.
3. Heterogeneous expert configurations: Mixed granularities within a single layer.

Theoretical Foundation

Granularity is defined as:

$$g = \frac{1}{\text{intermediate dimension of dense FFN}}$$

A granularity of 1 indicates expert FFNs the same size as the dense FFN; a granularity of $\frac{1}{4}$ indicates experts with intermediate dimension one-quarter that of the dense FFN.

FLOP-matching constraint: To fairly compare configurations, active expert parameters are matched:

$$\# \text{ of active experts} \cdot \text{expert granularity} = 1$$

Thus, to FLOP-match an MoE with granularity $g$ to a dense model, exactly $k = \frac{1}{g}$ experts are activated.

Activation sparsity is defined as:

$$s := n \cdot g$$

where $n$ is the total expert count—the factor by which total FFN parameters exceed active FFN parameters.

---

Methodology

Model Architecture

• MoE Transformer LMs based on Muennighoff et al. (2025)
• Models range from ~10M to 300M active parameters, and 10M to 6.6B total parameters
• Model scales: 10M, 20M, 50M, 80M, 110M, 200M, 300M (by active parameter count)
• FLOP-matched on a per-timestep basis by matching active FFN parameters

Experimental Grid

• Expert granularity: $g \in \{1, \frac{1}{2}, \frac{1}{4}, \frac{1}{8}, \frac{1}{16}, \frac{1}{32}, \frac{1}{64}\}$
• Total expert count: $n \in \{2, 4, 8, 16, 32, 64, 128, 256, 512, 1024\}$
• Activation sparsity: $s \in \{2, 4, 8, 16, 32, 64, 128\}$
• Active experts: $k = \frac{1}{g}$ (equivalently, $k \cdot g = 1$)

Additional Configurations Tested

• Heterogeneous experts: 2 pools of $(n_1, n_2)$ experts, equally dividing active and total parameters, with granularities $(\frac{1}{2}, \frac{1}{4})$, $(\frac{1}{4}, \frac{1}{8})$, $(\frac{1}{8}, \frac{1}{16})$ and counts $(n_1, n_2) \in \{(2, 4), (4, 8), (8, 16)\}$
• Generalists (shared experts): Always-active experts with granularity $g_{gen} \in \{\frac{1}{2}, \frac{1}{4}, \frac{1}{8}\}$
• FLOP-matching with generalists: $g_{gen} + \sum_i k_i \cdot g_i = 1$
• Sparsity with generalists: $s = g_{gen} + \sum_i n_i \cdot g_i$

Router Configurations Tested

Load balancing loss (Shazeer et al., 2017b):

$$\mathcal{L} = \mathcal{L}_{CE} + \alpha \mathcal{L}_{\text{auxiliary}} \tag{1}$$ $$\mathcal{L}_{LB} = N_E \cdot \sum_{i=1}^{N_E} f_i \cdot P_i \tag{2}$$

where $f_i$ is the fraction of tokens routed to expert $E_i$ and $P_i$ is its total routing probability.

Loss-free load balancing (DeepSeek-AI et al., 2025): Uses a per-expert bias adjusted by hyperparameter $\gamma$ at each training step.

Z-loss (Zoph et al., 2022a):

$$\mathcal{L}_{RZ}(x) = \frac{1}{B} \cdot \sum_{i=1}^{B} \left(\log \sum_{j=1}^{N_E} \exp(x_j^{(i)})\right)^2 \tag{3}$$

Dropless routing (Gale et al., 2022): Uses block-sparsity to ensure no tokens are dropped when local routing imbalance occurs.

Training Data & Evaluation

• Training mixture: web text, code, math, and encyclopedic text (from Muennighoff et al., 2025)
• Evaluation: macro-average cross-entropy loss on diverse held-out validation data
• 5 random seeds tested with near-zero standard deviation on LM loss

---

Empirical Validation / Results

1. Optimal Expert Configuration

MoEs outperform dense models (above a compute minimum). At 50M+ active parameters, even suboptimal MoE configurations often beat dense baselines. At 10M–20M scales, MoEs consistently fail to beat dense models—hypothesized to be a data/parameter budget interaction, confirmed by training with increased data budgets.

Key finding—increased inactive expert parameters improve performance: At fixed total expert count, performance improves with increased expert size (Figure 2, left). At fixed granularity, additional total (inactive) experts improve performance (Figure 2, center). No critical point of diminishing returns was observed in this setting.

Optimal granularity varies by active parameter scale:

Top-1 routing underperforms: Models with only one activated expert ($g = 1$) underperform the trend, likely because gradients update only one router row per token, preventing calibration across rows.

Granularity without sparsity is insufficient: Splitting FFNs into finer components without increasing total parameters (activation sparsity $s = 1$) substantially underperforms the dense baseline (Figure 3), confirming that increased total parameters—not granularity alone—drive MoE gains.

2. Generalists and Heterogeneity

• Heterogeneous experts alone do not improve performance: Results lie on an interpolated line between the most similar homogeneous settings (Figure 4).
• Generalists consistently degrade or match performance: Across all settings, adding any granularity of generalist results in comparable or degraded performance compared to no-generalist configurations (Figure 5).

3. Ablations

• Dropless routing improves performance: Consistent gains across scales (Figure 6). At the largest expert count ($n = 1024$), default routing becomes comparable, likely due to reduced token dropping when average expert load decreases.

• Load balancing settings have minimal impact within ranges: The worst setting used low load balancing weight ($\alpha_{LB} = 1e-4$) with no loss-free mechanism ($\gamma = 0$). At $n \leq 128$, no setting clearly dominates. At high $n$, loss-free load balancing with $\gamma = 1e-3$ damages performance, suggesting the bias requires tuning at scale (Figure 7).

---

Theoretical and Practical Implications

Simplified Design Recipe

The authors' findings suggest a dramatically simplified approach to MoE configuration:

1. Maximize total (inactive) expert parameters by increasing total expert count—performance improves monotonically even at extreme sparsity ratios.
2. Set expert granularity based on active parameter scale (coarse, around $\frac{1}{4}$ in their settings), not total parameter count.
3. Scale sparsity primarily by increasing expert count, keeping granularity nearly constant and slowly increasing.
4. Skip heterogeneous experts and generalists—they offer no benefit over well-configured homogeneous MoEs.
5. Use dropless routing for a small but consistent gain.
6. Tune load balancing minimally—a wide range of settings achieve near-optimal results, but avoid overly coarse loss-free bias adjustments at high expert counts.

Comparison with Prior Work

• Contradicts Zhao et al. (2025): Their claim that shared experts are essential and that a fixed optimal active expert count (~7) exists is contradicted; the authors find optima are scale-dependent.
• Contradicts Tian et al. (2025): Their recommendation of much more fine-grained experts likely stems from tying expert granularity to expert count; the strictly disentangled grid search here recommends coarser experts.
• Consistent with Muennighoff et al. (2025): Both find shared experts degrade performance.

Implications for Scaling Laws

The authors note that scaling laws derived from high-degree-of-freedom fits (Li et al., 2025) often fail to extrapolate reliably, suggesting that fixed constants like "optimal active expert count" may be artifacts of overfitting to a specific regime.

---

Conclusion

This work provides the first systematic, large-scale study of MoE design choices, spanning over 2,000 pretraining runs up to 6.6B total parameters. The central finding is that MoE performance is primarily driven by expert count and granularity—specifically, maximizing total expert parameters while setting expert size according to active parameter scale. Other design choices (heterogeneous experts, generalists, load balancing mechanisms) have minimal impact on final quality.

The practical recommendation is to minimally sweep hyperparameters and focus optimization effort on expert count and granularity, setting expert size according to active parameter scale, then setting total expert count to match memory allowances. Dropless routing provides a consistent, if small, additional gain.

Future directions suggested by this work include:

• Investigating whether these findings hold at even larger scales and with different data budgets
• Understanding the interaction between data budget and MoE benefits at small parameter scales
• Exploring whether the monotonic improvement with total expert parameters eventually reaches a critical point at even more extreme sparsity ratios

Code and data are released at: https://github.com/hadasah/slicing_and_dicing