# When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models

> Over-dispersed MoE routing breaks perplexity as a quality signal and creates domain trade-offs, which MESA's minimax expert scoring mitigates by protecting worst-affected domains.

- **Source:** [arXiv](https://arxiv.org/abs/2609.04453)
- **Published:** 2026-09-12
- **Permalink:** https://picx.dev/p/9163Yj
- **Whiteboard:** https://picx.dev/p/9163Yj/image

## Summary

## Summary (Overview)

- **Identifies a new pruning regime**: The paper characterizes "over-dispersed routing" in Mixture-of-Experts (MoE) models, induced by aggressive load-balancing loss coefficients ($\lambda_{aux} = 0.9$), where router probabilities become nearly uniform and standard importance signals collapse.
- **Demonstrates PPL–accuracy disconnect**: In over-dispersed models, perplexity does not predict downstream task accuracy—the lowest-PPL pruning configuration yields the worst mathematical reasoning performance, while the highest-PPL configuration preserves it.
- **Reveals a capability trade-off**: Under over-dispersed routing, no single scoring metric dominates across tasks—activation-aware methods (REAP) preserve math reasoning but degrade science tasks by an 18% gap on GPQA, while frequency-based methods exhibit the reverse.
- **Proposes MESA**: A Minimax Expert Score Allocation algorithm that iteratively boosts importance scores for experts serving the worst-affected domain, minimizing worst-case degradation rather than average accuracy.
- **Validates regime-specificity**: Cross-architecture experiments confirm the behavior is regime-specific—it does not occur under standard routing (e.g., Mixtral-8x7B-Instruct), where perplexity and accuracy degrade together as expected.

## Introduction and Theoretical Foundation

Mixture-of-Experts (MoE) architectures achieve favorable compute–quality tradeoffs by activating only a subset of parameters per token. However, the large total parameter count creates substantial memory and serving costs. Expert pruning—the structured removal of entire expert modules—is a natural strategy for reducing these costs.

The paper identifies a distinct regime termed **over-dispersed routing**: models trained with unusually high auxiliary load-balancing loss coefficients (e.g., $\lambda_{aux} = 0.9$). In these models, the router distributes tokens nearly uniformly across all experts, creating two challenges:

1. **Perplexity (PPL)–accuracy disconnect**: The expected negative correlation between PPL and accuracy does not hold. Spearman $\rho$ is positive on GSM8K $(\rho = 0.70, p > 0.1)$ and near zero on GPQA $(\rho = -0.10)$.

2. **Domain unfairness**: While macro-level routing appears near-uniform, experts retain subtle domain-specific characteristics. Global importance scores average across all tokens, cancelling out weak domain signals, causing disproportionate degradation of some domains.

The auxiliary load-balancing loss is formally defined as:

$$
\mathcal{L}_{aux} = \lambda_{aux} \cdot E \cdot \sum_{i=1}^{E} f_i \cdot \pi_i \tag{1}
$$

where $f_i$ is the fraction of tokens routed to expert $i$ and $\pi_i$ is the mean routing probability for expert $i$. In gpt-oss-20b, $\lambda_{aux} = 0.9$, which is orders of magnitude larger than typical values (~0.001–0.01 in Mixtral and Switch Transformer).

The critical diagnostic is **per-token routing entropy**:

$$
H(\mathbf{x}) = -\sum_{i=1}^{E} \pi_i(\mathbf{x}) \log_2 \pi_i(\mathbf{x})
$$

- **Mixtral-8x7B-Instruct** ($\lambda_{aux} \approx 0.001$, $E=8$, top-2): Mean per-token entropy $\bar{H}_{token} = 2.23$ bits (74.3% of maximum $\log_2(8) = 3.0$ bits), with entropy dropping to ~65% in later layers as specialization emerges.
- **gpt-oss-20b** ($\lambda_{aux} = 0.9$, $E=32$, top-4): Mean per-token entropy $\bar{H}_{token} = 4.54$ bits (90.8% of $\log_2(32) = 5.0$ bits), maintaining >82% entropy across all layers.

The paper uses $\hat{H} > 0.85 \cdot \log_2 E$ as a practical diagnostic for identifying over-dispersed regimes.

## Methodology

### MESA: Minimax Expert Score Allocation

Let $\mathbf{R} \in \mathbb{R}^{L \times E}$ denote per-layer normalized router mass scores and $\mathbf{D}_d \in \mathbb{R}^{L \times E}$ denote per-domain router mass for domain $d \in \{1, \dots, D\}$. Given a target pruning ratio $r$, the minimax algorithm iteratively adjusts importance scores to protect whichever domain is currently worst-affected:

$$
\min_{\mathcal{P} \subseteq [L] \times [E], |\mathcal{P}| = \lfloor r \cdot L \cdot E \rfloor} \max_{d \in \mathcal{D}} \frac{\sum_{(l,e) \in \mathcal{P}} \mathbf{D}_d[l, e]}{\sum_{l,e} \mathbf{D}_d[l, e]} \tag{2}
$$

**Algorithm 1** (greedy heuristic solver):

```
Require: Router mass R ∈ R^{L×E}, domain scores {D_d}_{d=1}^{D}, ratio r, boost η₀=0.15, steps T=20
Ensure: Pruning set P* (experts to remove)

1: S ← R; S* ← R; ℓ* ← ∞
2: for t = 1 to T do
3:   P ← ALLOCATE(S, r)  {Global ranking}
4:   ℓ_d ← Σ_{(l,e)∈P} D_d[l,e] / Σ_{l,e} D_d[l,e]  {∀d ∈ {1, ..., D}}
5:   d* ← arg max_d ℓ_d  {Worst domain}
6:   if ℓ_{d*} < ℓ* then
7:     ℓ* ← ℓ_{d*}; S* ← S
8:   end if
9:   η_t ← η₀ · (1 - 0.5t/(T-1))  {Decay}
10:  S ← ROWNorm(S + η_t · D_{d*})  {Boost + normalize}
11: end for
12: return S*
```

The full pipeline: (1) profile per-domain routing mass from calibration data, (2) run Algorithm 1, (3) apply global ranking to select experts for removal.

### Experimental Setup

- **Primary model**: gpt-oss-20b (24 layers, 32 experts per layer, 768 total, top-4 routing, $\lambda_{aux} = 0.9$)
- **Cross-architecture validation**: Mixtral-8x7B-Instruct (standard routing), OLMoE-1B-7B-0924 (over-dispersed base model), Gemma-4-26B-A4B-it (concentrated routing)
- **Calibration**: 1000 examples per domain from 6 domains (math, coding, commonsense, science, instruction following, general knowledge)
- **Evaluation**: 11 benchmarks spanning math reasoning (GSM8K, AIME 2025), graduate-level science (GPQA-Diamond), broad knowledge (MMLU, MMLU-Pro), commonsense reasoning (ARC-Challenge, ARC-Easy, OpenBookQA, PIQA, WinoGrande), and instruction following (IFEval)
- **Pruning ratio**: $r = 0.25$ (25% of experts removed, 32 → 24 per layer for gpt-oss)

## Empirical Validation / Results

### PPL–Accuracy Disconnect

**Table 1: WikiText-2 PPL vs. GSM8K accuracy at $r = 0.25$**

| Method | WikiText-2 PPL (↓) | GSM8K (%) (↑) |
|--------|-------------------|----------------|
| Baseline (unpruned) | 126.02 | 88.86 |
| MESA (ours) | 107.14 | 92.87 |
| REAP | 145.04 | 89.31 |
| RM-UNIFORM | 83.29 | 83.24 |
| Random | 40.95 | 78.54 |

Random pruning achieves the lowest PPL (40.95) but near-worst task accuracy, while MESA improves GSM8K above the unpruned baseline despite moderate PPL.

### Main Results at $r = 0.25$

**Table 2: Task accuracy (%) at $r = 0.25$ (5-seed evaluation with 95% CIs)**

| Method | AIME'25 | GSM8K | GPQA | ARC-C | ARC-E | OBQA | PIQA | MMLU | MMLU-Pro | WinoGr. | IFEval |
|--------|---------|-------|------|-------|-------|------|------|------|----------|---------|--------|
| Baseline | 76.67±3.27 | 88.86±0.38 | 64.47±1.46 | 95.30±0.27 | 98.32±0.15 | 92.99±0.45 | 83.34±0.36 | 84.01±0.10 | 72.51±0.22 | 76.78±1.04 | 91.67±0.67 |
| **MESA** | 63.33±6.06 | **92.87**±0.45 | **65.48**±4.2 | 92.66±0.8 | **97.60**±0.3* | 90.38±1.2 | **80.73**±1.5 | 79.51±0.6 | **67.25**±0.9 | 69.59±2.1 | 89.92±1.1 |
| REAP | **73.33**±9.7 | 89.31±1.8 | 47.72±5.6 | 91.63±1.0 | 95.03±0.7 | **92.18**±1.0 | 74.90±2.0 | 77.36±0.8 | 64.27±1.2 | **78.44**±2.4 | **90.88**±0.9 |
| RM-UNI. | 46.67±8.5 | 83.24±1.5 | 56.85±5.1 | 94.36±0.6 | 97.56±0.4 | 92.18±0.9 | 74.85±1.8 | **80.31**±0.5 | 66.32±0.8 | 69.27±2.3 | 86.07±1.4 |
| Random | 33.33±7.2 | 78.54±2.3 | 49.75±4.8 | 90.01±1.2 | 96.46±0.5 | 86.97±1.5 | 78.61±1.6 | 74.76±0.9 | 56.06±1.3 | 72.91±2.8 | 89.68±1.2 |
| EvoESAP† | 56.67±8.0 | 78.32±2.0 | 44.16±4.5 | **94.96**±0.6 | **97.64**±0.3* | 89.18±1.1 | 80.40±1.5 | 80.15±0.5 | 66.69±0.9 | 65.88±2.5 | 74.07±1.3 |

Key findings:
- MESA outperforms REAP on 7 benchmarks (GPQA +17.8%, PIQA +5.8%, GSM8K +3.6%, MMLU-Pro +3.0%, ARC-E +2.6%, MMLU +2.2%, ARC-C +1.0%)
- REAP leads on 4 (AIME25 +10.0%, WinoGrande +8.9%, OBQA +1.8%, IFEval +1.0%)
- The GPQA gap is statistically significant (z = 3.57, p < 0.001 on 198 questions)

### Capability Trade-off

**Table 9: Reasoning/Knowledge degradation at $r = 0.25$**

| Method | Δ_R (%) | Δ_K (%) | Character |
|--------|---------|---------|-----------|
| MESA (ours) | -4.67 | -1.68 | Balanced |
| REAP | -1.45 | -3.48 | Reasoning-preserving |
| EvoESAP† | -15.27 | -0.51 | Knowledge-preserving |
| RM-UNIFORM | -17.81 | -0.85 | Knowledge-preserving |
| Random | -26.83 | -3.58 | Severe collapse |

### Cross-Architecture Validation

**Mixtral-8x7B-Instruct** (standard routing): PPL behaves normally (increases with pruning). REAP retains 98% of baseline average accuracy while MESA retains only 63%, confirming MESA addresses a failure specific to over-dispersed routing.

**OLMoE-1B-7B-0924** (over-dispersed base model): Under eval-matched (prefill) calibration, MESA is the best pruned method on all six commonsense tasks, retaining 92.8% of unpruned accuracy at $r = 0.25$.

**Gemma-4-26B-A4B-it** (concentrated routing, 128 experts): MESA matches or exceeds both baselines on 10 of 11 benchmarks, with the largest gaps on AIME'25 (36.7pp over REAP) and GPQA-Diamond (+14.3pp over REAP).

### Calibration Efficiency

MESA converges at s=1000 samples, l=8192 tokens (97.9% prune map agreement). Even at s=20 samples per domain, MESA achieves 96.4% agreement. Compared to REAP's calibration requirement (402.7M tokens), MESA's 49.2M tokens is approximately 8× less calibration compute.

## Theoretical and Practical Implications

### Theoretical Implications

1. **Regime-specific pruning behavior**: The paper establishes that over-dispersed routing ($\lambda_{aux} \gg 0.01$) is a qualitatively distinct pruning regime where standard assumptions—that router probabilities provide reliable importance signals and that PPL correlates with task accuracy—fail.

2. **PPL can decrease with pruning**: Under over-dispersed routing, pruning concentrates routing mass onto fewer (potentially more specialized) experts, which can decrease perplexity while degrading downstream task performance.

3. **Global importance scores cancel micro-signals**: While macro-level routing appears uniform (each of 32 experts receives ~3.1% of global traffic), experts retain subtle domain-specific affinities. Global scores average these signals together, destroying domain-specific information.

4. **Capability trade-off structure**: PCA on the accuracy matrix identifies two principal components—PC1 loads on reasoning/generative tasks (GSM8K, AIME'25), PC2 loads on knowledge/pattern-matching tasks (MMLU, ARC)—with method rankings inverting across these clusters (permutation test $p = 0.0004$).

### Practical Implications

1. **Production MoE systems**: Production systems increasingly use high auxiliary loss coefficients to ensure load balance across hardware, making over-dispersed routing a practical concern for deployed models.

2. **Evaluation methodology**: Practitioners should report task accuracy rather than PPL for pruned over-dispersed models, as PPL is uninformative or misleading.

3. **Domain-aware pruning**: MESA provides a practical, post-training solution that requires no retraining or modification of the training pipeline, operating entirely on pre-computed router-mass matrices.

4. **Memory footprint**: At $r = 0.25$, removing a quarter of experts reduces the served model's parameter count and resident memory footprint by approximately the same fraction, though per-request decode latency is essentially unaffected since top-k routing remains unchanged.

## Conclusion

The paper establishes that over-dispersed MoE routing, induced by aggressive load-balancing ($\lambda_{aux} = 0.9$), constitutes a qualitatively distinct pruning regime with two key characteristics: perplexity becomes unreliable as a quality signal, and a capability trade-off emerges where no single scoring metric dominates across tasks.

**MESA** validates the practical relevance of this characterization by minimizing the worst-affected domain's degradation rather than average-case accuracy, outperforming REAP on 7 of 11 benchmarks at $r = 0.25$. However, it does not survive aggressive pruning ($r = 0.50$), where REAP's activation-aware signal remains robust.

**Key limitations**:
- Evaluation covers limited architectures; generalization to other over-dispersed models remains to be verified
- Algorithm 1 is a greedy heuristic without formal convergence guarantees (though empirical ablations show stable convergence within 15 iterations)
- Main results evaluated at a single pruning ratio ($r = 0.25$)

**Future directions**: Adaptive methods that automatically detect the routing regime and select scoring accordingly; multi-seed replication for OLMoE results; fully prefill-vs-prefill comparisons across all methods.

---

_Markdown view of https://picx.dev/p/9163Yj, served by PicX — AI-generated visual whiteboard summaries of research papers._