Summary (Overview)

• Identifies a new pruning regime: The paper characterizes "over-dispersed routing" in Mixture-of-Experts (MoE) models, induced by aggressive load-balancing loss coefficients ($\lambda_{aux} = 0.9$), where router probabilities become nearly uniform and standard importance signals collapse.
• Demonstrates PPL–accuracy disconnect: In over-dispersed models, perplexity does not predict downstream task accuracy—the lowest-PPL pruning configuration yields the worst mathematical reasoning performance, while the highest-PPL configuration preserves it.
• Reveals a capability trade-off: Under over-dispersed routing, no single scoring metric dominates across tasks—activation-aware methods (REAP) preserve math reasoning but degrade science tasks by an 18% gap on GPQA, while frequency-based methods exhibit the reverse.
• Proposes MESA: A Minimax Expert Score Allocation algorithm that iteratively boosts importance scores for experts serving the worst-affected domain, minimizing worst-case degradation rather than average accuracy.
• Validates regime-specificity: Cross-architecture experiments confirm the behavior is regime-specific—it does not occur under standard routing (e.g., Mixtral-8x7B-Instruct), where perplexity and accuracy degrade together as expected.

Introduction and Theoretical Foundation

Mixture-of-Experts (MoE) architectures achieve favorable compute–quality tradeoffs by activating only a subset of parameters per token. However, the large total parameter count creates substantial memory and serving costs. Expert pruning—the structured removal of entire expert modules—is a natural strategy for reducing these costs.

The paper identifies a distinct regime termed over-dispersed routing: models trained with unusually high auxiliary load-balancing loss coefficients (e.g., $\lambda_{aux} = 0.9$). In these models, the router distributes tokens nearly uniformly across all experts, creating two challenges:

1. Perplexity (PPL)–accuracy disconnect: The expected negative correlation between PPL and accuracy does not hold. Spearman $\rho$ is positive on GSM8K $(\rho = 0.70, p > 0.1)$ and near zero on GPQA $(\rho = -0.10)$.

2. Domain unfairness: While macro-level routing appears near-uniform, experts retain subtle domain-specific characteristics. Global importance scores average across all tokens, cancelling out weak domain signals, causing disproportionate degradation of some domains.

The auxiliary load-balancing loss is formally defined as:

$$\mathcal{L}_{aux} = \lambda_{aux} \cdot E \cdot \sum_{i=1}^{E} f_i \cdot \pi_i \tag{1}$$

where $f_i$ is the fraction of tokens routed to expert $i$ and $\pi_i$ is the mean routing probability for expert $i$. In gpt-oss-20b, $\lambda_{aux} = 0.9$, which is orders of magnitude larger than typical values (~0.001–0.01 in Mixtral and Switch Transformer).

The critical diagnostic is per-token routing entropy:

$$H(\mathbf{x}) = -\sum_{i=1}^{E} \pi_i(\mathbf{x}) \log_2 \pi_i(\mathbf{x})$$

• Mixtral-8x7B-Instruct ($\lambda_{aux} \approx 0.001$, $E=8$, top-2): Mean per-token entropy $\bar{H}_{token} = 2.23$ bits (74.3% of maximum $\log_2(8) = 3.0$ bits), with entropy dropping to ~65% in later layers as specialization emerges.
• gpt-oss-20b ($\lambda_{aux} = 0.9$, $E=32$, top-4): Mean per-token entropy $\bar{H}_{token} = 4.54$ bits (90.8% of $\log_2(32) = 5.0$ bits), maintaining >82% entropy across all layers.

The paper uses $\hat{H} > 0.85 \cdot \log_2 E$ as a practical diagnostic for identifying over-dispersed regimes.

Methodology

MESA: Minimax Expert Score Allocation

Let $\mathbf{R} \in \mathbb{R}^{L \times E}$ denote per-layer normalized router mass scores and $\mathbf{D}_d \in \mathbb{R}^{L \times E}$ denote per-domain router mass for domain $d \in \{1, \dots, D\}$. Given a target pruning ratio $r$, the minimax algorithm iteratively adjusts importance scores to protect whichever domain is currently worst-affected:

$$\min_{\mathcal{P} \subseteq [L] \times [E], |\mathcal{P}| = \lfloor r \cdot L \cdot E \rfloor} \max_{d \in \mathcal{D}} \frac{\sum_{(l,e) \in \mathcal{P}} \mathbf{D}_d[l, e]}{\sum_{l,e} \mathbf{D}_d[l, e]} \tag{2}$$

Algorithm 1 (greedy heuristic solver):

Require: Router mass R ∈ R^{L×E}, domain scores {D_d}_{d=1}^{D}, ratio r, boost η₀=0.15, steps T=20
Ensure: Pruning set P* (experts to remove)

1: S ← R; S* ← R; ℓ* ← ∞
2: for t = 1 to T do
3:   P ← ALLOCATE(S, r)  {Global ranking}
4:   ℓ_d ← Σ_{(l,e)∈P} D_d[l,e] / Σ_{l,e} D_d[l,e]  {∀d ∈ {1, ..., D}}
5:   d* ← arg max_d ℓ_d  {Worst domain}
6:   if ℓ_{d*} < ℓ* then
7:     ℓ* ← ℓ_{d*}; S* ← S
8:   end if
9:   η_t ← η₀ · (1 - 0.5t/(T-1))  {Decay}
10:  S ← ROWNorm(S + η_t · D_{d*})  {Boost + normalize}
11: end for
12: return S*

The full pipeline: (1) profile per-domain routing mass from calibration data, (2) run Algorithm 1, (3) apply global ranking to select experts for removal.

Experimental Setup

• Primary model: gpt-oss-20b (24 layers, 32 experts per layer, 768 total, top-4 routing, $\lambda_{aux} = 0.9$)
• Cross-architecture validation: Mixtral-8x7B-Instruct (standard routing), OLMoE-1B-7B-0924 (over-dispersed base model), Gemma-4-26B-A4B-it (concentrated routing)
• Calibration: 1000 examples per domain from 6 domains (math, coding, commonsense, science, instruction following, general knowledge)
• Evaluation: 11 benchmarks spanning math reasoning (GSM8K, AIME 2025), graduate-level science (GPQA-Diamond), broad knowledge (MMLU, MMLU-Pro), commonsense reasoning (ARC-Challenge, ARC-Easy, OpenBookQA, PIQA, WinoGrande), and instruction following (IFEval)
• Pruning ratio: $r = 0.25$ (25% of experts removed, 32 → 24 per layer for gpt-oss)

Empirical Validation / Results

PPL–Accuracy Disconnect

Table 1: WikiText-2 PPL vs. GSM8K accuracy at $r = 0.25$

Random pruning achieves the lowest PPL (40.95) but near-worst task accuracy, while MESA improves GSM8K above the unpruned baseline despite moderate PPL.

Main Results at $r = 0.25$

Table 2: Task accuracy (%) at $r = 0.25$ (5-seed evaluation with 95% CIs)

Key findings:

• MESA outperforms REAP on 7 benchmarks (GPQA +17.8%, PIQA +5.8%, GSM8K +3.6%, MMLU-Pro +3.0%, ARC-E +2.6%, MMLU +2.2%, ARC-C +1.0%)
• REAP leads on 4 (AIME25 +10.0%, WinoGrande +8.9%, OBQA +1.8%, IFEval +1.0%)
• The GPQA gap is statistically significant (z = 3.57, p < 0.001 on 198 questions)

Capability Trade-off

Table 9: Reasoning/Knowledge degradation at $r = 0.25$

Cross-Architecture Validation

Mixtral-8x7B-Instruct (standard routing): PPL behaves normally (increases with pruning). REAP retains 98% of baseline average accuracy while MESA retains only 63%, confirming MESA addresses a failure specific to over-dispersed routing.

OLMoE-1B-7B-0924 (over-dispersed base model): Under eval-matched (prefill) calibration, MESA is the best pruned method on all six commonsense tasks, retaining 92.8% of unpruned accuracy at $r = 0.25$.

Gemma-4-26B-A4B-it (concentrated routing, 128 experts): MESA matches or exceeds both baselines on 10 of 11 benchmarks, with the largest gaps on AIME'25 (36.7pp over REAP) and GPQA-Diamond (+14.3pp over REAP).

Calibration Efficiency

MESA converges at s=1000 samples, l=8192 tokens (97.9% prune map agreement). Even at s=20 samples per domain, MESA achieves 96.4% agreement. Compared to REAP's calibration requirement (402.7M tokens), MESA's 49.2M tokens is approximately 8× less calibration compute.

Theoretical and Practical Implications

Theoretical Implications

1. Regime-specific pruning behavior: The paper establishes that over-dispersed routing ($\lambda_{aux} \gg 0.01$) is a qualitatively distinct pruning regime where standard assumptions—that router probabilities provide reliable importance signals and that PPL correlates with task accuracy—fail.

2. PPL can decrease with pruning: Under over-dispersed routing, pruning concentrates routing mass onto fewer (potentially more specialized) experts, which can decrease perplexity while degrading downstream task performance.

3. Global importance scores cancel micro-signals: While macro-level routing appears uniform (each of 32 experts receives ~3.1% of global traffic), experts retain subtle domain-specific affinities. Global scores average these signals together, destroying domain-specific information.

4. Capability trade-off structure: PCA on the accuracy matrix identifies two principal components—PC1 loads on reasoning/generative tasks (GSM8K, AIME'25), PC2 loads on knowledge/pattern-matching tasks (MMLU, ARC)—with method rankings inverting across these clusters (permutation test $p = 0.0004$).

Practical Implications

1. Production MoE systems: Production systems increasingly use high auxiliary loss coefficients to ensure load balance across hardware, making over-dispersed routing a practical concern for deployed models.

2. Evaluation methodology: Practitioners should report task accuracy rather than PPL for pruned over-dispersed models, as PPL is uninformative or misleading.

3. Domain-aware pruning: MESA provides a practical, post-training solution that requires no retraining or modification of the training pipeline, operating entirely on pre-computed router-mass matrices.

4. Memory footprint: At $r = 0.25$, removing a quarter of experts reduces the served model's parameter count and resident memory footprint by approximately the same fraction, though per-request decode latency is essentially unaffected since top-k routing remains unchanged.

Conclusion

The paper establishes that over-dispersed MoE routing, induced by aggressive load-balancing ($\lambda_{aux} = 0.9$), constitutes a qualitatively distinct pruning regime with two key characteristics: perplexity becomes unreliable as a quality signal, and a capability trade-off emerges where no single scoring metric dominates across tasks.

MESA validates the practical relevance of this characterization by minimizing the worst-affected domain's degradation rather than average-case accuracy, outperforming REAP on 7 of 11 benchmarks at $r = 0.25$. However, it does not survive aggressive pruning ($r = 0.50$), where REAP's activation-aware signal remains robust.

Key limitations:

• Evaluation covers limited architectures; generalization to other over-dispersed models remains to be verified
• Algorithm 1 is a greedy heuristic without formal convergence guarantees (though empirical ablations show stable convergence within 15 iterations)
• Main results evaluated at a single pruning ratio ($r = 0.25$)

Future directions: Adaptive methods that automatically detect the routing regime and select scoring accordingly; multi-seed replication for OLMoE results; fully prefill-vs-prefill comparisons across all methods.