You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

Summary (Overview)

• Key finding: Fine-grained MoE models exhibit substantial routing redundancy—uniformly retaining approximately two-thirds (⌈2K/3⌉) of natively selected experts preserves 98.8% of unpruned performance on average across nine architecture families and three task families.
• Simple baseline strength: A one-integer change (lowering K uniformly) delivers 1.2–1.7× measured speedup across two serving backends (vLLM and HF Transformers), leaving little room for dynamic allocation at conservative budgets.
• Dynamic pruning value: Best published dynamic rules (NAEE, DynRoute, DiEP, BAN) differ from uniform truncation by less than one percentage point at conservative budgets (−0.53 to +0.67 points), but recover up to 3.0 points under aggressive pruning, with gains concentrated in generative tasks.
• Sensitivity drivers: Larger and "thinking" models are more resilient to aggressive pruning, while multimodal models are more vulnerable due to flatter routing distributions.
• Study scope: Systematic empirical study of twelve fine-grained MoE checkpoints spanning nine architecture families, evaluated on eleven benchmarks covering knowledge QA, mathematics, code generation, and general reasoning.

Introduction and Theoretical Foundation

Background and Motivation

Mixture-of-experts (MoE) architectures have evolved from routing among a few wide experts to combining many narrower ones. This evolution is dramatic:

• Mixtral activates 2 of 8 experts per token
• Qwen3-30B-A3B activates 8 of 128 routed experts
• Qwen3-Next-80B-A3B activates 10 of 512 routed experts
• Kimi K3 activates 16 of 896 routed experts

A top-K router assigns the same expert count to every token, regardless of whether its scores are sharply peaked or spread across selected experts. Dynamic expert pruning makes this count adaptive using router scores.

Theoretical Foundation

An MoE layer contains $E$ routed expert networks $f_1, \ldots, f_E$ and a router. Given a token representation $x$, the router selects $K$ experts with indices $e_1, \ldots, e_K$, ordered by descending gate weights $w_1 \geq \cdots \geq w_K$. The layer output is:

$$\text{MoE}(x) = \sum_{i=1}^{K} w_i f_{e_i}(x) + \sum_{j=1}^{n_s} f^{\text{shared}}_j(x)$$

where $n_s$ denotes the shared experts applied to every token.

Pruning retains a subset of selected experts. Let $S(x, \ell)$ denote their ranks at layer $\ell$, with $k(x, \ell) = |S(x, \ell)|$. Renormalising the retained weights gives:

$$\widehat{\text{MoE}}(x) = \sum_{i \in S(x,\ell)} \tilde{w}_i f_{e_i}(x) + \sum_{j=1}^{n_s} f^{\text{shared}}_j(x), \quad \tilde{w}_i = \frac{w_i}{\sum_{j \in S(x,\ell)} w_j}$$

Uniform truncation (Fixed-K) keeps the top $k$ experts for every token and layer. Dynamic rules determine retained experts from signals varying across tokens and layers.

Three Central Research Questions

1. Q1: How much redundancy does per-token expert selection carry in fine-grained MoE?
2. Q2: When does dynamic allocation improve on simply lowering the expert count?
3. Q3: What governs a model's sensitivity to expert pruning?

Methodology

Models

The study covers twelve checkpoints across nine architecture families:

Evaluation Tasks

Three task families with eleven benchmarks:

1. Knowledge QA (likelihood-scored): ARC-Easy, ARC-Challenge, WinoGrande, OpenBookQA
2. Math & Code Reasoning (generative): AIME-24, AIME-25, MATH-500, GSM8K, LiveCodeBench-v6
3. Knowledge & General Reasoning (generative): MMLU-Pro, GPQA-Diamond

Dynamic Pruning Rules Compared

NAEE: Keeps rank $i$ if $i < k_{\min}$ or $w_j \geq \beta w_1$ for all $j$ with $k_{\min} \leq j \leq i$.

DynRoute: Keeps shortest prefix reaching cumulative mass target:

$$k(x, \ell) = \min\left\{ i : \sum_{j \leq i} \hat{w}_j \geq p \right\}$$

DiEP: Threshold scaled by similarity: $\gamma_i = \left(\frac{\text{sim}_\ell(e_1, e_i)}{\overline{\text{sim}}_\ell}\right)^\alpha$, keep $i$ if $w_i \geq \beta \gamma_i w_1$.

BAN: Combines token sensitivity $s_{\text{tok}}$ with layer sensitivity $s_\ell$:

$$s = \text{clamp}\left(\frac{\lambda}{2} s_\ell + s_{\text{tok}}, 0, 1\right), \quad k(x, \ell) = \text{clamp}\left(\text{round}\left(k_{\min} + (K - k_{\min})s\right), k_{\min}, K\right)$$

Budget Measurement

The average number of active routed experts is measured as:

$$\bar{k} = \frac{1}{|T|L} \sum_{x \in T} \sum_{\ell=1}^{L} k(x, \ell), \quad \rho = \frac{\bar{k}}{K}$$

Empirical Validation / Results

Q1: Routing Redundancy

Uniformly retaining ⌈2K/3⌉ experts preserves 98.8% of unpruned performance on average across nine models and three task families.

Measured speedups at reduced budgets:

Q2: When Dynamics Help

Conservative budgets (retaining ~60-75% of experts): Best dynamic rule margin over Fixed-K ranges from −0.53 to +0.67 points, averaging just +0.06 points.

Aggressive budgets (retaining ~33-50% of experts): Best rule gains:

Key insight: Gains concentrate on generative tasks (up to +3.17 points on generative vs +1.20 on Knowledge QA at aggressive budgets). For example, on GPT-OSS-20B, DynRoute raises LiveCodeBench from 48.0 to 62.5—a 14.5-point gain.

Q3: Sensitivity Drivers

Reasoning mode: Thinking model (Qwen3-30B-A3B-Thinking) leads Instruct by +2.7 points at k=4 and +5.1 points at k=3 on the four hardest generative datasets.

Parameter scale: Qwen3-235B-A22B retains 76.7% of unpruned performance at k=3 versus 62.3% for Qwen3-30B-A3B—a 14.4-point gap.

Modality: Multimodal model (Qwen3-VL-30B-A3B) falls 11 points behind at k=3 and 42 points behind at k=2. The multimodal router is flatter: top-8 experts hold only 15.5% of softmax mass vs 44.3% for text model; top-3 of top-8 hold 49.8% vs 59.8%.

Theoretical and Practical Implications

For Deployment

1. Start with uniform truncation: Retaining ⌈2K/3⌉ experts provides a useful initial operating point with a single routing parameter change, delivering 1.2–1.7× speedup with negligible quality loss.

2. Use dynamic allocation only for aggressive pruning: Dynamic rules earn their complexity only when budgets drop below ~50% of native K, where they recover up to 3.0 points.

3. Match budget to capability requirements: Generative tasks (math, code, reasoning) deteriorate more sharply than likelihood-scored QA under aggressive pruning.

4. Consider model properties: Larger and thinking models tolerate aggressive pruning better; multimodal models require more conservative budgets.

For Future Method Design

1. Layer sensitivity as a promising signal: BAN's calibrated layer-sensitivity profile provides the strongest aggressive-budget results on three of four models.

2. Budget-aware evaluation: The winning rule changes across budgets and models, so evaluations at a single operating point can miss both gains and the best rule.

3. Router-width-invariant parameters: Cumulative retained mass and per-rank statistics keep their meaning across router widths, unlike ratios to top-1 gate weight.

Conclusion

This systematic study of dynamic expert pruning in fine-grained MoEs—spanning twelve checkpoints across nine architecture families—demonstrates that routing is far more redundant than the field's operating points assume. Retaining approximately two-thirds of natively selected experts preserves 98.8% of baseline performance with a simple one-integer change, delivering 1.2–1.7× decoding throughput.

Dynamic allocation's value emerges under aggressive pruning, where the best rules recover up to 3.0 points over uniform truncation, with gains concentrated in generative tasks. The winning rule changes across models and budgets, with BAN's layer-sensitivity approach performing best under aggressive budgets.

Model properties become consequential under aggressive pruning: larger and thinking models retain more capability, while multimodal models are more sensitive due to flatter routing distributions.

The findings establish uniform truncation as the essential reference for dynamic expert pruning, and the clearest opportunity for new allocation methods lies in preserving the generative capabilities that uniform truncation begins to lose at tight budgets.