# You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

> Retaining only two-thirds of natively selected experts preserves 98.8% of MoE performance, with dynamic pruning gaining at most 3 points only under aggressive budgets.

- **Source:** [arXiv](https://arxiv.org/abs/2609.25809)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/wStnJg
- **Whiteboard:** https://picx.dev/p/wStnJg/image

## Summary

# You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

## Summary (Overview)

- **Key finding**: Fine-grained MoE models exhibit substantial routing redundancy—uniformly retaining approximately two-thirds (⌈2K/3⌉) of natively selected experts preserves **98.8%** of unpruned performance on average across nine architecture families and three task families.
- **Simple baseline strength**: A one-integer change (lowering K uniformly) delivers **1.2–1.7× measured speedup** across two serving backends (vLLM and HF Transformers), leaving little room for dynamic allocation at conservative budgets.
- **Dynamic pruning value**: Best published dynamic rules (NAEE, DynRoute, DiEP, BAN) differ from uniform truncation by less than one percentage point at conservative budgets (−0.53 to +0.67 points), but recover up to **3.0 points** under aggressive pruning, with gains concentrated in generative tasks.
- **Sensitivity drivers**: Larger and "thinking" models are more resilient to aggressive pruning, while multimodal models are more vulnerable due to flatter routing distributions.
- **Study scope**: Systematic empirical study of **twelve fine-grained MoE checkpoints** spanning nine architecture families, evaluated on **eleven benchmarks** covering knowledge QA, mathematics, code generation, and general reasoning.

## Introduction and Theoretical Foundation

### Background and Motivation

Mixture-of-experts (MoE) architectures have evolved from routing among a few wide experts to combining many narrower ones. This evolution is dramatic:

- Mixtral activates **2 of 8** experts per token
- Qwen3-30B-A3B activates **8 of 128** routed experts
- Qwen3-Next-80B-A3B activates **10 of 512** routed experts
- Kimi K3 activates **16 of 896** routed experts

A top-K router assigns the same expert count to every token, regardless of whether its scores are sharply peaked or spread across selected experts. Dynamic expert pruning makes this count adaptive using router scores.

### Theoretical Foundation

An MoE layer contains $E$ routed expert networks $f_1, \ldots, f_E$ and a router. Given a token representation $x$, the router selects $K$ experts with indices $e_1, \ldots, e_K$, ordered by descending gate weights $w_1 \geq \cdots \geq w_K$. The layer output is:

$$
\text{MoE}(x) = \sum_{i=1}^{K} w_i f_{e_i}(x) + \sum_{j=1}^{n_s} f^{\text{shared}}_j(x)
$$

where $n_s$ denotes the shared experts applied to every token.

**Pruning** retains a subset of selected experts. Let $S(x, \ell)$ denote their ranks at layer $\ell$, with $k(x, \ell) = |S(x, \ell)|$. Renormalising the retained weights gives:

$$
\widehat{\text{MoE}}(x) = \sum_{i \in S(x,\ell)} \tilde{w}_i f_{e_i}(x) + \sum_{j=1}^{n_s} f^{\text{shared}}_j(x), \quad \tilde{w}_i = \frac{w_i}{\sum_{j \in S(x,\ell)} w_j}
$$

**Uniform truncation (Fixed-K)** keeps the top $k$ experts for every token and layer. **Dynamic rules** determine retained experts from signals varying across tokens and layers.

### Three Central Research Questions

1. **Q1**: How much redundancy does per-token expert selection carry in fine-grained MoE?
2. **Q2**: When does dynamic allocation improve on simply lowering the expert count?
3. **Q3**: What governs a model's sensitivity to expert pruning?

## Methodology

### Models

The study covers **twelve checkpoints** across nine architecture families:

| Model | Total | Active | E | K | n_s | Role |
|-------|-------|--------|---|---|-----|------|
| Qwen3-30B-A3B-Instruct | 30.5B | 3.3B | 128 | 8 | 0 | Core model |
| Qwen3-Next-80B-A3B-Instruct | 80B | 3B | 512 | 10 | 1 | Widest router |
| Ling-lite-1.5-2507 | 16.8B | 2.75B | 64 | 6 | 2 | Shared experts |
| GPT-OSS-20B | 21B | 3.6B | 32 | 4 | 0 | Narrowest router |
| MiniMax-M2.7 | 229B | 11B | 256 | 8 | 0 | Sigmoid router |
| DeepSeek-V2-Lite-Chat | 15.7B | 2.4B | 64 | 6 | 2 | Earliest fine-grained |
| Hy3 | 295B | 21B | 192 | 8 | 1 | Largest model |
| DeepSeek-V4-Flash-0731 | 165B | 11.5B | 256 | 6 | 1 | Sqrt-softplus router |
| Gemma 4 26B-A4B | 25.2B | 3.8B | 128 | 8 | 1 | GELU experts |
| Qwen3-30B-A3B-Thinking | 30.5B | 3.3B | 128 | 8 | 0 | Reasoning mode |
| Qwen3-235B-A22B-Instruct | 235B | 22B | 128 | 8 | 0 | Parameter scale |
| Qwen3-VL-30B-A3B-Instruct | 30B | 3B | 128 | 8 | 0 | Modality |

### Evaluation Tasks

Three task families with **eleven benchmarks**:

1. **Knowledge QA** (likelihood-scored): ARC-Easy, ARC-Challenge, WinoGrande, OpenBookQA
2. **Math & Code Reasoning** (generative): AIME-24, AIME-25, MATH-500, GSM8K, LiveCodeBench-v6
3. **Knowledge & General Reasoning** (generative): MMLU-Pro, GPQA-Diamond

### Dynamic Pruning Rules Compared

| Rule | Retention Decision | Layer Budget | Needs Calibration | k_min | Knob |
|------|-------------------|-------------|-------------------|-------|------|
| **Fixed-K** (baseline) | Constant k for all tokens | ✗ | ✗ | — | k |
| **NAEE** | Gate weight relative to top-1 | ✗ | ✗ | 2 | β |
| **DynRoute** | Cumulative retained probability mass | ✗ | ✗ | — | p |
| **DiEP** | Top-1 ratio scaled by expert similarity | ✗ | ✓ | 2 | β |
| **BAN** | Layer sensitivity + token sensitivity | ✓ | ✓ | 3 | λ |

**NAEE**: Keeps rank $i$ if $i < k_{\min}$ or $w_j \geq \beta w_1$ for all $j$ with $k_{\min} \leq j \leq i$.

**DynRoute**: Keeps shortest prefix reaching cumulative mass target:
$$k(x, \ell) = \min\left\{ i : \sum_{j \leq i} \hat{w}_j \geq p \right\}$$

**DiEP**: Threshold scaled by similarity: $\gamma_i = \left(\frac{\text{sim}_\ell(e_1, e_i)}{\overline{\text{sim}}_\ell}\right)^\alpha$, keep $i$ if $w_i \geq \beta \gamma_i w_1$.

**BAN**: Combines token sensitivity $s_{\text{tok}}$ with layer sensitivity $s_\ell$:
$$s = \text{clamp}\left(\frac{\lambda}{2} s_\ell + s_{\text{tok}}, 0, 1\right), \quad k(x, \ell) = \text{clamp}\left(\text{round}\left(k_{\min} + (K - k_{\min})s\right), k_{\min}, K\right)$$

### Budget Measurement

The average number of active routed experts is measured as:
$$\bar{k} = \frac{1}{|T|L} \sum_{x \in T} \sum_{\ell=1}^{L} k(x, \ell), \quad \rho = \frac{\bar{k}}{K}$$

## Empirical Validation / Results

### Q1: Routing Redundancy

**Uniformly retaining ⌈2K/3⌉ experts preserves 98.8% of unpruned performance** on average across nine models and three task families.

| Model | Native K → 2K/3 | Accuracy Retained |
|-------|-----------------|-------------------|
| Qwen3-30B-A3B | 8→6 | 98.6% |
| Qwen3-Next-80B | 10→7 | 100.5% |
| Ling-lite-1.5 | 6→4 | 97.4% |
| GPT-OSS-20B | 4→3 | 100.7% |
| MiniMax-M2.7 | 8→6 | 98.1% |
| DeepSeek-V2-Lite | 6→4 | 99.9% |
| Hy3 | 8→6 | 98.0% |
| DeepSeek-V4-Flash | 6→4 | 98.8% |
| Gemma 4 26B-A4B | 8→6 | 97.5% |

**Measured speedups** at reduced budgets:

| Model | K → k̄ | ρ | vLLM Prefill | vLLM Decode | HF Prefill | HF Decode |
|-------|--------|---|-------------|-------------|------------|-----------|
| Qwen3-30B-A3B | 8→5 | 0.63 | 1.15 | 1.18 | 1.26 | 1.70 |
| Qwen3-Next-80B | 10→6 | 0.60 | 1.25 | 1.65 | 1.31 | 1.73 |
| Ling-lite-1.5 | 6→4 | 0.67 | 1.21 | 1.38 | 1.21 | 1.52 |
| GPT-OSS-20B | 4→3 | 0.75 | 1.25 | 1.23 | 1.24 | 1.44 |

### Q2: When Dynamics Help

**Conservative budgets** (retaining ~60-75% of experts): Best dynamic rule margin over Fixed-K ranges from **−0.53 to +0.67 points**, averaging just **+0.06 points**.

**Aggressive budgets** (retaining ~33-50% of experts): Best rule gains:

| Model | Budget | Best Rule | Gain over Fixed-K |
|-------|--------|-----------|-------------------|
| Qwen3-30B-A3B | 8→3 | BAN | +2.45 |
| Qwen3-Next-80B | 10→4 | BAN | +2.71 |
| Ling-lite-1.5 | 6→2 | BAN | +2.98 |
| GPT-OSS-20B | 4→2 | DynRoute | +0.90 |

**Key insight**: Gains concentrate on generative tasks (up to +3.17 points on generative vs +1.20 on Knowledge QA at aggressive budgets). For example, on GPT-OSS-20B, DynRoute raises LiveCodeBench from 48.0 to 62.5—a 14.5-point gain.

### Q3: Sensitivity Drivers

**Reasoning mode**: Thinking model (Qwen3-30B-A3B-Thinking) leads Instruct by **+2.7 points at k=4** and **+5.1 points at k=3** on the four hardest generative datasets.

**Parameter scale**: Qwen3-235B-A22B retains **76.7%** of unpruned performance at k=3 versus 62.3% for Qwen3-30B-A3B—a **14.4-point gap**.

**Modality**: Multimodal model (Qwen3-VL-30B-A3B) falls **11 points behind** at k=3 and **42 points behind** at k=2. The multimodal router is flatter: top-8 experts hold only **15.5%** of softmax mass vs **44.3%** for text model; top-3 of top-8 hold **49.8%** vs **59.8%**.

## Theoretical and Practical Implications

### For Deployment

1. **Start with uniform truncation**: Retaining ⌈2K/3⌉ experts provides a useful initial operating point with a single routing parameter change, delivering 1.2–1.7× speedup with negligible quality loss.

2. **Use dynamic allocation only for aggressive pruning**: Dynamic rules earn their complexity only when budgets drop below ~50% of native K, where they recover up to 3.0 points.

3. **Match budget to capability requirements**: Generative tasks (math, code, reasoning) deteriorate more sharply than likelihood-scored QA under aggressive pruning.

4. **Consider model properties**: Larger and thinking models tolerate aggressive pruning better; multimodal models require more conservative budgets.

### For Future Method Design

1. **Layer sensitivity as a promising signal**: BAN's calibrated layer-sensitivity profile provides the strongest aggressive-budget results on three of four models.

2. **Budget-aware evaluation**: The winning rule changes across budgets and models, so evaluations at a single operating point can miss both gains and the best rule.

3. **Router-width-invariant parameters**: Cumulative retained mass and per-rank statistics keep their meaning across router widths, unlike ratios to top-1 gate weight.

## Conclusion

This systematic study of dynamic expert pruning in fine-grained MoEs—spanning twelve checkpoints across nine architecture families—demonstrates that routing is **far more redundant** than the field's operating points assume. Retaining approximately two-thirds of natively selected experts preserves 98.8% of baseline performance with a simple one-integer change, delivering 1.2–1.7× decoding throughput.

**Dynamic allocation's value emerges under aggressive pruning**, where the best rules recover up to 3.0 points over uniform truncation, with gains concentrated in generative tasks. The winning rule changes across models and budgets, with BAN's layer-sensitivity approach performing best under aggressive budgets.

**Model properties become consequential under aggressive pruning**: larger and thinking models retain more capability, while multimodal models are more sensitive due to flatter routing distributions.

The findings establish uniform truncation as the essential reference for dynamic expert pruning, and the clearest opportunity for new allocation methods lies in preserving the generative capabilities that uniform truncation begins to lose at tight budgets.

---

_Markdown view of https://picx.dev/p/wStnJg, served by PicX — AI-generated visual whiteboard summaries of research papers._