# Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs

> Dynamic expert-skipping methods fail to consistently beat a simple static top-k baseline at matched compute, so static per-modality expert reduction is the recommended default.

- **Source:** [arXiv](https://arxiv.org/abs/2610.28369)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/s5g6Og
- **Whiteboard:** https://picx.dev/p/s5g6Og/image

## Summary

## Summary (Overview)

- This paper presents a controlled comparison of dynamic expert-skipping methods for Mixture-of-Experts (MoE) models against a simple static baseline: reducing the number of active experts $k$ uniformly across all layers and tokens.
- The authors classify four representative dynamic skipping methods (NAEE, DiEP, MoDES, Ada-K) into a unified taxonomy and benchmark them at matched expert-compute budget ($K_{eff}$) on two modern MoEs (Gemma4-26B-A4B, Qwen3-30B-A3B) and a vision-language model (Qwen3-VL).
- **Key finding**: The simple static baseline is surprisingly hard to outperform consistently. Dynamic methods often underperform it, and gains, when present, are minor (at most ~2.6 percentage points).
- The paper introduces a calibration-free, per-modality static top-k configuration ($k_{vision} = 1-2$, $k_{text} = 6$) that recovers ≈95% of original quality at ≈85% fewer active experts in vision-heavy tasks.
- A critical insight: the effective $k$ achieved during calibration can diverge substantially from the effective $k$ at test time (up to 6× difference), causing over- or underestimation of inference compute.

---

## Introduction and Theoretical Foundation

Mixture-of-Experts (MoE) models activate only a small subset of all parameters (called "experts") during each forward pass. Typical models activate $k = 8$ out of 128 to 512 available experts—fixed across all layers and tokens. The central research question is: **given a trained model, can we reduce the active number of experts while preserving quality, potentially dynamically per token and/or per layer?**

Static, fixed-$k$ MoE inference has clear practical advantages:
- Simple to implement
- Requires no additional calibration
- Matches the routing behavior learned during training
- Produces predictable, uniform compute and memory-access patterns across tokens

However, research has shown that not every token requires the same number of experts (Lu et al., 2024). Yang et al. (2025b) describe three benefits dynamic skipping can offer over fixed-$k$ inference:
1. Reduces total computations per forward pass
2. Reduces the number of unique experts active in small-batch scenarios (lowering memory bandwidth)
3. Reduces communication overhead and improves load balancing in distributed deployment

The authors note that some recent methods combine dynamic skipping with static model pruning, making it difficult to disentangle the benefit of skipping itself. Furthermore, methods have become increasingly complex and costly, ranging from calibrated scores to similarity approaches and learned allocators. The combination of different calibration regimes, operating points, and evaluations makes comparison difficult.

---

## Methodology

### Baselines

The natural baseline builds on the model's native routing: **statically reducing the number of experts $k$** (softmax top-$k$). For multimodal models, the authors extend this with separate $k$ values per modality: $k_{\text{text}}$ and $k_{\text{vision}}$. Both baseline variants are calibration-free and static.

### Methods for Comparison

Four representative dynamic skipping methods are evaluated:

| Method | Criteria | Decision | Tunable | Calibration (h) |
|--------|----------|----------|---------|-----------------|
| Softmax top-$k$ (baseline) | Gate weights | Keep top-$k$ | $k$ | — |
| Per-modality (ours) | Gate weights | Top-$k$ per modality | $k_{\text{text}}, k_{\text{vision}}$ | — |
| NAEE | Gate weights (cum.) | Cumulative-mass threshold | $\beta$ quantile | 0.5 |
| DiEP | Gate weights + CKA | Redundancy-scaled ratio | $\gamma_1$ quantile | 0.5 |
| MoDES | Gate prob. × layer imp. | Per-modality threshold | target skip ratio | 20 |
| Ada-K | Hidden state (learned) | Predicts expert count | $\lambda$ | 2.5 |

### Quantifying Per-Token Expert Effort

The authors define **effective K ($K_{eff}$)** as a hardware-agnostic proxy metric for compute:

$$K_{eff} = \frac{1}{T \cdot L} \sum_{t=1}^{T} \sum_{l=1}^{L} k_{t,l}$$

where $k_{t,l}$ is the number of experts activated for token $t$ in layer $l$. For the static baseline, $K_{eff} = k$ by construction; for dynamic strategies, it is a measured average. The authors distinguish between calibrated $K_{eff}^{calibrated}$ (based on calibration set) and measured $K_{eff}$ (counted at test time), which may diverge substantially.

### Evaluation Setup

- **Text-only models**: Qwen3-30B-A3B and Gemma4-26B-A4B (both with $k = 8$)
- **Multimodal model**: Qwen3-VL-30B-A3B-Instruct
- **Benchmarks**: HellaSwag, ARC-C, GSM8K, HumanEval (text); ChartQA, TextVQA, MME, DocVQA, InfoVQA, OCRBench (multimodal)
- **Calibration**: A realistic mixed corpus (C4_en, C4_zh, open-web-math, python-codes-25k) simulating calibration–evaluation mismatch typical of deployments

---

## Empirical Validation / Results

### Text-Only Evaluation

The Pareto frontier analysis (Figure 2) reveals:
- **NAEE** most closely matches the fixed top-$k$ baseline, with minor improvements
- **Allocator (Ada-K)** shows some gains in the $K_{eff} \approx 1$ regime for Qwen3
- **MoDES and DiEP** mostly perform worse than the baseline
- Overall benefit is very limited: at most ~2.6 percentage points compared to the interpolated baseline

**Critical context**: At $K_{eff} \leq 2.5$, the best strategy on Gemma4-26B-A4B performs worse on average than Gemma4-E4B, a dense model with 85% fewer effective parameters.

The composition of quality drops matters:
- Multiple-choice benchmarks degrade smoothly toward random-guessing floors
- **Generative tasks (GSM8K, HumanEval) collapse sharply** once $K_{eff}$ falls below a model-dependent threshold (~2 for Qwen3)

### Multimodal Evaluation

Key results from Figure 5:
- **MoDES** (modality-aware) outperforms NAEE and softmax top-$k$ by a large margin
- **However, MoDES fails to outperform the per-modality softmax top-$k$ baseline**
- The per-modality baseline with $k_{\text{vision}} = 1$ incurs only ~4.1% drop while reducing active experts by ≈85% (measured $K_{eff} = 1.08$)
- Setting $k_{\text{vision}} = 2$ incurs less than 1.8% average drop

### Calibrated vs. Measured $K_{eff}$

Table 2 shows substantial deviation between calibrated and measured $K_{eff}$ for MoDES:

| MoDES Target Skip Ratio | $K_{eff}^{calibrated}$ | Measured $K_{eff}$ (Vision) | Measured $K_{eff}$ (Text-Only) |
|---|---|---|---|
| 0.85 | 1.20 | 0.58 | 7.96 |
| 0.73 | 1.99 | 1.50 | — |
| 0.60 | 3.17 | 2.81 | — |

**Critical finding**: On text-only GSM8K, MoDES's $K_{eff}$ shifts to 7.96, far from the calibrated 1.2—the skipping effectively deactivates, and quality is retained only by reverting to nearly the full model. The calibrated compute saving vanishes entirely.

### Held-Out Validation

The recommended operating point ($k_{\text{vision}} = 1$, $k_{\text{text}} = 6$) was validated on:
- **MVBench** (held-out video benchmark): 69.95 vs. 71.40 accuracy (2.0% relative drop) at measured $K_{eff} = 1.19$ (85% reduction in active experts)
- **GLM-4.6V** (larger held-out model): 4.2% average drop at the same operating point

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **The value of dynamic skipping is questionable**: Despite added complexity, calibration cost, and computational overhead, dynamic methods do not systematically outperform a simple static baseline at matched measured compute.

2. **Modality is the key structural signal**: The primary benefit in multimodal settings comes from treating vision and text tokens differently, not from per-token dynamic decisions. This aligns with prior observations that vision tokens tolerate substantially fewer experts.

3. **Calibration–evaluation mismatch is a fundamental problem**: Threshold-based methods only enforce target skip ratios on calibration sets, which may transfer imperfectly across datasets and modality mixes.

### Practical Implications

1. **Recommendation**: Statically reduce $k_{\text{vision}}$ to 1–2 and $k_{\text{text}}$ to 6 for models with eight active experts. This provides a strong quality–compute trade-off (<5% drop) with substantial inference speed-up.

2. **For text-only models**: Reducing $k$ to 6 provides measurable speed-up with minor quality loss. Going further becomes heavily model-dependent and should be validated with generative benchmarks.

3. **Method design guidance**: Every future dynamic skipping method should compare against a static control at matched measured $K_{eff}$ that also uses any structural signal (e.g., modality) the method exploits.

4. **The $K_{eff}$ metric itself favors adaptive strategies**: Dynamic methods add routing decisions per forward pass and forfeit uniform per-token compute, load balancing, and latency guarantees—all excluded from $K_{eff}$-matched analysis.

---

## Conclusion

Across three models and two held-out checks, **none of the calibrated or learned skipping methods consistently outperform a static reduction of $k$ at matched measured $K_{eff}$**. The methods add calibration and tuning overhead, and their realized $K_{eff}$ can deviate from the calibrated target by up to 6×.

The authors recommend:
- **Static per-modality $k$ as the default** for practical deployment
- **A static control at matched measured $K_{eff}$ as the baseline** for future method evaluation

Future directions implied by this work include:
- Investigating why vision tokens tolerate so few experts
- Developing methods that provide guarantees on realized $K_{eff}$
- Exploring per-layer granularity for sub-one $K_{eff}$ regimes
- Understanding the sharp collapse of generative tasks at low $K_{eff}$

---

_Markdown view of https://picx.dev/p/s5g6Og, served by PicX — AI-generated visual whiteboard summaries of research papers._