# Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

> Decoupling the router's renormalization reference set from the number of activated experts makes top-4 sparsity nearly lossless, turning a 4.65-point MMLU drop into 0.35.

- **Source:** [arXiv](https://arxiv.org/abs/2609.04575)
- **Published:** 2026-09-12
- **Permalink:** https://picx.dev/p/p8Mq40
- **Whiteboard:** https://picx.dev/p/p8Mq40/image

## Summary

## Summary (Overview)

- **Core contribution**: The paper introduces a training-free method to halve the number of activated experts in fine-grained Mixture-of-Experts (MoE) models by decoupling the number of activated experts ($k_1$) from the reference set size ($k_2$) used in router probability renormalization.
- **Key finding**: Reducing activated experts from 8 to 4 on Qwen3.6-35B-A3B causes a 4.65-point MMLU drop under standard renormalization, but only 0.35 points when normalizing by the top-$k_2 = 16$ mass instead.
- **Scalability**: The result replicates on the 11× larger Qwen3.5-397B-A17B model, where reducing from 10 to 5 activated experts loses only 0.55 MMLU points with an appropriate reference set.
- **No overhead**: The method introduces one integer parameter with no training, no parameters, and no measurable compute overhead—the router already computes all expert probabilities.
- **Practical recommendation**: When reducing $k$, scan $k_2 \in \{k_1, k, 2k\}$ with paired statistical testing; this recovers 4.30 points (35B) and 1.55 points (397B) at identical compute.

---

## Introduction and Theoretical Foundation

### Background: Fine-Grained MoE

Sparse Mixture-of-Experts (MoE) architectures decouple parameter count from per-token compute by activating only a few of many expert sub-networks. Recent open-weight models push this toward **fine-grained sparsity**: hundreds of small experts per layer, with each token routed to only a handful.

In Qwen3.6-35B-A3B (the primary study object):
- 40 layers, all MoE layers with 256 experts each
- $k = 8$ experts selected per token
- Routed experts hold 32.2B of 35.9B parameters (89.6%)
- Each token touches only 3.1% of expert parameters—a 32:1 memory-to-compute asymmetry

### The Renormalization Problem

Nearly all modern MoE implementations renormalize selected router probabilities:

$$w_i = \frac{p_i}{\sum_{j \in \mathcal{T}_k} p_j}$$

where $\mathcal{T}_k$ is the set of top-$k$ experts. In coarse-grained routers this is nearly a no-op, but in fine-grained routers it is not:

- The top-8 of 256 probabilities sum to only **0.182**
- Renormalization multiplies them by **5.5 on average**

The cumulative router softmax mass by rank (Table 3 in the paper) shows:
- Single highest-scoring expert: 0.046 of probability mass
- Top-8 together: 0.182
- Top-32: 0.380

### The Conflated Effects

Lowering $k$ from 8 to 4 while keeping renormalization:
1. **Discards four experts** (capacity loss)
2. **Redistributes their weight onto survivors**, inflating the expert branch's contribution beyond what training calibrated (gain miscalibration)

The measured degradation of "top-4" conflates these two effects. The paper's key insight is that the numerator and denominator of renormalization need not use the same expert set.

---

## Methodology

### Decoupled Reference Set

The paper proposes activating the top $k_1$ experts but normalizing by the mass of the top $k_2$ experts:

$$w_i = \frac{p_i}{\sum_{j \in \mathcal{T}_{k_2}} p_j}, \quad i \in \mathcal{T}_{k_1}$$

where:
- $k_1$ controls compute (only $k_1$ expert FFNs are evaluated)
- $k_2$ controls gain (the renormalization reference set)
- The router already computes all $E$ probabilities, so varying $k_2$ costs nothing measurable

When $k_2 > k_1$, weights no longer sum to one; they sum to $m_{k_1}(x)/m_{k_2}(x) < 1$, which is a per-token, input-adaptive down-scaling rather than a constant.

### Models Studied

| Model | Layers | Hidden Size | Experts/Layer | Expert Size | $k$ | Shared Expert |
|-------|--------|-------------|---------------|-------------|-----|---------------|
| Qwen3.6-35B-A3B | 40 | 2048 | 256 | 512 | 8 | Yes (learned scalar gate) |
| Qwen3.5-397B-A17B | 60 | 4096 | 512 | 1024 | 10 | Yes |

### Evaluation Protocol

- **Primary**: 2000-question, 5-shot MMLU with paired McNemar testing
- **Secondary**: GSM8K (generation task), C-Eval (Chinese benchmark), WikiText perplexity
- All results obtained **without any training, distillation, or fine-tuning**

---

## Empirical Validation / Results

### Main Results: Halving Activated Experts

**Qwen3.6-35B-A3B** ($k = 8 \to k_1 = 4$):
- Standard renormalization ($k_2 = k_1 = 4$): **−4.65 MMLU points** ($p = 1.7 \times 10^{-9}$)
- Decoupled reference ($k_2 = 16$): **−0.35 MMLU points** ($p = 0.26$, not significant)
- Near-symmetric flip distribution (99/92)

**Qwen3.5-397B-A17B** ($k = 10 \to k_1 = 5$):
- Standard renormalization ($k_2 = k_1 = 5$): **−2.10 MMLU points**
- Decoupled reference ($k_2 = 10$): **−0.55 MMLU points** ($p = 0.16$, not significant)

**Compute savings** (35B model, $k_1 = 4$ vs. $k = 8$):
- Routed-expert compute: 1.006B → 0.503B activated routed parameters per token (halved)
- Total expert term: 1.132B → 0.629B (44% reduction)
- End-to-end: 17% reduction against ~3.0B total

### The Catastrophic Endpoint

Removing renormalization entirely ($k_2 = E = 256$) costs:
- **27.5 MMLU points**
- **356% perplexity increase**

This shows the model is not indifferent to weight magnitudes—the reference set matters.

### Reference Set Size Sweep

Table 2 (routing ablations at fixed compute, 8 experts activated):

| Setting | Perplexity | MMLU |
|---------|-----------|------|
| Baseline ($k_1 = k_2 = 8$) | 0% | 0 |
| $k_1 = 6, k_2 = 6$ | +4.08% | −2.60 |
| $k_1 = 6, k_2 = 8$ | **−1.61%** | +0.15 |
| $k_1 = 4, k_2 = 4$ | +4.08% | −4.65 |
| $k_1 = 4, k_2 = 16$ | +1.2% | **−0.35** |

Key observations:
- On 35B, perplexity is minimized at $k_2 = 8$ but MMLU at $k_2 = 16$—the perplexity-optimal setting is significantly worse on MMLU (−1.10 points, $p = 0.03$)
- Shrinking the reference set together with $k_1$ is significantly harmful in three of four settings
- At $k_1 = 6$, standard renormalization costs +4.08% perplexity, while $k_2 = 8$ yields −1.61%—slightly better than native top-8 baseline while activating 22% fewer expert parameters

### GSM8K Validation (Generation Task)

Reducing to $k_1 = k/2$ under standard renormalization:
- 35B: **−6.60 points** (larger than MMLU loss, as expected if miscalibrated gain compounds token-by-token)
- 397B: **−4.20 points**
- Both highly significant

This confirms that single-token multiple-choice scores cannot reveal damage accumulating over generated sequences.

### Expert Identity vs. Weighting

Three ablations at fixed compute (8 experts activated) show **expert identity matters substantially more than expert weighting**:
- Changing which experts are selected (while keeping weights) causes large degradation
- Changing weights (while keeping experts) causes minimal degradation
- Router is well balanced with essentially no dead experts
- Domain specialization is strong: WikiText and code top-64 expert sets overlap at 0.164, below the 0.250 random baseline

---

## Theoretical and Practical Implications

### Theoretical Significance

1. **The renormalization reference set is a free variable**: The numerator (activation set) and denominator (reference set) of router renormalization need not coincide. This decoupling reveals that prior top-k reduction numbers on fine-grained models are **likely pessimistic**.

2. **Not a binary choice**: The choice is not "renormalize or do not"—those are endpoints ($k_2 = k_1$ and $k_2 = E$) of a family whose useful values lie strictly between. The invariant is not a rule about renormalization but the **reference set**.

3. **The 32:1 memory-to-compute asymmetry** organizes the design space: reducing experts saves memory but not compute; reducing $k$ saves compute but not memory. This work attacks the compute side.

### Practical Recommendations

> "When compute or decode bandwidth binds, decouple the reference set from the activation count whenever $k$ is changed and scan $k_2 \in \{k_1, k, 2k\}$ with paired testing: one integer, three evaluations, and the difference between a significant 2–5 point MMLU regression and none."

1. **Scan $k_2$ over $\{k_1, k, 2k\}$** whenever $k_1$ is reduced—do not assume a universal value
2. **Validate on downstream tasks with paired testing**, not perplexity alone (perplexity and downstream accuracy select different reference sets)
3. **Use as a control** for any published top-k reduction result, which absent a decoupled $k_2$ overstates the cost of reducing $k$
4. **Expert pruning is not recommended** here: balanced routing and domain specialization leave limited room for pruning, whereas reducing $k_1$ with appropriate $k_2$ is nearly lossless

### Cautionary Notes

- A non-significant result is not proof of equivalence—the paired design only bounds residual loss
- Had only baseline and final candidate been evaluated (common practice), the 35B result would read as "top-4 is simply lossless" and attribute nothing to the reference set
- Conversely, the 397B $k_1 = 8$ block (where default is already fine, +0.15) shows reporting only that row would conclude renormalization never matters

---

## Conclusion

### Main Takeaways

1. **Decoupling activation count ($k_1$) from reference set size ($k_2$) turns a 4.65-point MMLU regression at half the activated experts into 0.35 points (35B)**, and a 2.10-point regression into 0.55 points (397B)—for one integer and no measurable compute.

2. **The reference set is a free variable** with a one-line implementation change, requiring no training, distillation, or fine-tuning.

3. **The effect is a direct consequence of fine granularity**: it is the flatness of a 256-way router's top-k mass that makes the renormalization gain large and thus makes the decoupling valuable.

4. **Expert identity matters more than expert weighting**, and balanced/domain-specialized routing leaves limited room for expert pruning—making $k_1$ reduction with decoupled $k_2$ the preferred training-free compute reduction strategy.

### Future Directions

- The paper does not propose a universal $k_2$ value; instead it recommends scanning over $\{k_1, k, 2k\}$ with paired testing
- The method should be validated on additional tasks and model families
- The interaction between $k_2$ and other compression techniques (quantization, pruning) remains unexplored

---

_Markdown view of https://picx.dev/p/p8Mq40, served by PicX — AI-generated visual whiteboard summaries of research papers._