Summary (Overview)

• Core contribution: The paper introduces a training-free method to halve the number of activated experts in fine-grained Mixture-of-Experts (MoE) models by decoupling the number of activated experts ($k_1$) from the reference set size ($k_2$) used in router probability renormalization.
• Key finding: Reducing activated experts from 8 to 4 on Qwen3.6-35B-A3B causes a 4.65-point MMLU drop under standard renormalization, but only 0.35 points when normalizing by the top-$k_2 = 16$ mass instead.
• Scalability: The result replicates on the 11× larger Qwen3.5-397B-A17B model, where reducing from 10 to 5 activated experts loses only 0.55 MMLU points with an appropriate reference set.
• No overhead: The method introduces one integer parameter with no training, no parameters, and no measurable compute overhead—the router already computes all expert probabilities.
• Practical recommendation: When reducing $k$, scan $k_2 \in \{k_1, k, 2k\}$ with paired statistical testing; this recovers 4.30 points (35B) and 1.55 points (397B) at identical compute.

---

Introduction and Theoretical Foundation

Background: Fine-Grained MoE

Sparse Mixture-of-Experts (MoE) architectures decouple parameter count from per-token compute by activating only a few of many expert sub-networks. Recent open-weight models push this toward fine-grained sparsity: hundreds of small experts per layer, with each token routed to only a handful.

In Qwen3.6-35B-A3B (the primary study object):

• 40 layers, all MoE layers with 256 experts each
• $k = 8$ experts selected per token
• Routed experts hold 32.2B of 35.9B parameters (89.6%)
• Each token touches only 3.1% of expert parameters—a 32:1 memory-to-compute asymmetry

The Renormalization Problem

Nearly all modern MoE implementations renormalize selected router probabilities:

$$w_i = \frac{p_i}{\sum_{j \in \mathcal{T}_k} p_j}$$

where $\mathcal{T}_k$ is the set of top-$k$ experts. In coarse-grained routers this is nearly a no-op, but in fine-grained routers it is not:

• The top-8 of 256 probabilities sum to only 0.182
• Renormalization multiplies them by 5.5 on average

The cumulative router softmax mass by rank (Table 3 in the paper) shows:

• Single highest-scoring expert: 0.046 of probability mass
• Top-8 together: 0.182
• Top-32: 0.380

The Conflated Effects

Lowering $k$ from 8 to 4 while keeping renormalization:

1. Discards four experts (capacity loss)
2. Redistributes their weight onto survivors, inflating the expert branch's contribution beyond what training calibrated (gain miscalibration)

The measured degradation of "top-4" conflates these two effects. The paper's key insight is that the numerator and denominator of renormalization need not use the same expert set.

---

Methodology

Decoupled Reference Set

The paper proposes activating the top $k_1$ experts but normalizing by the mass of the top $k_2$ experts:

$$w_i = \frac{p_i}{\sum_{j \in \mathcal{T}_{k_2}} p_j}, \quad i \in \mathcal{T}_{k_1}$$

where:

• $k_1$ controls compute (only $k_1$ expert FFNs are evaluated)
• $k_2$ controls gain (the renormalization reference set)
• The router already computes all $E$ probabilities, so varying $k_2$ costs nothing measurable

When $k_2 > k_1$, weights no longer sum to one; they sum to $m_{k_1}(x)/m_{k_2}(x) < 1$, which is a per-token, input-adaptive down-scaling rather than a constant.

Models Studied

Evaluation Protocol

• Primary: 2000-question, 5-shot MMLU with paired McNemar testing
• Secondary: GSM8K (generation task), C-Eval (Chinese benchmark), WikiText perplexity
• All results obtained without any training, distillation, or fine-tuning

---

Empirical Validation / Results

Main Results: Halving Activated Experts

Qwen3.6-35B-A3B ($k = 8 \to k_1 = 4$):

• Standard renormalization ($k_2 = k_1 = 4$): −4.65 MMLU points ($p = 1.7 \times 10^{-9}$)
• Decoupled reference ($k_2 = 16$): −0.35 MMLU points ($p = 0.26$, not significant)
• Near-symmetric flip distribution (99/92)

Qwen3.5-397B-A17B ($k = 10 \to k_1 = 5$):

• Standard renormalization ($k_2 = k_1 = 5$): −2.10 MMLU points
• Decoupled reference ($k_2 = 10$): −0.55 MMLU points ($p = 0.16$, not significant)

Compute savings (35B model, $k_1 = 4$ vs. $k = 8$):

• Routed-expert compute: 1.006B → 0.503B activated routed parameters per token (halved)
• Total expert term: 1.132B → 0.629B (44% reduction)
• End-to-end: 17% reduction against ~3.0B total

The Catastrophic Endpoint

Removing renormalization entirely ($k_2 = E = 256$) costs:

• 27.5 MMLU points
• 356% perplexity increase

This shows the model is not indifferent to weight magnitudes—the reference set matters.

Reference Set Size Sweep

Table 2 (routing ablations at fixed compute, 8 experts activated):

Key observations:

• On 35B, perplexity is minimized at $k_2 = 8$ but MMLU at $k_2 = 16$—the perplexity-optimal setting is significantly worse on MMLU (−1.10 points, $p = 0.03$)
• Shrinking the reference set together with $k_1$ is significantly harmful in three of four settings
• At $k_1 = 6$, standard renormalization costs +4.08% perplexity, while $k_2 = 8$ yields −1.61%—slightly better than native top-8 baseline while activating 22% fewer expert parameters

GSM8K Validation (Generation Task)

Reducing to $k_1 = k/2$ under standard renormalization:

• 35B: −6.60 points (larger than MMLU loss, as expected if miscalibrated gain compounds token-by-token)
• 397B: −4.20 points
• Both highly significant

This confirms that single-token multiple-choice scores cannot reveal damage accumulating over generated sequences.

Expert Identity vs. Weighting

Three ablations at fixed compute (8 experts activated) show expert identity matters substantially more than expert weighting:

• Changing which experts are selected (while keeping weights) causes large degradation
• Changing weights (while keeping experts) causes minimal degradation
• Router is well balanced with essentially no dead experts
• Domain specialization is strong: WikiText and code top-64 expert sets overlap at 0.164, below the 0.250 random baseline

---

Theoretical and Practical Implications

Theoretical Significance

1. The renormalization reference set is a free variable: The numerator (activation set) and denominator (reference set) of router renormalization need not coincide. This decoupling reveals that prior top-k reduction numbers on fine-grained models are likely pessimistic.

2. Not a binary choice: The choice is not "renormalize or do not"—those are endpoints ($k_2 = k_1$ and $k_2 = E$) of a family whose useful values lie strictly between. The invariant is not a rule about renormalization but the reference set.

3. The 32:1 memory-to-compute asymmetry organizes the design space: reducing experts saves memory but not compute; reducing $k$ saves compute but not memory. This work attacks the compute side.

Practical Recommendations

"When compute or decode bandwidth binds, decouple the reference set from the activation count whenever $k$ is changed and scan $k_2 \in \{k_1, k, 2k\}$ with paired testing: one integer, three evaluations, and the difference between a significant 2–5 point MMLU regression and none."

1. Scan $k_2$ over $\{k_1, k, 2k\}$ whenever $k_1$ is reduced—do not assume a universal value
2. Validate on downstream tasks with paired testing, not perplexity alone (perplexity and downstream accuracy select different reference sets)
3. Use as a control for any published top-k reduction result, which absent a decoupled $k_2$ overstates the cost of reducing $k$
4. Expert pruning is not recommended here: balanced routing and domain specialization leave limited room for pruning, whereas reducing $k_1$ with appropriate $k_2$ is nearly lossless

Cautionary Notes

• A non-significant result is not proof of equivalence—the paired design only bounds residual loss
• Had only baseline and final candidate been evaluated (common practice), the 35B result would read as "top-4 is simply lossless" and attribute nothing to the reference set
• Conversely, the 397B $k_1 = 8$ block (where default is already fine, +0.15) shows reporting only that row would conclude renormalization never matters

---

Conclusion

Main Takeaways

1. Decoupling activation count ($k_1$) from reference set size ($k_2$) turns a 4.65-point MMLU regression at half the activated experts into 0.35 points (35B), and a 2.10-point regression into 0.55 points (397B)—for one integer and no measurable compute.

2. The reference set is a free variable with a one-line implementation change, requiring no training, distillation, or fine-tuning.

3. The effect is a direct consequence of fine granularity: it is the flatness of a 256-way router's top-k mass that makes the renormalization gain large and thus makes the decoupling valuable.

4. Expert identity matters more than expert weighting, and balanced/domain-specialized routing leaves limited room for expert pruning—making $k_1$ reduction with decoupled $k_2$ the preferred training-free compute reduction strategy.

Future Directions

• The paper does not propose a universal $k_2$ value; instead it recommends scanning over $\{k_1, k, 2k\}$ with paired testing
• The method should be validated on additional tasks and model families
• The interaction between $k_2$ and other compression techniques (quantization, pruning) remains unexplored