When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

Authors: Youngsik Yoon, Siwei Wang, Wei Chen, Jungseul Ok (POSTECH & Microsoft Research Asia)

---

Summary (Overview)

• Core finding: The standard top-k router in MoE language models is well-aligned with route utility on confident tokens but systematically misaligned on "fragile" tokens—the low-confidence tokens that drive hard reasoning—where lower-loss equal-compute routes consistently exist inside the frozen model but are not selected.
• Token-conditional misalignment: On Fragile tokens, the standard route is best among alternatives only ~0.8% of the time, while the best sampled equal-compute route improves next-token probability by up to 20.4 percentage points.
• Structural cause identified: The authors formalize a "counterfactual blind spot" in standard MoE training—the language modeling loss evaluates only the executed route, and load-balancing losses depend only on aggregate routing statistics, providing no token-level signal on unexecuted equal-compute alternatives.
• Empirical validation: The pattern holds across four open-weight MoE models (Qwen3-30B-A3B, GPT-OSS-20B, DeepSeek-V2-Lite, OLMoE-1B-7B), across layers, and across multiple benchmarks (MATH Level-5, AIME, HMMT, GPQA-Diamond).
• Router-only intervention: A minimal update to only the final-layer router (Expert Preference Optimization, EPO, <0.001% of parameters) shifts pass@K on AIME 2024+2025 and HMMT 2025, demonstrating that part of the failure is router-reachable misallocation, not purely expert capacity limits.

---

Introduction and Theoretical Foundation

Background

Mixture-of-Experts (MoE) language models scale model capacity by activating only a small subset of parameters per token. The router—typically a single learned linear projection followed by top-k selection—decides which experts process each token. While attention and expert components receive extensive scrutiny, the router has been treated largely as an implementation detail of sparsity, with load-balancing regularizers and downstream performance taken as sufficient evidence of good routing.

Key Question

The paper asks a fundamental but previously unaddressed question: Are the routes selected by a trained top-k router actually good ones? Rather than evaluating routing through aggregate statistics or downstream benchmarks, the authors perform a counterfactual analysis: holding the model fixed, they compare each standard route against sampled equal-compute alternatives for the same token, scoring each by the next-token probability it assigns to the realized token in a verified reasoning trajectory.

Theoretical Foundation

The paper formalizes a counterfactual blind spot in standard MoE training. For a token $t$, only the executed route $S_t^{\text{std}}$ enters the forward pass. The gradient of the cross-entropy loss $\mathcal{L}_t$ with respect to router score $s_{tj}$ within a fixed top-k cell is:

$$\frac{\partial \mathcal{L}_t}{\partial s_{tj}} = \mathbf{1}\{j \in S_t^{\text{std}}\} \, g_{tj}^{\text{std}} \Big(\nabla_{h_t^{\text{std}}} \mathcal{L}_t \cdot (E_j(x_t) - h_t^{\text{std}})\Big).\tag{3}$$

The indicator term is crucial: for experts not in the executed route, the gradient vanishes—their outputs never enter the computation graph. Two distinct routes $S, S' \subseteq \{1, \dots, N\}$ with $|S| = |S'| = k$ produce different cross-entropies $\mathcal{L}_t(S)$ and $\mathcal{L}_t(S')$ on the same token, but standard training observes only one of them.

Load Balancing Is Aggregate, Not Token-Level

The authors formally show that load-balancing losses do not provide token-level counterfactual signal. For the Switch-style loss with batch of $T$ tokens:

$$f_i = \frac{1}{Tk} \sum_{t=1}^{T} \mathbf{1}\{i \in S_t^{\text{std}}\}, \qquad \bar{p}_i = \frac{1}{T} \sum_{t=1}^{T} p_{ti},\tag{4}$$

the gradient is:

$$\nabla_{s_t} \mathcal{R}_{\text{lb}} = \frac{\lambda N}{T} \big(\text{diag}(p_t) - p_t p_t^\top\big) f.\tag{5}$$

This depends only on aggregate statistics $f$ and $p_t$, not on expert outputs $\{E_j(x_t)\}$ or counterfactual cross-entropies $\{\mathcal{L}_t(S)\}$. The same property holds for GShard, Importance/Load losses, and auxiliary-loss-free bias updates.

---

Methodology

Analysis Protocol

1. Models: Four open-weight MoE models are analyzed:

2. Trajectories: Verified-correct reasoning trajectories are generated per problem (MATH Level-5, AIME, HMMT, GPQA-Diamond), restricted to assistant-response tokens.

3. Alternative route sampling: For token $t$ at layer $\ell$, a candidate pool $P_{t,\ell}$ of the $m=32$ highest-scoring experts is formed. Then $G=32$ equal-compute alternatives are drawn via Gumbel-top-k sampling: $S_{t,\ell}^{(g)} = \text{TopK}_{i \in P_{t,\ell}}(s_{t,\ell,i} + \epsilon^{(g)}, k)$ with $\epsilon^{(g)} \sim \text{Gumbel}(0,1)^m$.

4. Intervention: For each candidate route $S$, the standard route is replaced at position $(t,\ell)$, gate weights are recomputed, and the forward pass runs with all other parameters fixed.

5. Metrics:

   • $p_{t,\ell}(S)$: next-token probability assigned to the realized token under route $S$
   • Top-K rates: fraction of tokens where the standard route ranks within top K of the 33 candidates
   • Confidence stratification by route-averaged probability $\bar{p}_{t,\ell}$ into Confident ($>0.9$), Ambiguous ($0.5$–$0.9$), and Fragile ($\leq 0.5$)

Expert Preference Optimization (EPO)

For the router-only intervention, EPO updates only the final-layer router:

• Hard token filter: Tokens with next-token cross-entropy exceeding $\tau = 0.1$ under the trainable router
• Preference construction: For each hard token, sample $G$ alternative routes; if the lowest-CE sample improves over the current route, use it as the chosen route $r_t^+$ (rejected route is $r_t^-$)
• Objective (CE-gap-weighted DPO-style):

$$\ell_t = -\Delta_t \log \sigma \left(\beta \log \frac{\pi_\theta(r_t^+ \mid x_t)}{\pi_{\text{ref}}(r_t^+ \mid x_t)} - \beta \log \frac{\pi_\theta(r_t^- \mid x_t)}{\pi_{\text{ref}}(r_t^- \mid x_t)}\right),\tag{6}$$

where $\Delta_t = \max(0, \mathcal{L}(r_t^-) - \mathcal{L}(r_t^+))$ is the observed CE-gap weight. Since $r_t^+$ and $r_t^-$ have the same cardinality, shared experts cancel, and the update acts only on distinguishing experts.

---

Empirical Validation / Results

Core Pattern (Qwen3-30B-A3B, final MoE layer, MATH Level-5)

Key observations:

• On Confident tokens, all routes perform nearly perfectly—routing choice has little consequence
• On Fragile tokens, the standard route is best only 0.8% of the time, yet the best alternative improves probability by 20.4 pp
• Top-5 and Top-10 rates also collapse outside the Confident range

Consistency Across Layers (Qwen3-30B-A3B)

The Fragile-token gap grows monotonically with depth—consistent with errors at the final layer feeding directly into the next-token distribution with no further routing to correct them.

Consistency Across Domains and Models

• Benchmarks: AIME (Fragile gap: 19.4 pp), HMMT (19.0 pp), GPQA-Diamond (21.2 pp)—same qualitative pattern
• Models: GPT-OSS-20B (Fragile gap: 11.8 pp), DeepSeek-V2-Lite (24.9 pp), OLMoE-1B-7B (20.8 pp)—pattern holds across all, with varying absolute levels

Router-Only Update Results (EPO)

Pass@K on AIME 2024+2025 and HMMT 2025 shifts above the standard curve for both Qwen3-30B-A3B and GPT-OSS-20B, with 95% bootstrap bands separating over part of the reported range. Table 5 decomposes the effect:

Interpretation:

• EPO shifts token distribution toward easier bins (Confident fraction increases, Fragile decreases)
• Route-rank gains occur mainly outside the Confident bin
• The Confident-bin rank drop is not a meaningful probability loss ($p_{\text{std}} \approx 99.5$, gap stays at 0.3 pp)—EPO reduces easy-token-biased rank alignment while improving alignment where route choice matters

---

Theoretical and Practical Implications

Theoretical Significance

1. First counterfactual evaluation of routing quality: The paper provides the first systematic framework for asking whether trained routers select good routes, independent of downstream performance.

2. Formal characterization of the blind spot: Equation (3) rigorously demonstrates why standard MoE training cannot optimize routing decisions beyond the executed route—the gradient vanishes for unselected experts.

3. Separation of capacity vs. routing failure: The EPO experiment shows that high loss on hard tokens is partially attributable to router-reachable misallocation rather than pure expert capacity limits.

4. Token-conditional nature of routing quality: The finding that routing quality degrades precisely on the tokens where it matters most (fragile, hard-reasoning tokens) suggests that aggregate routing metrics systematically hide the most consequential failures.

Practical Implications

1. Routing as a first-class training target: Routing quality on hard tokens warrants explicit attention in MoE training objectives, not just load-balancing and stability regularizers.

2. Potential for counterfactual-aware pretraining: The authors suggest designing pretraining objectives that place token-level signal on equal-compute alternative routes—e.g., lightweight evaluation of sampled alternatives alongside the executed route, incorporating their cross-entropies into the loss.

3. Router-only adaptation as a lightweight intervention: EPO demonstrates that meaningful shifts in reasoning performance can be achieved by updating <0.001% of parameters, suggesting cost-effective post-training routing improvements.

4. Diagnostic value: The confidence-stratified routing analysis provides a new diagnostic tool for evaluating MoE model quality beyond aggregate metrics.

---

Conclusion

Main Takeaways

1. The standard top-k router in MoE language models is well-aligned with route utility on confident tokens but uninformative on fragile tokens that drive hard reasoning.
2. This pattern is structurally inevitable given how standard MoE training evaluates routing: the LM loss scores only the executed route, and load balancing depends only on aggregate statistics—neither provides token-level counterfactual signal.
3. A minimal router-only update (EPO, <0.001% of parameters) shifts pass@K on mathematical reasoning benchmarks, proving that part of the failure reflects router-reachable misallocation rather than expert capacity limits.
4. The pattern is consistent across four MoE families, layers, and benchmarks, pointing to a shared cause independent of specific router training.

Future Directions

• Counterfactual-aware pretraining objectives that evaluate sampled alternative routes alongside the executed one
• Joint multi-layer routing interventions (the current analysis holds other layers fixed; whether per-layer failures compound or cancel is open)
• Refined proxies for routing quality beyond verified-correct trajectory probability (the current proxy may overstate improvement room on tokens where the realized token was a fortunate sample)
• Scaling counterfactual routing evaluation to pretraining scale without sacrificing sparse computation efficiency

Limitations

• Each layer's routing is analyzed independently; joint multi-layer modification is not addressed
• Analysis is restricted to verified-correct trajectories, where the realized next token already lies on a successful path
• The probability proxy may overstate the room for improvement on tokens where the realized token was itself a fortunate sample