What MoE Routing Controls: Causal Interventions on Expert Trajectories

Authors: Mukund Athmaram, Manoj Rao, Liang Ma (AMD, Cornell University)


Summary (Overview)

  • Path-level steering method: The authors introduce a novel intervention technique that forces tokens onto cross-layer expert paths in Mixture-of-Experts (MoE) models by adding a constant bias to router logits, without editing hidden states.
  • Format vs. concept paths: Format paths (e.g., colon in clock times) change generated text dramatically (44× digit fraction increase over random paths), while concept paths (e.g., religion, money) shift next-token probabilities but leave generated text largely unchanged.
  • Cross-lingual concept transfer: A path built from Chinese religion tokens raises English religion word probabilities by 7× on primary and 12× on held-out vocabularies, demonstrating concept-level (not token-level) encoding.
  • Self-reinforcement mechanism: Formats persist in output because they reinforce themselves during generation (steering only the first few tokens starts a digit run), while concepts do not.
  • Architecture dependence: Steering effectiveness tracks routing width across four MoE architectures, with wider routing (top-8) producing more distinct concept paths than narrower routing.

Introduction and Theoretical Foundation

Background

Mixture-of-Experts (MoE) models route each token to a small subset of experts per layer. At layer ℓ\ell, a linear router scores each expert:

rℓ=Wℓghℓr_{\ell} = W_{\ell}^{g} h_{\ell}

The top-k experts are selected with softmax weights:

Eℓ=top-kj rℓ,j,gℓ,j=erℓ,j∑j′∈Eℓerℓ,j′,yℓ=∑j∈Eℓgℓ,j fℓ,j(hℓ)\mathcal{E}_{\ell} = \underset{j}{\text{top-k}} \, r_{\ell,j}, \qquad g_{\ell,j} = \frac{e^{r_{\ell,j}}}{\sum_{j' \in \mathcal{E}_{\ell}} e^{r_{\ell,j'}}}, \qquad y_{\ell} = \sum_{j \in \mathcal{E}_{\ell}} g_{\ell,j} \, f_{\ell,j}(h_{\ell})

Key Theoretical Foundations

  • Polysemantic vs. monosemantic paths: Building on Ye et al. [2026], single experts are polysemantic while multi-layer paths are monosemantic, grouping tokens by semantic function across languages and surface forms (e.g., ":" follows different paths for type annotations vs. clock times).
  • Router-logit intervention: Unlike prior work that perturbs hidden states (Turner et al. [2024], Zou et al. [2025]), this method directly biases router logits, changing only which experts run while each expert receives the same input—enabling causal attribution.
  • Distinction from prior steering work: Fayyaz et al. [2026] steer individual experts for specific behaviors; this work steers full cross-layer paths built from a concept's own tokens.
  • Debate on expert interpretability: Some work finds fine-grained specialists (Herbst et al. [2026]), while others argue specialization reflects hidden-state geometry (Wang et al. [2026]). This work tests directly whether routed experts causally change outputs.

Methodology

Models

  • Primary model: Qwen3-30B-A3B (48 layers, 128 experts per layer, top-8 routing)
  • Additional architectures: OLMoE (top-8, 64 experts), Qwen1.5-MoE (top-4, 60 experts), DeepSeek-V2-Lite (top-6, 64 experts)

Building Target Paths

For a concept cc with exemplar tokens Wc\mathcal{W}_c, the target expert at each layer is the most common top-1 expert:

jℓ⋆(c)=arg⁡max⁡j ∣{w∈Wc:arg⁡max⁡j′ rℓ,j′(w)=j}∣j_{\ell}^{\star}(c) = \underset{j}{\arg\max} \, \left| \{ w \in \mathcal{W}_c : \underset{j'}{\arg\max} \, r_{\ell,j'}(w) = j \} \right|

Steering Intervention

Add a constant bias ss to the target expert's router logit at each steered layer:

r~ℓ=rℓ+s⋅1jℓ⋆\tilde{r}_{\ell} = r_{\ell} + s \cdot \mathbf{1}_{j_{\ell}^{\star}}

The target joins the selected set, and the router still chooses the other k−1k-1 experts. Default settings: middle layers ℓ∈[16,31]\ell \in [16, 31] at s=8s = 8.

Measuring Effects

Two readouts are used:

  1. Distribution readout: Next-token probability on target tokens (e.g., digits, religion words)
  2. Text readout: Generated text scoring (e.g., fraction of digit characters, fraction of concept words)

Controls:

  • Random paths with the same bias (one expert drawn uniformly per layer)
  • Other concepts' paths (to check specificity)
  • Cross-lingual paths (built in Chinese, measured in English) to rule out token repetition effects
  • Held-out word lists that played no part in building the path

Empirical Validation / Results

Format Steering Changes Generated Text

Moving a type-annotation ":" onto the clock-time path transforms type completions into digit strings:

promptunsteered (writes a type)steered to time path (writes digits)
values:List[int] = [1, 2, 3188108824081
y:float = f(3.0)040610000061
total:int = 0120,3,4,5,,8
result:int = h(5)843.01 in,002
  • Digit fraction rises from 0.087 to 0.792 (44× over random path's 0.018)
  • Output entropy stays between 2.3 and 2.7 (no collapse)

Concept Steering Across Languages

  • Religion tokens agree on the same expert far more often than generic tokens (mean purity 0.47 vs. 0.19)
  • Chinese-built religion path raises English religion word probability to 7.1× the random path (12.4× on held-out vocabulary)
  • Positive on 54–57 of 60 prompts; exceeds each of 30 different random paths
  • Money steers at 3.6× random path

Distribution vs. Text Discrepancy

ReadoutFormat (digits)Concept (money)
Next-token probability✓ Strong effect✓ 3.6× random
Generated text✓ Changes✗ Stays near random (~0.01)

Self-Reinforcement Analysis

  • Steering only the first token raises digit fraction from 0.085 to 0.21
  • Steering first four tokens raises it to 0.60 (vs. 0.79 for all tokens)
  • Steering that starts after token 4 still reaches 0.41

Concept Steerability Ordering

ConceptSteerability (vs. random)Held-out effect
Religion20×✓
Money6.3×✗ (near baseline)
Law2.4×✗
Medicine, nature, food, travel, emotion, timeNo reliable effect—

Negative findings on explanations:

  • Distinctiveness (embedding-space isolation): Pearson r=+0.11r = +0.11 (not predictive)
  • Word frequency: Pearson r=+0.11r = +0.11 (not predictive)
  • Religion (strongest) has higher baseline probability than most concepts—opposite of rarity prediction

Where Steering Acts

  • Middle layers (L16–31) show the strongest effect: clock-time probability 0.418 vs. 0.076 (early) and 0.088 (late)
  • At s=6s = 6, target is already top-1 at every layer, yet raising to s=8s = 8 still increases probability from 0.139 to 0.457—so the bias raises gate weight, not just selection

Expert Identity vs. Gate Weight

Steered conditionCompared withPrimaryHeld-out
Weight 0.42 (delivered 0.55)Native3.45× [1.44, 9.82], p=0.003p=0.0032.91× [0.76, 5.52], n.s.
Weight 0.42 (delivered 0.55)Random path3.91×, p=0.04p=0.04—
Weight 0.125 (delivered 0.27)Native0.95× [0.49, 1.81]—
Random pathNative0.88×, n.s.—
Full steeringFrequency-matched3.7×, p<10−4p < 10^{-4}7.4×, p<10−4p < 10^{-4}
Full steeringNorm-matched6.6×, p<10−4p < 10^{-4}1.5×, p=0.005p = 0.005

Key finding: Expert identity sets the direction of the effect; weight sets its size. Frequency- and norm-matched random experts do not reproduce the effect.

Cross-Architecture Results

ModelRoutingDistinct pathsWinner voteReligion steering
Qwen3-30B-A3B (ref.)top-8, 128 experts6/60.647.1×
OLMoE-1B-7Btop-8, 64 experts6/60.741.3×
Qwen1.5-MoE-A2.7Btop-4, 60 experts3/60.863.2×
DeepSeek-V2-Litetop-6, 64 experts5/60.830.2×

Weight preservation matters: OLMoE does not rescale weights to sum to one, so the same bias raises its total routed weight from ~0.4 to ~0.98. Preserving the native weight makes format steering work (digit fraction rises from 0.043 to 0.281).

Single-Expert Removal (Natural Use)

Removing one naturally selected target expert at one layer produced no reliable selective drop:

ConceptPromptsMeasureTarget - control
Religion42 (norm-matched)Diagnostic probability-0.09 [-0.43, 0.24]
Religion42 (norm-matched)Reference log-probability-0.04 [-0.11, 0.02]
Religion64Diagnostic probability-0.26 [-0.62, 0.04]
Time64Reference log-probability0.02 [-0.03, 0.08]

Theoretical and Practical Implications

Theoretical Implications

  1. Causal role of routing: Expert paths demonstrably causally influence model outputs—routing is not merely epiphenomenal to hidden-state geometry.
  2. Path-level semantics: Multi-layer expert paths carry concept information that transfers across languages, supporting the monosemantic path view.
  3. Self-reinforcement as persistence mechanism: Whether a steered behavior appears in generated text depends on whether the target reinforces itself during autoregressive generation.
  4. Middle-layer concentration: Steering effects concentrate in middle layers, consistent with causal tracing of factual recall (Meng et al. [2023]).

Practical Implications

  1. Output format control: Expert paths offer a simple, interpretable handle on the form of MoE model outputs (e.g., forcing digit vs. code completions).
  2. Cross-lingual concept manipulation: Paths built in one language transfer to another, enabling language-agnostic concept steering.
  3. Architecture design guidance: Routing width (top-k) directly affects whether concepts have distinct paths—narrow routing may conflate concepts.
  4. Weight normalization matters: Cross-model steering comparisons must hold routed weight fixed; models with different normalization schemes behave differently.

Conclusion

Main Takeaways

  • Promoting a data-derived target-expert trajectory causally changes MoE model behavior.
  • Format paths change generated text; concept paths shift next-token distributions but not text.
  • The difference follows self-reinforcement during generation: formats reinforce themselves, concepts do not.
  • Religion and money steer most strongly—an ordering not explained by distinctiveness or word frequency.
  • Expert identity matters beyond gate weight, activation frequency, or output norm.
  • Steering acts in the middle layers.
  • Across architectures, steering depends on whether a concept has its own path (routing width).

Future Directions

  • Testing whole-path necessity during natural use (single-expert removal was inconclusive)
  • Explaining which concepts steer and why (distinctiveness and frequency do not account for the ordering)
  • Investigating whether routing mediates the model's natural semantic computation
  • Exploring the interaction between gate weight and expert identity more precisely
  • Extending to larger and more diverse MoE architectures with controlled comparisons

Open Questions

"Whether routing also mediates the model's natural semantic computation is an open question."

The authors suggest that expert paths could serve as a simple handle on the form of an MoE model's output and as a tool for studying how these models organize concepts.

Related papers