# What MoE Routing Controls: Causal Interventions on Expert Trajectories

> Adding a constant bias to router logits to force tokens onto cross-layer expert paths causally steers MoE output format, but concept paths shift probabilities without changing generated text.

- **Source:** [arXiv](https://arxiv.org/abs/2611.22120)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/CnreU0
- **Whiteboard:** https://picx.dev/p/CnreU0/image

## Summary

# What MoE Routing Controls: Causal Interventions on Expert Trajectories

**Authors:** Mukund Athmaram, Manoj Rao, Liang Ma (AMD, Cornell University)

---

## Summary (Overview)

- **Path-level steering method**: The authors introduce a novel intervention technique that forces tokens onto cross-layer expert paths in Mixture-of-Experts (MoE) models by adding a constant bias to router logits, without editing hidden states.
- **Format vs. concept paths**: Format paths (e.g., colon in clock times) change generated text dramatically (44× digit fraction increase over random paths), while concept paths (e.g., religion, money) shift next-token probabilities but leave generated text largely unchanged.
- **Cross-lingual concept transfer**: A path built from Chinese religion tokens raises English religion word probabilities by 7× on primary and 12× on held-out vocabularies, demonstrating concept-level (not token-level) encoding.
- **Self-reinforcement mechanism**: Formats persist in output because they reinforce themselves during generation (steering only the first few tokens starts a digit run), while concepts do not.
- **Architecture dependence**: Steering effectiveness tracks routing width across four MoE architectures, with wider routing (top-8) producing more distinct concept paths than narrower routing.

---

## Introduction and Theoretical Foundation

### Background

Mixture-of-Experts (MoE) models route each token to a small subset of experts per layer. At layer $\ell$, a linear router scores each expert:

$$r_{\ell} = W_{\ell}^{g} h_{\ell}$$

The top-k experts are selected with softmax weights:

$$\mathcal{E}_{\ell} = \underset{j}{\text{top-k}} \, r_{\ell,j}, \qquad g_{\ell,j} = \frac{e^{r_{\ell,j}}}{\sum_{j' \in \mathcal{E}_{\ell}} e^{r_{\ell,j'}}}, \qquad y_{\ell} = \sum_{j \in \mathcal{E}_{\ell}} g_{\ell,j} \, f_{\ell,j}(h_{\ell})$$

### Key Theoretical Foundations

- **Polysemantic vs. monosemantic paths**: Building on Ye et al. [2026], single experts are polysemantic while multi-layer paths are monosemantic, grouping tokens by semantic function across languages and surface forms (e.g., ":" follows different paths for type annotations vs. clock times).
- **Router-logit intervention**: Unlike prior work that perturbs hidden states (Turner et al. [2024], Zou et al. [2025]), this method directly biases router logits, changing only *which* experts run while each expert receives the same input—enabling causal attribution.
- **Distinction from prior steering work**: Fayyaz et al. [2026] steer individual experts for specific behaviors; this work steers full cross-layer paths built from a concept's own tokens.
- **Debate on expert interpretability**: Some work finds fine-grained specialists (Herbst et al. [2026]), while others argue specialization reflects hidden-state geometry (Wang et al. [2026]). This work tests directly whether routed experts causally change outputs.

---

## Methodology

### Models

- **Primary model**: Qwen3-30B-A3B (48 layers, 128 experts per layer, top-8 routing)
- **Additional architectures**: OLMoE (top-8, 64 experts), Qwen1.5-MoE (top-4, 60 experts), DeepSeek-V2-Lite (top-6, 64 experts)

### Building Target Paths

For a concept $c$ with exemplar tokens $\mathcal{W}_c$, the target expert at each layer is the most common top-1 expert:

$$j_{\ell}^{\star}(c) = \underset{j}{\arg\max} \, \left| \{ w \in \mathcal{W}_c : \underset{j'}{\arg\max} \, r_{\ell,j'}(w) = j \} \right|$$

### Steering Intervention

Add a constant bias $s$ to the target expert's router logit at each steered layer:

$$\tilde{r}_{\ell} = r_{\ell} + s \cdot \mathbf{1}_{j_{\ell}^{\star}}$$

The target joins the selected set, and the router still chooses the other $k-1$ experts. Default settings: middle layers $\ell \in [16, 31]$ at $s = 8$.

### Measuring Effects

**Two readouts** are used:
1. **Distribution readout**: Next-token probability on target tokens (e.g., digits, religion words)
2. **Text readout**: Generated text scoring (e.g., fraction of digit characters, fraction of concept words)

**Controls**:
- Random paths with the same bias (one expert drawn uniformly per layer)
- Other concepts' paths (to check specificity)
- Cross-lingual paths (built in Chinese, measured in English) to rule out token repetition effects
- Held-out word lists that played no part in building the path

---

## Empirical Validation / Results

### Format Steering Changes Generated Text

Moving a type-annotation ":" onto the clock-time path transforms type completions into digit strings:

| prompt | unsteered (writes a type) | steered to time path (writes digits) |
|--------|---------------------------|--------------------------------------|
| values: | List[int] = [1, 2, 3 | 188108824081 |
| y: | float = f(3.0) | 040610000061 |
| total: | int = 0 | 120,3,4,5,,8 |
| result: | int = h(5) | 843.01 in,002 |

- Digit fraction rises from 0.087 to 0.792 (44× over random path's 0.018)
- Output entropy stays between 2.3 and 2.7 (no collapse)

### Concept Steering Across Languages

- Religion tokens agree on the same expert far more often than generic tokens (mean purity 0.47 vs. 0.19)
- Chinese-built religion path raises English religion word probability to **7.1×** the random path (12.4× on held-out vocabulary)
- Positive on 54–57 of 60 prompts; exceeds each of 30 different random paths
- Money steers at 3.6× random path

### Distribution vs. Text Discrepancy

| Readout | Format (digits) | Concept (money) |
|---------|-----------------|-----------------|
| Next-token probability | ✓ Strong effect | ✓ 3.6× random |
| Generated text | ✓ Changes | ✗ Stays near random (~0.01) |

### Self-Reinforcement Analysis

- Steering only the **first token** raises digit fraction from 0.085 to 0.21
- Steering first **four tokens** raises it to 0.60 (vs. 0.79 for all tokens)
- Steering that starts after token 4 still reaches 0.41

### Concept Steerability Ordering

| Concept | Steerability (vs. random) | Held-out effect |
|---------|---------------------------|-----------------|
| Religion | 20× | ✓ |
| Money | 6.3× | ✗ (near baseline) |
| Law | 2.4× | ✗ |
| Medicine, nature, food, travel, emotion, time | No reliable effect | — |

**Negative findings on explanations**:
- Distinctiveness (embedding-space isolation): Pearson $r = +0.11$ (not predictive)
- Word frequency: Pearson $r = +0.11$ (not predictive)
- Religion (strongest) has *higher* baseline probability than most concepts—opposite of rarity prediction

### Where Steering Acts

- **Middle layers** (L16–31) show the strongest effect: clock-time probability 0.418 vs. 0.076 (early) and 0.088 (late)
- At $s = 6$, target is already top-1 at every layer, yet raising to $s = 8$ still increases probability from 0.139 to 0.457—so the bias raises gate weight, not just selection

### Expert Identity vs. Gate Weight

| Steered condition | Compared with | Primary | Held-out |
|-------------------|---------------|---------|----------|
| Weight 0.42 (delivered 0.55) | Native | 3.45× [1.44, 9.82], $p=0.003$ | 2.91× [0.76, 5.52], n.s. |
| Weight 0.42 (delivered 0.55) | Random path | 3.91×, $p=0.04$ | — |
| Weight 0.125 (delivered 0.27) | Native | 0.95× [0.49, 1.81] | — |
| Random path | Native | 0.88×, n.s. | — |
| Full steering | Frequency-matched | 3.7×, $p < 10^{-4}$ | 7.4×, $p < 10^{-4}$ |
| Full steering | Norm-matched | 6.6×, $p < 10^{-4}$ | 1.5×, $p = 0.005$ |

**Key finding**: Expert identity sets the direction of the effect; weight sets its size. Frequency- and norm-matched random experts do not reproduce the effect.

### Cross-Architecture Results

| Model | Routing | Distinct paths | Winner vote | Religion steering |
|-------|---------|----------------|-------------|-------------------|
| Qwen3-30B-A3B (ref.) | top-8, 128 experts | 6/6 | 0.64 | 7.1× |
| OLMoE-1B-7B | top-8, 64 experts | 6/6 | 0.74 | 1.3× |
| Qwen1.5-MoE-A2.7B | top-4, 60 experts | 3/6 | 0.86 | 3.2× |
| DeepSeek-V2-Lite | top-6, 64 experts | 5/6 | 0.83 | 0.2× |

**Weight preservation matters**: OLMoE does not rescale weights to sum to one, so the same bias raises its total routed weight from ~0.4 to ~0.98. Preserving the native weight makes format steering work (digit fraction rises from 0.043 to 0.281).

### Single-Expert Removal (Natural Use)

Removing one naturally selected target expert at one layer produced no reliable selective drop:

| Concept | Prompts | Measure | Target - control |
|---------|---------|---------|------------------|
| Religion | 42 (norm-matched) | Diagnostic probability | -0.09 [-0.43, 0.24] |
| Religion | 42 (norm-matched) | Reference log-probability | -0.04 [-0.11, 0.02] |
| Religion | 64 | Diagnostic probability | -0.26 [-0.62, 0.04] |
| Time | 64 | Reference log-probability | 0.02 [-0.03, 0.08] |

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Causal role of routing**: Expert paths demonstrably causally influence model outputs—routing is not merely epiphenomenal to hidden-state geometry.
2. **Path-level semantics**: Multi-layer expert paths carry concept information that transfers across languages, supporting the monosemantic path view.
3. **Self-reinforcement as persistence mechanism**: Whether a steered behavior appears in generated text depends on whether the target reinforces itself during autoregressive generation.
4. **Middle-layer concentration**: Steering effects concentrate in middle layers, consistent with causal tracing of factual recall (Meng et al. [2023]).

### Practical Implications

1. **Output format control**: Expert paths offer a simple, interpretable handle on the *form* of MoE model outputs (e.g., forcing digit vs. code completions).
2. **Cross-lingual concept manipulation**: Paths built in one language transfer to another, enabling language-agnostic concept steering.
3. **Architecture design guidance**: Routing width (top-k) directly affects whether concepts have distinct paths—narrow routing may conflate concepts.
4. **Weight normalization matters**: Cross-model steering comparisons must hold routed weight fixed; models with different normalization schemes behave differently.

---

## Conclusion

### Main Takeaways

- Promoting a data-derived target-expert trajectory **causally changes** MoE model behavior.
- **Format paths change generated text**; **concept paths shift next-token distributions** but not text.
- The difference follows **self-reinforcement** during generation: formats reinforce themselves, concepts do not.
- **Religion and money steer most strongly**—an ordering not explained by distinctiveness or word frequency.
- **Expert identity matters** beyond gate weight, activation frequency, or output norm.
- Steering acts in the **middle layers**.
- Across architectures, steering depends on whether a concept has its own path (routing width).

### Future Directions

- Testing **whole-path necessity** during natural use (single-expert removal was inconclusive)
- Explaining **which concepts steer** and why (distinctiveness and frequency do not account for the ordering)
- Investigating whether routing mediates the model's **natural semantic computation**
- Exploring the interaction between **gate weight and expert identity** more precisely
- Extending to **larger and more diverse** MoE architectures with controlled comparisons

### Open Questions

> "Whether routing also mediates the model's natural semantic computation is an open question."

The authors suggest that expert paths could serve as a simple handle on the form of an MoE model's output and as a tool for studying how these models organize concepts.

---

_Markdown view of https://picx.dev/p/CnreU0, served by PicX — AI-generated visual whiteboard summaries of research papers._
