What MoE Routing Controls: Causal Interventions on Expert Trajectories
Authors: Mukund Athmaram, Manoj Rao, Liang Ma (AMD, Cornell University)
Summary (Overview)
- Path-level steering method: The authors introduce a novel intervention technique that forces tokens onto cross-layer expert paths in Mixture-of-Experts (MoE) models by adding a constant bias to router logits, without editing hidden states.
- Format vs. concept paths: Format paths (e.g., colon in clock times) change generated text dramatically (44× digit fraction increase over random paths), while concept paths (e.g., religion, money) shift next-token probabilities but leave generated text largely unchanged.
- Cross-lingual concept transfer: A path built from Chinese religion tokens raises English religion word probabilities by 7× on primary and 12× on held-out vocabularies, demonstrating concept-level (not token-level) encoding.
- Self-reinforcement mechanism: Formats persist in output because they reinforce themselves during generation (steering only the first few tokens starts a digit run), while concepts do not.
- Architecture dependence: Steering effectiveness tracks routing width across four MoE architectures, with wider routing (top-8) producing more distinct concept paths than narrower routing.
Introduction and Theoretical Foundation
Background
Mixture-of-Experts (MoE) models route each token to a small subset of experts per layer. At layer , a linear router scores each expert:
The top-k experts are selected with softmax weights:
Key Theoretical Foundations
- Polysemantic vs. monosemantic paths: Building on Ye et al. [2026], single experts are polysemantic while multi-layer paths are monosemantic, grouping tokens by semantic function across languages and surface forms (e.g., ":" follows different paths for type annotations vs. clock times).
- Router-logit intervention: Unlike prior work that perturbs hidden states (Turner et al. [2024], Zou et al. [2025]), this method directly biases router logits, changing only which experts run while each expert receives the same input—enabling causal attribution.
- Distinction from prior steering work: Fayyaz et al. [2026] steer individual experts for specific behaviors; this work steers full cross-layer paths built from a concept's own tokens.
- Debate on expert interpretability: Some work finds fine-grained specialists (Herbst et al. [2026]), while others argue specialization reflects hidden-state geometry (Wang et al. [2026]). This work tests directly whether routed experts causally change outputs.
Methodology
Models
- Primary model: Qwen3-30B-A3B (48 layers, 128 experts per layer, top-8 routing)
- Additional architectures: OLMoE (top-8, 64 experts), Qwen1.5-MoE (top-4, 60 experts), DeepSeek-V2-Lite (top-6, 64 experts)
Building Target Paths
For a concept with exemplar tokens , the target expert at each layer is the most common top-1 expert:
Steering Intervention
Add a constant bias to the target expert's router logit at each steered layer:
The target joins the selected set, and the router still chooses the other experts. Default settings: middle layers at .
Measuring Effects
Two readouts are used:
- Distribution readout: Next-token probability on target tokens (e.g., digits, religion words)
- Text readout: Generated text scoring (e.g., fraction of digit characters, fraction of concept words)
Controls:
- Random paths with the same bias (one expert drawn uniformly per layer)
- Other concepts' paths (to check specificity)
- Cross-lingual paths (built in Chinese, measured in English) to rule out token repetition effects
- Held-out word lists that played no part in building the path
Empirical Validation / Results
Format Steering Changes Generated Text
Moving a type-annotation ":" onto the clock-time path transforms type completions into digit strings:
| prompt | unsteered (writes a type) | steered to time path (writes digits) |
|---|---|---|
| values: | List[int] = [1, 2, 3 | 188108824081 |
| y: | float = f(3.0) | 040610000061 |
| total: | int = 0 | 120,3,4,5,,8 |
| result: | int = h(5) | 843.01 in,002 |
- Digit fraction rises from 0.087 to 0.792 (44× over random path's 0.018)
- Output entropy stays between 2.3 and 2.7 (no collapse)
Concept Steering Across Languages
- Religion tokens agree on the same expert far more often than generic tokens (mean purity 0.47 vs. 0.19)
- Chinese-built religion path raises English religion word probability to 7.1× the random path (12.4× on held-out vocabulary)
- Positive on 54–57 of 60 prompts; exceeds each of 30 different random paths
- Money steers at 3.6× random path
Distribution vs. Text Discrepancy
| Readout | Format (digits) | Concept (money) |
|---|---|---|
| Next-token probability | ✓ Strong effect | ✓ 3.6× random |
| Generated text | ✓ Changes | ✗ Stays near random (~0.01) |
Self-Reinforcement Analysis
- Steering only the first token raises digit fraction from 0.085 to 0.21
- Steering first four tokens raises it to 0.60 (vs. 0.79 for all tokens)
- Steering that starts after token 4 still reaches 0.41
Concept Steerability Ordering
| Concept | Steerability (vs. random) | Held-out effect |
|---|---|---|
| Religion | 20× | ✓ |
| Money | 6.3× | ✗ (near baseline) |
| Law | 2.4× | ✗ |
| Medicine, nature, food, travel, emotion, time | No reliable effect | — |
Negative findings on explanations:
- Distinctiveness (embedding-space isolation): Pearson (not predictive)
- Word frequency: Pearson (not predictive)
- Religion (strongest) has higher baseline probability than most concepts—opposite of rarity prediction
Where Steering Acts
- Middle layers (L16–31) show the strongest effect: clock-time probability 0.418 vs. 0.076 (early) and 0.088 (late)
- At , target is already top-1 at every layer, yet raising to still increases probability from 0.139 to 0.457—so the bias raises gate weight, not just selection
Expert Identity vs. Gate Weight
| Steered condition | Compared with | Primary | Held-out |
|---|---|---|---|
| Weight 0.42 (delivered 0.55) | Native | 3.45× [1.44, 9.82], | 2.91× [0.76, 5.52], n.s. |
| Weight 0.42 (delivered 0.55) | Random path | 3.91×, | — |
| Weight 0.125 (delivered 0.27) | Native | 0.95× [0.49, 1.81] | — |
| Random path | Native | 0.88×, n.s. | — |
| Full steering | Frequency-matched | 3.7×, | 7.4×, |
| Full steering | Norm-matched | 6.6×, | 1.5×, |
Key finding: Expert identity sets the direction of the effect; weight sets its size. Frequency- and norm-matched random experts do not reproduce the effect.
Cross-Architecture Results
| Model | Routing | Distinct paths | Winner vote | Religion steering |
|---|---|---|---|---|
| Qwen3-30B-A3B (ref.) | top-8, 128 experts | 6/6 | 0.64 | 7.1× |
| OLMoE-1B-7B | top-8, 64 experts | 6/6 | 0.74 | 1.3× |
| Qwen1.5-MoE-A2.7B | top-4, 60 experts | 3/6 | 0.86 | 3.2× |
| DeepSeek-V2-Lite | top-6, 64 experts | 5/6 | 0.83 | 0.2× |
Weight preservation matters: OLMoE does not rescale weights to sum to one, so the same bias raises its total routed weight from ~0.4 to ~0.98. Preserving the native weight makes format steering work (digit fraction rises from 0.043 to 0.281).
Single-Expert Removal (Natural Use)
Removing one naturally selected target expert at one layer produced no reliable selective drop:
| Concept | Prompts | Measure | Target - control |
|---|---|---|---|
| Religion | 42 (norm-matched) | Diagnostic probability | -0.09 [-0.43, 0.24] |
| Religion | 42 (norm-matched) | Reference log-probability | -0.04 [-0.11, 0.02] |
| Religion | 64 | Diagnostic probability | -0.26 [-0.62, 0.04] |
| Time | 64 | Reference log-probability | 0.02 [-0.03, 0.08] |
Theoretical and Practical Implications
Theoretical Implications
- Causal role of routing: Expert paths demonstrably causally influence model outputs—routing is not merely epiphenomenal to hidden-state geometry.
- Path-level semantics: Multi-layer expert paths carry concept information that transfers across languages, supporting the monosemantic path view.
- Self-reinforcement as persistence mechanism: Whether a steered behavior appears in generated text depends on whether the target reinforces itself during autoregressive generation.
- Middle-layer concentration: Steering effects concentrate in middle layers, consistent with causal tracing of factual recall (Meng et al. [2023]).
Practical Implications
- Output format control: Expert paths offer a simple, interpretable handle on the form of MoE model outputs (e.g., forcing digit vs. code completions).
- Cross-lingual concept manipulation: Paths built in one language transfer to another, enabling language-agnostic concept steering.
- Architecture design guidance: Routing width (top-k) directly affects whether concepts have distinct paths—narrow routing may conflate concepts.
- Weight normalization matters: Cross-model steering comparisons must hold routed weight fixed; models with different normalization schemes behave differently.
Conclusion
Main Takeaways
- Promoting a data-derived target-expert trajectory causally changes MoE model behavior.
- Format paths change generated text; concept paths shift next-token distributions but not text.
- The difference follows self-reinforcement during generation: formats reinforce themselves, concepts do not.
- Religion and money steer most strongly—an ordering not explained by distinctiveness or word frequency.
- Expert identity matters beyond gate weight, activation frequency, or output norm.
- Steering acts in the middle layers.
- Across architectures, steering depends on whether a concept has its own path (routing width).
Future Directions
- Testing whole-path necessity during natural use (single-expert removal was inconclusive)
- Explaining which concepts steer and why (distinctiveness and frequency do not account for the ordering)
- Investigating whether routing mediates the model's natural semantic computation
- Exploring the interaction between gate weight and expert identity more precisely
- Extending to larger and more diverse MoE architectures with controlled comparisons
Open Questions
"Whether routing also mediates the model's natural semantic computation is an open question."
The authors suggest that expert paths could serve as a simple handle on the form of an MoE model's output and as a tool for studying how these models organize concepts.
Related papers
- UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute
UpLoop trains looped Transformers with identical parameters, tokens, and FLOPs to dense baselines, improving reasoning and language modeling across scales via progressive depth growth and shared-weight looping.
- BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
BuggyMLE generates 6,600+ bug-injected ML repositories and a structural complexity metric (CCI) predicting solve rates with R²=0.951, enabling difficulty-graded benchmarks where a 9B model rivals frontier agents.
- InfuseLoop: Cross-Trajectory Latent Injection for Scalable Parallel Looped Transformers
InfuseLoop injects cross-trajectory latent states into parallel looped transformers, eliminating training spikes and enabling quality gains at 8 loops with minimal overhead.