Understanding Mixture-of-Experts Language Models Beyond Expert Boundaries

Summary (Overview)

• Neuron-level domain specialization: The paper introduces Routing-Aware Magnitude Profiling (RAMP) to identify domain-specialized neurons in MoE models, showing 58–74% overlap within domains vs. 15–24% across domains at 6.25% selection, spanning mixed experts rather than aligning with expert boundaries.
• Reconciliation of expert specialization debate: Domain-enriched neurons concentrate in a subset of experts (13.39–23.26% of experts carry 66.40–83.40% of exclusive neurons), with enrichment positively correlated with routing preferences (median Spearman correlations 0.42–0.71), reconciling domain-associated routing with the absence of clear expert-level domain boundaries.
• Training-time evolution findings: Neuron importance continues to reorganize during pre-training (median rank correlations 0.842–0.939 between adjacent checkpoints) even when expert-level importance appears stable (0.916–0.990), while post-training largely preserves both levels (correlations >0.99).
• RAMPr pruning method: RAMP-based Pruning (RAMPr) preserves 91–98% of original performance at 75% neuron retention across six models, and at 50% retention reduces GPU memory by 36–43% with 17–39% prefill throughput improvement.
• Practical advantage: RAMPr-D outperforms expert-pruning baseline REAP by 2.91 percentage points on GLM-4.7-Flash at 75% retention, and exceeds Fisher pruning baselines in every model at 50% retention.

Introduction and Theoretical Foundation

Mixture-of-Experts (MoE) architectures increase model capacity by routing each token through only a subset of experts (OpenAI et al., 2025; DeepSeek-AI et al., 2026). The design naturally fosters the intuition that each expert serves as a unit of computational specialization. However, contrasting evidence suggests routing decisions can be strongly tied to token identity and hidden-state geometry rather than domain-level expertise (Xue et al., 2024; Wang et al., 2026b).

The paper addresses a central open question: to what extent is computational specialization in MoE models aligned with input domains? Recent studies suggest that neurons within MoE experts contribute unequally to model computation (Cheng et al., 2025; He et al., 2026a), motivating a finer-grained view.

The theoretical foundation builds on two related lines of work:

• Expert specialization research: Architectural approaches organize conditional computation through expert granularity; analyses connect expert responses to input patterns, routing geometry, and router scores (Zoph et al., 2022; Dai et al., 2024).
• Neuron and feature interpretability: Neuron-level interpretations connect activations and parameters to input patterns through attribution and memory-based analyses (Geva et al., 2021; Dai et al., 2022a), with sparse dictionary learning decomposing activations into interpretable directions (Bricken et al., 2023).

Methodology

Routing-Aware Magnitude Profiling (RAMP)

RAMP measures each neuron's contribution magnitude weighted by its expert's routing coefficient. For an MoE layer with $E$ routed experts of width $N$, expert $e$ maps hidden state $x(t)$ to the gated activation vector:

$$a^{e}(t) = \mathrm{SiLU}\bigl(W_{\mathrm{gate}}^{e}x(t)\bigr) \odot W_{\mathrm{up}}^{e}x(t).\tag{1}$$

The routed branch output is:

$$y^{\mathrm{routed}}(t) = \sum_{e=1}^{E} p^{e}(t) \sum_{i=1}^{N} a_{i}^{e}(t) w_{i}^{e,\mathrm{down}},\tag{2}$$

where $p^{e}(t)$ is the routing weight for expert $e$ at token $t$ (zero if not selected). RAMP assigns each neuron $i$ in expert $e$ the importance score:

$$I_{i}^{e}(D_{d}) = \left(\sum_{x \in D_{d}} \sum_{t \in x} [p^{e}(t) a_{i}^{e}(t)]^{2}\right)^{1/2} \left\|w_{i}^{e,\mathrm{down}}\right\|_{2}.\tag{3}$$

Experimental Design

Models: Six MoE models: Qwen3-30B-A3B, Qwen3.5-35B-A3B, Qwen3.6-35B-A3B, GLM-4.7-Flash, Qwen1.5-MoE-A2.7B-Chat, and OLMoE-1B-7B-0125-Instruct.

Domain comparisons: Calibration text from chat, mathematics, medicine, Chinese text, and code domains, with nine groups total (including task-format variations). Each group contains 128 documents, split equally into discovery and validation sets.

Overlap metrics: With neuron sets $T_{d}^{\mathrm{disc}}$ and $T_{d}^{\mathrm{val}}$ of size $k$:

$$O_{\mathrm{same}}(d) = \frac{|T_{d}^{\mathrm{disc}} \cap T_{d}^{\mathrm{val}}|}{k}, \quad O_{\mathrm{cross}}(d, c) = \frac{|T_{d}^{\mathrm{disc}} \cap T_{c}^{\mathrm{disc}}|}{k}.\tag{4}$$

RAMP-based Pruning (RAMPr)

Two selection rules are proposed:

$$S_{lei}^{L} = \sum_{d} I_{li}^{e}(D_{d}), \qquad S_{lei}^{D} = \max_{d} \frac{I_{li}^{e}(D_{d})}{Z_{l}(D_{d})},\tag{9}$$

where $Z_{l}(D_{d}) = \sum_{e,i} I_{li}^{e}(D_{d})$ is the total importance of layer $l$ for domain $d$. RAMPr-L uses the summed rule; RAMPr-D takes the max after normalization.

Training dynamics: Comparisons across adjacent checkpoints use Spearman correlations for neuron-level rankings ($\rho_N$) and expert-aggregate rankings ($\rho_E$), with difference $\Delta\rho = \rho_E - \rho_N$ positive when experts are more stable.

Empirical Validation / Results

Domain-Specialized Neurons

At 1% selection, within-domain overlap is 50.79–68.07% vs. 8.47–12.64% cross-domain. At 6.25% selection, these reach 57.77–74.33% and 15.23–23.68%, respectively. Same-domain overlap exceeds cross-domain overlap in all 24 model–budget configurations.

Task format: Within-bucket overlap averages 67.31% (multiple-choice) and 60.93% (generation), versus 35.08% cross-format, confirming specialization reflects both content domain and task format.

Enrichment Within Mixed Experts

Table 1: Enrichment and validation routing at 6.25% selection

Note: |H_d|/E is the share of enriched experts, M is their share of exclusive neurons, ρ̃ is the median Spearman correlation between F (enrichment) and Δr (routing preference), and Δ_H is the target-minus-other-domain dispatch-share difference.

Training Dynamics

Pre-training: Median $\rho_N$ is 0.9393 (OLMoE) and 0.8418 (OpenMoE), below expert medians of 0.9898 and 0.9164. Expert rankings more stable in 99.43% and 76.14% of cells respectively. Crucially, 209 of 240 OLMoE transitions show neuron reorganization even when routing shifts ≤5%, with within-expert redistribution the dominant component (median norm share 0.7830).

Post-training: Correlations exceed 0.99 at both levels. The largest change concentrates at Base-to-SFT (neuron medians 0.9715 and 0.8531), with later stages adding no systematic reorganization.

Pruning Results

Table 2: Mean performance retention (%) relative to original model

Table 3: 11-task macro-average against baselines at 50% retention

Runtime efficiency (at 50% retention, batch 32, 2048 tokens): Resident memory decreases by 36–43%, prefill throughput improves by 17–39% (mean 1.27× speedup), and decode speedup averages 1.01×. Active routed-FFN parameters per token are cut by 32.29–49.43%.

Theoretical and Practical Implications

Theoretical Implications

1. Resolving the expert specialization debate: The findings reconcile conflicting views by showing that domain-specialized computation exists at the neuron level but spans mixed experts. Experts serve several domains yet are preferentially routed to domains whose neurons they disproportionately hold. This explains both domain-associated routing observations (supporting specialization) and the absence of clear expert-level domain boundaries (supporting token-identity accounts).

2. Neuron-level reorganization during pre-training: Expert-level stability masks continued neuron-level reorganization, meaning interpretations based solely on expert routing dynamics provide an incomplete picture of how specialization evolves. The within-expert redistribution component dominates changes, connecting to the finding that domain specialization lies in which neurons are selected within experts.

3. Shaping of specialization timing: Neuron importance is shaped mainly during pre-training and supervised fine-tuning, while preference optimization (DPO) largely preserves inherited structure, identifying when domain specialization crystallizes.

Practical Implications

1. Efficient deployment: RAMPr demonstrates that neuron-level pruning can preserve multi-domain capabilities while reducing memory (36–43%) and improving throughput (17–39%), making MoE deployment more cost-effective and energy-efficient.

2. Calibration control: The ablation study shows calibration text composition offers a practical lever for targeted capability recovery—e.g., calibrating Qwen3 on five generation domains instead of all eleven raised MBPP from 4.00 to 66.80 and HumanEval from 0.61 to 63.41 at 50% retention.

3. Advantage over existing methods: RAMPr preserves both QA and generation capabilities, whereas REAP preserves generation at the expense of QA tasks. RAMPr-D offers better knowledge-generation balance than expert pruning, directly enabling more balanced model compression.

Conclusion

This paper demonstrates that looking beyond expert boundaries treats neurons as the unit at which MoE models organize domain specialization. The key findings are:

1. Domain-specialized neurons exist within and across experts, with 58–74% overlap within domains vs. 15–24% across domains, concentrated in a few experts yet recurring within them beyond expert-level count predictions.

2. Training dynamics show neuron importance reorganizes inside experts during pre-training even when expert-level views remain stable, and the organization largely settles after supervised fine-tuning.

3. RAMP-based Pruning (RAMPr) retains one shared multi-domain neuron set that preserves capability better than gradient-based pruning (Fisher) and competes with expert pruning (REAP) at 75–90% retention, while significantly reducing memory and computation at 50% retention.

The work establishes that expert-level analyses capture only part of how MoE language models specialize across domains, opening the door for neuron-level approaches to both interpret and optimize these increasingly important architectures. Future research could extend RAMP-based analysis to other MoE architectures, explore training-time interventions to shape neuron specialization, or investigate how calibration domain selection can further target pruning outcomes.