# Understanding Mixture-of-Experts Language Models Beyond Expert Boundaries

> Neuron-level domain specialization in MoE models spans mixed experts, and RAMP-based pruning preserves 91-98% performance at 75% retention while cutting memory 36-43%.

- **Source:** [arXiv](https://arxiv.org/abs/2611.13975)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/VpD4Tu
- **Whiteboard:** https://picx.dev/p/VpD4Tu/image

## Summary

# Understanding Mixture-of-Experts Language Models Beyond Expert Boundaries

## Summary (Overview)

- **Neuron-level domain specialization**: The paper introduces Routing-Aware Magnitude Profiling (RAMP) to identify domain-specialized neurons in MoE models, showing 58–74% overlap within domains vs. 15–24% across domains at 6.25% selection, spanning mixed experts rather than aligning with expert boundaries.
- **Reconciliation of expert specialization debate**: Domain-enriched neurons concentrate in a subset of experts (13.39–23.26% of experts carry 66.40–83.40% of exclusive neurons), with enrichment positively correlated with routing preferences (median Spearman correlations 0.42–0.71), reconciling domain-associated routing with the absence of clear expert-level domain boundaries.
- **Training-time evolution findings**: Neuron importance continues to reorganize during pre-training (median rank correlations 0.842–0.939 between adjacent checkpoints) even when expert-level importance appears stable (0.916–0.990), while post-training largely preserves both levels (correlations >0.99).
- **RAMPr pruning method**: RAMP-based Pruning (RAMPr) preserves 91–98% of original performance at 75% neuron retention across six models, and at 50% retention reduces GPU memory by 36–43% with 17–39% prefill throughput improvement.
- **Practical advantage**: RAMPr-D outperforms expert-pruning baseline REAP by 2.91 percentage points on GLM-4.7-Flash at 75% retention, and exceeds Fisher pruning baselines in every model at 50% retention.

## Introduction and Theoretical Foundation

Mixture-of-Experts (MoE) architectures increase model capacity by routing each token through only a subset of experts (OpenAI et al., 2025; DeepSeek-AI et al., 2026). The design naturally fosters the intuition that each expert serves as a unit of computational specialization. However, contrasting evidence suggests routing decisions can be strongly tied to token identity and hidden-state geometry rather than domain-level expertise (Xue et al., 2024; Wang et al., 2026b).

The paper addresses a central open question: **to what extent is computational specialization in MoE models aligned with input domains?** Recent studies suggest that neurons within MoE experts contribute unequally to model computation (Cheng et al., 2025; He et al., 2026a), motivating a finer-grained view.

The theoretical foundation builds on two related lines of work:
- **Expert specialization research**: Architectural approaches organize conditional computation through expert granularity; analyses connect expert responses to input patterns, routing geometry, and router scores (Zoph et al., 2022; Dai et al., 2024).
- **Neuron and feature interpretability**: Neuron-level interpretations connect activations and parameters to input patterns through attribution and memory-based analyses (Geva et al., 2021; Dai et al., 2022a), with sparse dictionary learning decomposing activations into interpretable directions (Bricken et al., 2023).

## Methodology

### Routing-Aware Magnitude Profiling (RAMP)

RAMP measures each neuron's contribution magnitude weighted by its expert's routing coefficient. For an MoE layer with $E$ routed experts of width $N$, expert $e$ maps hidden state $x(t)$ to the gated activation vector:

$$
a^{e}(t) = \mathrm{SiLU}\bigl(W_{\mathrm{gate}}^{e}x(t)\bigr) \odot W_{\mathrm{up}}^{e}x(t).\tag{1}
$$

The routed branch output is:

$$
y^{\mathrm{routed}}(t) = \sum_{e=1}^{E} p^{e}(t) \sum_{i=1}^{N} a_{i}^{e}(t) w_{i}^{e,\mathrm{down}},\tag{2}
$$

where $p^{e}(t)$ is the routing weight for expert $e$ at token $t$ (zero if not selected). RAMP assigns each neuron $i$ in expert $e$ the importance score:

$$
I_{i}^{e}(D_{d}) = \left(\sum_{x \in D_{d}} \sum_{t \in x} [p^{e}(t) a_{i}^{e}(t)]^{2}\right)^{1/2} \left\|w_{i}^{e,\mathrm{down}}\right\|_{2}.\tag{3}
$$

### Experimental Design

**Models**: Six MoE models: Qwen3-30B-A3B, Qwen3.5-35B-A3B, Qwen3.6-35B-A3B, GLM-4.7-Flash, Qwen1.5-MoE-A2.7B-Chat, and OLMoE-1B-7B-0125-Instruct.

**Domain comparisons**: Calibration text from chat, mathematics, medicine, Chinese text, and code domains, with nine groups total (including task-format variations). Each group contains 128 documents, split equally into discovery and validation sets.

**Overlap metrics**: With neuron sets $T_{d}^{\mathrm{disc}}$ and $T_{d}^{\mathrm{val}}$ of size $k$:

$$
O_{\mathrm{same}}(d) = \frac{|T_{d}^{\mathrm{disc}} \cap T_{d}^{\mathrm{val}}|}{k}, \quad O_{\mathrm{cross}}(d, c) = \frac{|T_{d}^{\mathrm{disc}} \cap T_{c}^{\mathrm{disc}}|}{k}.\tag{4}
$$

### RAMP-based Pruning (RAMPr)

Two selection rules are proposed:

$$
S_{lei}^{L} = \sum_{d} I_{li}^{e}(D_{d}), \qquad S_{lei}^{D} = \max_{d} \frac{I_{li}^{e}(D_{d})}{Z_{l}(D_{d})},\tag{9}
$$

where $Z_{l}(D_{d}) = \sum_{e,i} I_{li}^{e}(D_{d})$ is the total importance of layer $l$ for domain $d$. **RAMPr-L** uses the summed rule; **RAMPr-D** takes the max after normalization.

**Training dynamics**: Comparisons across adjacent checkpoints use Spearman correlations for neuron-level rankings ($\rho_N$) and expert-aggregate rankings ($\rho_E$), with difference $\Delta\rho = \rho_E - \rho_N$ positive when experts are more stable.

## Empirical Validation / Results

### Domain-Specialized Neurons

At 1% selection, within-domain overlap is 50.79–68.07% vs. 8.47–12.64% cross-domain. At 6.25% selection, these reach 57.77–74.33% and 15.23–23.68%, respectively. Same-domain overlap exceeds cross-domain overlap in all 24 model–budget configurations.

**Task format**: Within-bucket overlap averages 67.31% (multiple-choice) and 60.93% (generation), versus 35.08% cross-format, confirming specialization reflects both content domain and task format.

### Enrichment Within Mixed Experts

**Table 1: Enrichment and validation routing at 6.25% selection**

| Model    | \|H_d\|/E (%) | M (%)  | ρ̃      | Δ_H (%) |
|----------|---------------|--------|---------|---------|
| Qwen3    | 16.18         | 83.11  | 0.6668  | 26.01   |
| Qwen3.5  | 13.61         | 83.02  | 0.5671  | 19.19   |
| Qwen3.6  | 13.39         | 83.40  | 0.5677  | 20.71   |
| GLM-4.7  | 18.12         | 83.29  | 0.6498  | 19.61   |
| Qwen1.5  | 23.26         | 66.40  | 0.3682  | 3.65    |
| OLMoE    | 17.47         | 72.43  | 0.6357  | 13.51   |

*Note: \|H_d\|/E is the share of enriched experts, M is their share of exclusive neurons, ρ̃ is the median Spearman correlation between F (enrichment) and Δr (routing preference), and Δ_H is the target-minus-other-domain dispatch-share difference.*

### Training Dynamics

**Pre-training**: Median $\rho_N$ is 0.9393 (OLMoE) and 0.8418 (OpenMoE), below expert medians of 0.9898 and 0.9164. Expert rankings more stable in 99.43% and 76.14% of cells respectively. Crucially, 209 of 240 OLMoE transitions show neuron reorganization even when routing shifts ≤5%, with within-expert redistribution the dominant component (median norm share 0.7830).

**Post-training**: Correlations exceed 0.99 at both levels. The largest change concentrates at Base-to-SFT (neuron medians 0.9715 and 0.8531), with later stages adding no systematic reorganization.

### Pruning Results

**Table 2: Mean performance retention (%) relative to original model**

| Model    | RAMPr-L (50/75/90) | RAMPr-D (50/75/90) |
|----------|---------------------|---------------------|
| Qwen3    | 61.04 / 92.64 / 97.44 | 44.47 / 93.99 / 98.74 |
| Qwen3.5  | 83.05 / 96.82 / 98.52 | 77.53 / 98.20 / 99.54 |
| Qwen3.6  | 83.83 / 97.63 / 98.66 | 80.01 / 97.63 / 99.15 |
| GLM-4.7  | 66.16 / 91.12 / 100.77 | 67.33 / 95.61 / 100.67 |
| Qwen1.5  | 88.47 / 93.93 / 95.26 | 85.73 / 92.95 / 103.60 |
| OLMoE    | 83.42 / 95.30 / 95.62 | 80.08 / 89.91 / 97.16 |

**Table 3: 11-task macro-average against baselines at 50% retention**

| Model    | Base  | F-G  | F-M  | REAP | RAMPr-L | RAMPr-D |
|----------|-------|------|------|------|---------|---------|
| Qwen3    | 78.45 | 29.25| 29.37| 60.77| 52.66   | 40.18   |
| Qwen3.5  | 85.20 | 49.75| 56.34| -    | 76.12   | 74.21   |
| Qwen3.6  | 84.10 | 51.65| 58.26| -    | 75.23   | 75.71   |
| GLM-4.7  | 74.70 | 29.87| 31.01| -    | 49.05   | 52.11   |
| Qwen1.5  | 45.92 | 32.59| 33.76| -    | 39.60   | 39.15   |
| OLMoE    | 46.37 | 26.89| 30.60| -    | 38.29   | 38.05   |
| GLM-4.7 (75%) | 74.70 | - | - | 68.99 | 67.86 | 71.90 |

**Runtime efficiency** (at 50% retention, batch 32, 2048 tokens): Resident memory decreases by 36–43%, prefill throughput improves by 17–39% (mean 1.27× speedup), and decode speedup averages 1.01×. Active routed-FFN parameters per token are cut by 32.29–49.43%.

## Theoretical and Practical Implications

### Theoretical Implications

1. **Resolving the expert specialization debate**: The findings reconcile conflicting views by showing that domain-specialized computation exists at the neuron level but spans mixed experts. Experts serve several domains yet are preferentially routed to domains whose neurons they disproportionately hold. This explains both domain-associated routing observations (supporting specialization) and the absence of clear expert-level domain boundaries (supporting token-identity accounts).

2. **Neuron-level reorganization during pre-training**: Expert-level stability masks continued neuron-level reorganization, meaning interpretations based solely on expert routing dynamics provide an incomplete picture of how specialization evolves. The within-expert redistribution component dominates changes, connecting to the finding that domain specialization lies in *which neurons* are selected within experts.

3. **Shaping of specialization timing**: Neuron importance is shaped mainly during pre-training and supervised fine-tuning, while preference optimization (DPO) largely preserves inherited structure, identifying when domain specialization crystallizes.

### Practical Implications

1. **Efficient deployment**: RAMPr demonstrates that neuron-level pruning can preserve multi-domain capabilities while reducing memory (36–43%) and improving throughput (17–39%), making MoE deployment more cost-effective and energy-efficient.

2. **Calibration control**: The ablation study shows calibration text composition offers a practical lever for targeted capability recovery—e.g., calibrating Qwen3 on five generation domains instead of all eleven raised MBPP from 4.00 to 66.80 and HumanEval from 0.61 to 63.41 at 50% retention.

3. **Advantage over existing methods**: RAMPr preserves both QA and generation capabilities, whereas REAP preserves generation at the expense of QA tasks. RAMPr-D offers better knowledge-generation balance than expert pruning, directly enabling more balanced model compression.

## Conclusion

This paper demonstrates that looking beyond expert boundaries treats neurons as the unit at which MoE models organize domain specialization. The key findings are:

1. **Domain-specialized neurons** exist within and across experts, with 58–74% overlap within domains vs. 15–24% across domains, concentrated in a few experts yet recurring within them beyond expert-level count predictions.

2. **Training dynamics** show neuron importance reorganizes inside experts during pre-training even when expert-level views remain stable, and the organization largely settles after supervised fine-tuning.

3. **RAMP-based Pruning (RAMPr)** retains one shared multi-domain neuron set that preserves capability better than gradient-based pruning (Fisher) and competes with expert pruning (REAP) at 75–90% retention, while significantly reducing memory and computation at 50% retention.

The work establishes that expert-level analyses capture only part of how MoE language models specialize across domains, opening the door for neuron-level approaches to both interpret and optimize these increasingly important architectures. Future research could extend RAMP-based analysis to other MoE architectures, explore training-time interventions to shape neuron specialization, or investigate how calibration domain selection can further target pruning outcomes.

---

_Markdown view of https://picx.dev/p/VpD4Tu, served by PicX — AI-generated visual whiteboard summaries of research papers._