Summary (Overview)

• Core finding: Across 60 metric-layer combinations in three high-redundancy MoE architectures (OLMoE-1B-7B-0924, Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite), no observational routing metric (utilization rate, activation norm, mean routing weight when active, activation std) predicts causal expert importance at the token level.

• Effect sizes: All observational metric effects stay below Cohen's $d = 0.23$, with inconsistent signs across layers and no metric reaching Bonferroni-corrected significance under a dual-test (paired t-test + Wilcoxon) criterion.

• Control experiment: A per-token routing weight control with identical $n$ recovers one significant signal at OLMoE's final MoE layer ($d = +0.231$, 95% CI [+0.09, +0.37], $p = 0.0013$), ruling out insufficient statistical power as an explanation for the null.

• Mechanistic explanation: Progressive ablation confirms that pruning methods succeed not by identifying dispensable experts, but because early-layer redundancy (removing 7 of 8 active experts at layers 0–9 causes <0.083 nats loss change) makes most selection criteria interchangeable.

• Theoretical contribution: The paper provides an explicit counterexample to the common inferential step from population-level observational summaries (Pearl's rung 1) to token-level interventional claims (rung 2), complementing prior failures in attention weights (Jain & Wallace, 2019) and saliency maps (Adebayo et al., 2020).

---

Introduction and Theoretical Foundation

The Observation-to-Intervention Problem

Interpretability methods routinely treat statistics computed over observed model behaviour as predictors of what targeted interventions would do. The paper identifies this as a general pattern with three concrete instances:

• Attention weights read as explanations of individual predictions (shown invalid by Jain & Wallace, 2019)
• Gradient saliency read as identifying input features whose removal changes output (shown invalid by Adebayo et al., 2020)
• Routing statistics in MoE models read as identifying experts whose ablation leaves the model intact (tested here)

The paper formalizes this via Pearl's causal hierarchy (Bareinboim et al., 2022): observational summaries are rung-1 associational evidence, but pruning claims require rung-2 interventional conclusions. The specific failure mode examined is the move from $\mathbb{E}_t[m_t(e)]$ (a population-level summary over observed routing behaviour) to $\Delta \mathcal{L}_e^{(t)}$ (loss change under intervention at a specific token).

The MoE Pruning Assumption

The pruning literature implicitly assumes that experts routed to more frequently, or more selectively, do more important work. This supports the standard pipeline:

"rank experts by an observational criterion, remove the lowest-ranked, recover capability through fine-tuning" (Chen et al., 2022; Muzio et al., 2024; Jaiswal et al., 2025)

The interventional question — whether observational rankings predict which expert ablations actually change model behaviour at individual token positions — had not been tested in any of the three architectures prior to this work.

---

Methodology

MoE Layer Formalization

A MoE layer replaces a dense feedforward block with $N$ experts $\{E_i\}_{i=1}^N$ and a router $G$. For hidden state $x$ at token position $t$, the layer output is:

$$\mathbf{y}_t = \sum_{i \in \mathrm{top-}k} g_i(\mathbf{x}_t) \cdot E_i(\mathbf{x}_t),\tag{1}$$

where $g_i(\mathbf{x}_t) = \mathrm{softmax}(W_g \mathbf{x}_t)_i$ is the routing weight.

Functional Importance Definition

For expert $i$ active at token $t$, the causal contribution is measured through ablation. Let $\mathbf{y}_t^{(-i)}$ denote the layer output with $E_i$ replaced by zero. The functional importance is:

$$\Delta \mathcal{L}_i^{(t)} = \mathcal{L}_t^{(-i)} - \mathcal{L}_t$$

where $\mathcal{L}_t = -\log p_\theta(x_{t+1} \mid x_{\le t})$. The gap norm is also reported:

$$\delta_i^{(t)} = \|\mathbf{y}_t - \mathbf{y}_t^{(-i)}\|_2$$

Metric Validity (Definition 2.1)

An observational metric $m : \mathcal{E} \to \mathbb{R}$ is causally valid at the token level if, for token position $t$ with active expert set $\mathcal{A}_t$, higher $m(e)$ among $e \in \mathcal{A}_t$ predicts larger functional importance $\Delta \mathcal{L}_e^{(t)}$.

Audit Protocol

• Sample size: $n = 200$ token positions per metric-layer-model cell
• Procedure: At each position, identify the active routed-expert set, rank by target metric, ablate the highest-ranked expert (record $\Delta \mathcal{L}_{\mathrm{high}}$), then ablate the lowest-ranked (record $\Delta \mathcal{L}_{\mathrm{low}}$)
• Validity test: The paired difference $\Delta \mathcal{L}_{\mathrm{high}} - \Delta \mathcal{L}_{\mathrm{low}}$ should be reliably positive
• Statistical tests: Paired t-test (Cohen's $d$ with 95% CI) plus Wilcoxon signed-rank test; cells where t and Wilcoxon disagree are classified as noise
• Multiple comparison correction: Bonferroni per model — $\alpha_{\mathrm{adj}} = 0.05/20 = 0.0025$ for the audit, $\alpha_{\mathrm{adj}} = 0.05/5 = 0.01$ for the routing weight control

Models and Data

Audited layers (proportional to depth): OLMoE: L0, L4, L7, L11, L15; Qwen: L0, L6, L12, L18, L23; DeepSeek: L1, L7, L13, L20, L26

Evaluation corpus: WikiText-2 test split (Merity et al., 2016)

---

Empirical Validation / Results

The Observational Null Replicates in All Three Models

Across all 60 metric-layer cells (20 per model):

• No observational metric reaches corrected significance under the dual-test criterion
• Effect sizes stay below Cohen's $d = 0.23$ throughout
• Signs are inconsistent across layers within each metric — the pattern of null distributions, not weak true effects

Key per-model findings:

Activation norm (identified by Jaiswal et al., 2025 as the strongest of 16 criteria) shows $|d| \leq 0.157$ at every tested layer in every model — never reaching Bonferroni significance.

The Routing Weight Control

Applied with identical machinery, the per-token routing weight ranking yields:

• OLMoE: Effect size grows monotonically with depth, reaching $d = +0.231$ at Layer 15 ($p = 0.0013$) — the only result in the experiment surviving Bonferroni correction; 95% CI [+0.09, +0.37] excludes zero
• Qwen: Stays within $|d| \leq 0.124$ across all layers
• DeepSeek: Stays within $|d| \leq 0.098$ across all layers

"The contrast with the observational cells is therefore one of where effects are centred, not of statistical power."

Progressive Ablation Confirms the Redundancy Regime

On OLMoE, removing the $k$ highest-weight active experts at a target layer ($n = 500$ token positions per layer-k cell, layers {0, 1, 7, 8, 9, 15}, $k \in \{1, \ldots, 7\}$):

Mechanism Underlying the OLMoE Late-Layer Effect

The Layer 15 signal coincides with a sharp gap norm scaling: mean residual-stream displacement under single-expert ablation grows from 0.0041 at Layer 0 to 0.1697 at Layer 15 — a 41× increase. However, Qwen shows comparable gap-norm scaling without OLMoE's late-layer functional concentration, ruling out gap-norm growth alone as sufficient explanation.

---

Theoretical and Practical Implications

What the Audit Shows (and Does Not)

• Does not show that metric-guided pruning fails as a deployment pipeline
• Shows that when pruning succeeds, the success is not attributable to the metric identifying experts that matter at the token level
• The redundancy regime explains consistency with literature's reported gains: any selection rule at early layers looks approximately harmless because nearly all selections are approximately harmless — "the metric and the random baseline are doing the same thing"

The Inferential Move

The paper frames the core issue within Pearl's causal hierarchy:

"The pruning literature treats $\mathbb{E}_t[m_t(e)]$, a summary over observed routing behaviour, as a predictor of $\Delta \mathcal{L}_e^{(t)}$, the loss change under intervention at a specific token. In the framing of Joshi et al. (2026), this is a move from rung 1 to rung 2 of the hierarchy."

The failure is uniform across metric families, layers, and architectures — the closest precedents being Jain & Wallace (2019) on attention and Adebayo et al. (2020) on saliency. The paper argues the pattern is not about attention, saliency, or routing specifically:

"It is about what kind of evidence licenses what kind of claim."

Scope Limitations

• Token-level audit: Tests one-expert ablation at one position; deployed pruning makes one-shot global decisions and recovers through fine-tuning
• Wide intervals: At $n = 200$ per cell, half-widths near 0.14 — the paper claims only that no observational metric produces a reliably positive token-level effect, not that effects are exactly zero
• High-redundancy regime: All three models have top-k activation rates between 6.7% and 12.5%; whether observational metrics regain validity in low-redundancy architectures (Switch-style, Mixtral-8x7B) remains open

---

Conclusion

The paper's main contributions:

1. A three-model null: No observational routing metric predicts causal expert importance at the token level after correction, across 60 metric-layer combinations spanning the major design dimensions of contemporary MoE literature
2. A single control signal: Per-token routing weight recovers one significant effect (OLMoE Layer 15, $d = +0.231$), demonstrating that the null is not a power artefact
3. A mechanistic explanation: Early-layer redundancy makes most selection criteria interchangeable, explaining why pruning pipelines succeed despite invalid per-token proxies
4. An explicit counterexample: Population-level observational summaries do not license token-level interventional claims in this regime, regardless of the statistic used

Future directions suggested:

• Bayesian model comparison (Bayes factors or information-criterion model selection) to quantify relative support for each criterion from token-level interventional data
• Controlled within-architecture follow-up on OLMoE to attribute the late-layer effect to specific training/architectural factors
• Testing whether observational metrics regain validity in low-redundancy architectures

The paper closes by situating its contribution:

"The pattern of summary statistics standing in for interventional evidence has appeared elsewhere in interpretability methodology, and the present null is one more case where the inferential move does not hold up to its own test."