# Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models

> Expert-aware causal tracing reveals factual recall localizes to a single expert in Qwen3 but distributes across expert coalitions in Mixtral, showing localization is architecture-dependent.

- **Source:** [arXiv](https://arxiv.org/abs/2606.03780)
- **Published:** 2026-08-29
- **Permalink:** https://picx.dev/p/pUGl34
- **Whiteboard:** https://picx.dev/p/pUGl34/image

## Summary

# Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models

**Authors:** Yuetian Lu, Ali Modarressi, Yihong Liu, Hinrich Schütze (LMU Munich, TU Darmstadt, MCML)

---

## Summary (Overview)

- **Novel protocol for expert-level causal tracing**: The paper introduces a two-stage expert-aware causal tracing protocol specifically designed for sparse Mixture-of-Experts (MoE) language models, moving beyond layer-level analysis to individual routed expert contributions.
- **Key finding on Qwen3-30B-A3B-BASE**: Layer-level tracing selects and validates layer 44, and expert-level tracing identifies expert **L44E069** as a recurrent expert whose held-out patch significantly outperforms other active same-layer experts (Specificity = +0.400).
- **Contrasting finding on Mixtral-8X7B-V0.1**: While layer 19 validates as a mid-layer signal, the selected singleton expert **L19E006** underperforms active controls (Specificity = -0.175); instead, routed multi-expert coalitions recover the layer-level signal.
- **Methodological contribution**: Distinguishes three levels of factual-recall evidence: aggregate MoE-block rescue, single-expert localization, and multi-expert routed recovery.
- **Key insight**: Expert-level localization is model- and protocol-dependent rather than universal across MoE architectures.

---

## Introduction and Theoretical Foundation

### Background

Factual recall in language models refers to the ability to assign high probability to correct objects given subject–relation prompts (e.g., "The capital of France is" → *Paris*). Causal tracing and activation patching are established techniques for understanding which internal components mediate this behavior by:

1. Corrupting a prediction (e.g., adding noise to embeddings)
2. Restoring internal components from a clean run
3. Measuring which components mediate recovery

### The MoE Challenge

Sparse MoE language models introduce a critical granularity problem: a token is processed not by a single dense feed-forward module but by a small routed subset of experts. This means:

- A layer-level patch shows that an MoE block matters
- But the patched block is an **aggregate** over routed expert updates
- It does not answer **which selected expert contributions** support the factual prediction

### Research Question

> Can factual-recall causal tracing be made expert-aware in sparse MoE language models?

### Theoretical Foundation

The paper builds on:
- **COUNTERFACT facts** (Meng et al., 2022) for controlled factual contrasts
- **Causal tracing** and **activation patching** paradigms (Vig et al., 2020; Wang et al., 2023)
- **Feed-forward memory** theories of factual knowledge (Geva et al., 2021; Dai et al., 2022)
- Prior MoE interpretability work on routing patterns and expert specialization

---

## Methodology

### 3.1 COUNTERFACT Factual Tracing Setup

For a prompt $x$, true object $a_t$, and foil object $a_f$, the factual preference is defined as the logit difference:

$$
\Delta(x; a_t, a_f) = \ell_x(a_t) - \ell_x(a_f), \tag{1}
$$

where $\ell_x(a)$ is the next-token logit for object token $a$. The object token is **not** included in the input; true and foil objects only define the next-token contrast.

### 3.2 Subject-Noise Corrupted Run

The corrupted run adds Gaussian noise to subject-token embeddings:

$$
h_s^{\mathrm{noised}} = h_s + \epsilon, \quad \epsilon \sim \mathcal{N}(0, \sigma^2 I), \tag{2}
$$

where $\sigma = 3.0 \times$ (embedding-matrix standard deviation). The subject-noise drop is:

$$
\Delta_{\mathrm{clean}} - \Delta_{\mathrm{noised}}. \tag{3}
$$

### 3.3 MoE-Block Output Patching

At layer $\ell$, the clean final-position MoE-block output is patched into the noised run. The rescue score is:

$$
\mathrm{Rescue}_{\ell} = \Delta_{\mathrm{patched}, \ell} - \Delta_{\mathrm{noised}}. \tag{4}
$$

**Discovery/validation split**: Layers are selected only on discovery cases, then evaluated as fixed hypotheses on validation cases.

### 3.4 Expert-Contribution Patching

At a selected MoE layer, for a clean-active expert $e$, its routed update is estimated by ablation difference:

$$
c_e(x) = \mathrm{MoEOut}(x) - \mathrm{MoEOut}_{\backslash e}(x), \tag{5}
$$

where $\mathrm{MoEOut}_{\backslash e}$ suppresses expert $e$'s routed contribution under the original top-k routing decision (no rerouting, no renormalization).

The expert patch vector is:

$$
\delta_e = c_e(x_{\mathrm{clean}}) - c_e(x_{\mathrm{noised}}), \tag{6}
$$

added to the noised MoE-block output at the final position.

### Confound Controls

1. **Recurrence-first selection**: Experts must appear in at least half of discovery cases (64/128) to avoid single-case artifacts.
2. **Specificity metric**: Selected expert is compared against same-source active-random experts:

$$
\mathrm{Spec} = \mathrm{Rescue}(e^\star) - \frac{1}{|\mathcal{R}|} \sum_{r \in \mathcal{R}} \mathrm{Rescue}(r), \tag{7}
$$

where $e^\star$ is the selected expert and $\mathcal{R}$ is the active-random control set.

### Models Evaluated

| Model | MoE Layers | Experts/Layer | Top-k Routing |
|-------|-----------|---------------|---------------|
| Qwen3-30B-A3B-Base | 48 | 128 | 8 |
| Mixtral-8X7B-V0.1 | 32 | 8 | 2 |

### Data

- 256 usable COUNTERFACT cases per model (128 discovery + 128 validation)
- Model-specific filters for single-token true/foil objects and sufficient clean/noise margins

---

## Empirical Validation / Results

### Layer-Level Tracing Results

| Model | Layer | Layer Rescue [95% CI] | Expert | Expert Rescue [95% CI] | Spec [95% CI] |
|-------|-------|----------------------|--------|------------------------|---------------|
| **Qwen3** | L44 | +0.901 [+0.752, +1.053] | L44E069 | +0.463 [+0.344, +0.590] | **+0.400** [+0.276, +0.533] |
| **Mixtral** | L19 | +0.457 [+0.331, +0.579] | L19E006 | +0.099 [+0.018, +0.175] | **-0.175** [-0.284, -0.072] |

**Table 1**: Main validation results on 128 held-out cases per model. Spec subtracts same-layer active-random rescue.

### Key Findings

#### Qwen3-30B-A3B-Base
- Layer 44 strongly validates as a layer-level site
- Expert **L44E069** validates with **positive specificity** (+0.400)
- The expert is not merely an arbitrary active layer-44 expert; its patch vector $\delta_e$ rescues more than active-random controls
- Multiple additional controls preserve this conclusion (Appendices C–E)

#### Mixtral-8X7B-V0.1
- Layer 19 validates, but nearby layers are comparable → interpreted as **mid-layer transfer** rather than sharp single-layer localization
- Selected expert L19E006 has **negative specificity** (-0.175)
- **Coalition check**: Clean top-2 and clean/noised routing-union coalitions recover rescue close to the L19 MoE-block effect
- Layer-level signal is better explained by **routed expert coalitions** than single-expert localization

### Visual Summary (Figure 1)

> **Left panel**: MoE-block rescue across layers — Qwen3 shows sharp L44 peak; Mixtral shows broader mid-layer band.
> **Middle panel**: Selected-expert specificity — Qwen3 L44E069 positive; Mixtral L19E006 negative.
> **Right panel**: Mixtral coalition patching — dashed line marks L19 MoE-block rescue; coalitions recover near-block-level rescue.

---

## Theoretical and Practical Implications

### Methodological Implications

1. **Block-level rescue ≠ expert-level evidence**: MoE factual tracing should not stop at layer-level activation patching. A clean MoE-block output can rescue a noised factual prompt, but this only localizes the signal to an aggregate routed block.

2. **Router selection is not causal**: An expert being selected by the router identifies candidates, but causal evidence requires patching the estimated expert update and comparing with active controls.

3. **Three distinct levels of evidence**:
   - Aggregate MoE-block rescue
   - Single-expert localization
   - Multi-expert routed recovery

### Practical Implications

- **Model debugging**: More precise localization of factual behavior supports targeted model debugging and factual-behavior analysis.
- **Dual-use risk**: Precise localization could enable targeted manipulation, suppression, or editing of model outputs — such uses require careful evaluation before deployment.

### Architectural Insights

- **Qwen3** (128 experts, top-8): Factual recall can concentrate in a **recurrent expert** (L44E069)
- **Mixtral** (8 experts, top-2): Factual recall remains **distributed across routed expert coalitions**

The asymmetry suggests that expert specialization for factual recall is **not universal** and depends on architecture, routing design, and training.

---

## Conclusion

The paper introduces **expert-aware causal tracing** for sparse MoE factual-recall diagnostics, with two controlled case studies:

1. **Qwen3-30B-A3B-Base**: Layer-level site (L44) further localized to a robust routed expert contribution, **L44E069**, with positive specificity across held-out cases.

2. **Mixtral-8X7B-V0.1**: Layer-level rescue transfers (L19 band), but the signal is better recovered by **routed expert coalitions** than by the selected single expert.

### Future Directions

- Extension to more datasets and MoE backbones
- Multi-token objects
- Subject/relation-position interventions
- Systematic coalition analyses across layers and models
- The relaxed-filter 512-case check preserves the main pattern, but the filtering funnel may bias retained cases

### Limitations

- Controlled diagnostic rather than full account of natural factual recall
- Expert contributions estimated by ablation difference under fixed routing (not a complete subject-to-answer circuit)
- Two-stage design decomposes the selected layer only — useful experts outside the selected layer may be missed
- Model contrast treated as two case studies, not a broad survey
- Nontrivial compute requirements for reproduction

---

_Markdown view of https://picx.dev/p/pUGl34, served by PicX — AI-generated visual whiteboard summaries of research papers._