Summary (Overview)

• Pre-registered causal test: The paper tests whether expert families in Command A+, a frontier open-weights MoE (218B total / 25B active; 128 experts, 8 active, +1 shared), form functional modules tied to capabilities or languages, using a pre-registered protocol with a size-matched random-expert null and a falsifiable selectivity criterion.
• Main finding: Only 1 of 6 pre-registered expert families (Arabic) qualifies as a robust, selective module under a conservative statistical bar; a more permissive point rule admits 3/6, but the boundary is crowded (Spanish clears selectivity by only 0.002; code misses by 0.009).
• Measurement-dependence: Apparent modularity is fragile to the choice of corpus (Spanish is selective on one corpus but bleeds into Arabic on a second), metric (math is entangled with general reasoning under task accuracy yet looks selective under solution-likelihood), and statistical bar.
• Positive control: The pipeline recovers published disjoint structure on Qwen3-30B-A3B, confirming the method detects modularity when present; the verdict reproduces on un-quantized BF16, ruling out a 4-bit quantization artifact.
• Key methodological contribution: Ablation-based modularity verdicts are not safe unless corpus, metric, and statistical bar are controlled; the paper releases the atlas and ablation data.

Introduction and Theoretical Foundation

The paper addresses a central question in mechanistic interpretability: do sparse Mixture-of-Experts (MoE) models exhibit functional modularity, where experts specialize into families that each own a capability (math, code) or a language?

The authors emphasize that modularity is a causal claim — observational routing signals (which experts a token is routed to) do not establish what the output depends on. Prior work has repeatedly shown that observational routing metrics fail to predict ablation importance.

The paper makes three methodological commitments:

1. Pre-registration: Families are defined from routing and their hypothesized roles frozen before any ablation.
2. Size-matched random null: An effect counts only if it beats ablating the same number of random experts.
3. Falsifiable selectivity criterion: "Not modular" is an admissible outcome.

The central methodological move is treating the measurement as a variable — scoring the same ablations under four metrics with bootstrap confidence intervals under both permissive and conservative decision rules.

Methodology

Model and Ablation Operator

• Model: Command A+ served on a single RunPod B200 via an in-process vLLM engine (under 30 B200-hours total).
• Precision: Open-weights W4A4 (NVFP4) build, with a BF16 re-run for validation.
• Ablation mechanism: Router masking with gate renormalization — masked experts' router logits set to $-\infty$ before the top-k softmax at every MoE layer. A smoke test confirms exactness (masked experts go from thousands of live routings to zero; leak = 0).

Atlas and Pre-registration

A labeled probe corpus is driven through the router, recording per-(layer, expert) routing mass. Each family is pre-registered as its top-16 experts by lift, excluding always-on shared experts. Six families are frozen before any ablation: math, code, general; Arabic (ar), Chinese (zh), Spanish (es).

Causal Test and Decision Rules

For family $F$ with axis $A$, the family is modular if:

1. Its on-target effect beats the size-16 random-expert null (mean $+ 2\sigma$)
2. Its worst of-target effect is $\leq \frac{1}{3}$ of on-target (selectivity)

Two rules are reported:

• Point rule: On raw effect sizes (pre-registered form)
• Conservative rule: Using bootstrap 95% CIs — beats-null requires the lower CI of on-target to exceed the null; selectivity requires the upper CI of the worst of-target to fall below one third of the lower CI of on-target.

The null is estimated from ~10 random draws (a noise band, not a calibrated p-value). The $\frac{1}{3}$ selectivity ratio and null construction were pre-specified.

Four Metrics, Two Language Corpora

Positive Control

The identical pipeline runs on Qwen3-30B-A3B (same 128/8 shape), where the literature reports clean language/task families, to verify detector sensitivity.

Empirical Validation / Results

Detector Works; Command A+ Routing is Weakly Separated

On Qwen3-30B-A3B, the pipeline recovers the published structure: language–language top-set Jaccard 0.60 vs. language–task 0.19 (gap 0.41). On Command A+, the separation gap is only 0.09 (language–language 0.49, capability–language 0.40). Coverage is only mildly concentrated (~64 of 128 experts carry ~62% of mass, vs. 50% under uniform routing).

Hardened Verdict: One Survivor

Table 1: Hardened verdict (independent corpus, per-item NLL, bootstrap 95% CIs)

The One Robust Module: Arabic

Ablating the Arabic family raises Arabic per-token NLL by +1.80 nats [95% CI 1.73, 1.86], a ~6× rise in perplexity (from 8.2 to 49), against a random-null band of only 0.64 nats, on a corpus it was never tuned to. Its worst of-target (English, +0.43) stays well under the selectivity bar.

Why the Rest Fail: Measurement-Dependence with Mechanism

• Spanish: Corpus-dependent — clean on FLoRes-200 passages, but on independent Wikipedia text ablating Spanish bleeds +0.43 into Arabic. The es and ar families overlap in only 2 of 16 experts ({30, 86}; Jaccard 0.067).
• Math: Metric-dependent — entangled with general reasoning under task accuracy (drops MATH by 0.16 and MMLU-Pro by 0.11), apparently selective under solution-likelihood point estimates, non-selective once CIs applied.
• Code: Metric-dependent the other way — no detectable accuracy effect (0.00 on HumanEval) yet the largest problem-text NLL effect of any capability (+1.29), bleeding +0.62 into math.

Not a Quantization Artifact (BF16)

The verdict reproduces at full precision: BF16 confirms exactly one modular family, Arabic (on-target +1.81 nats vs. W4A4's +1.80; null band 0.69). The only family to move is the knife-edge es, whose point-rule selectivity flips from just passing in W4A4 (+0.002) to just failing in BF16 (−0.015).

Routing Mass Does Not Predict Modularity

The routing-defined "universal" shared-core is causally redundant: a dose-response masking up to 32 of the most-shared experts degrades MMLU-Pro by only ~0.11, gracefully with no cliff. Family lift correlates loosely with causal importance but not with selectivity.

Theoretical and Practical Implications

The authors identify a pattern: representational axes separate more cleanly than computational ones.

• Arabic, the one robust module, is a surface property of the token distribution (distinct script and lexicon) that routing can factor out.
• Capability axes (math, code, general) are computations that share machinery, so they entangle — visibly only under metrics that exercise the shared computation.
• Even representational modularity is fragile: Spanish fails on a held-out corpus because it shares experts with Arabic.

The practical lesson for interpretability is concrete: an ablation-based modularity claim is not safe unless it survives an independent corpus, more than one metric, and a conservative statistical bar. So tested, only one of six families qualifies as a clean module.

The authors make no base-rate claim about MoEs in general from a single model; what generalizes is the methodological requirement, not the count.

Conclusion

Robust functional modularity is rare in the frontier MoE tested. Of six pre-registered expert families, only the Arabic language family is a clean, selective module that survives an independent corpus and a conservative bar. Every other family sits at the decision boundary, with apparent modularity made or unmade by the corpus, metric, or statistical treatment.

The paper's contributions are:

1. A pre-registered, control-anchored causal protocol for expert modularity with explicit corpus/metric/statistical robustness
2. One clean positive result: a causally localized Arabic-language module
3. A threshold-independent result that routing mass predicts neither causal importance nor selectivity
4. A cautionary demonstration that apparent modularity is fragile to measurement choices

Future directions include causal-attribution atlases, injection probes, and large-fraction ablations to probe what 16/128 routing ablation cannot. The atlas and ablation data are released at github.com/transformerlab/exp-command-a-plus-moe-modularity under CC BY 4.0.