# Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

> Sparse MoE routing works through coherent overlap, not geometric complementarity, as selected experts share subspaces yet still improve prediction, making geometric similarity alone insufficient for pruning decisions.

- **Source:** [arXiv](https://arxiv.org/abs/2607.28308)
- **Published:** 2026-08-22
- **Permalink:** https://picx.dev/p/vUWmrq
- **Whiteboard:** https://picx.dev/p/vUWmrq/image

## Summary

# Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

## Summary (Overview)

- **Core Finding**: The paper challenges the assumption that sparse MoE routing works through "geometric complementarity"—the idea that co-selected experts contribute distinct, non-overlapping representation directions. Instead, the authors propose **"coherent overlap"**: expert subspaces overlap substantially, yet actual routes still fit tokens better than alternatives.
- **Novel Framework**: The authors introduce the **Expert Subspace Separation Index (ESSI)** and a **prefix-controlled 2×2 factorial design** to separately identify candidate quality, contextual opportunity, and their interaction in expert routing.
- **Key Empirical Pattern**: Across 39 factorial cells in OLMoE, Mixtral, and DeepSeek, selected candidates consistently explain more of the residual representation than rivals (candidate quality), but the actual prefix *narrows* rather than amplifies this advantage (negative interaction) in all 39 cells.
- **Functional Value Persists**: Despite geometric overlap, later experts improve next-token prediction in 24/39 frozen-route comparisons, and a controlled Top-2 vs. Top-1 training study favors Top-2 in all three seeds.
- **Critical Implication**: Geometric similarity alone cannot determine redundancy or pruning value—output-level interventions are necessary.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Mixture-of-experts (MoE) models combine input-dependent gating with specialized local computations. Sparse variants scale total capacity while activating only a small fraction of parameters per token. Modern language models commonly use learned Top-k routing—but the mechanistic question of *why multiple experts help* remains open.

### The Geometric Complementarity Hypothesis

A common explanation is **geometric complementarity**: one selected expert captures a token's dominant directions while remaining experts cover what is left. This intuition motivates:
- Explanations for multi-expert quality
- Design principles for sparse computation
- Expert pruning and compression objectives

### Three Distinct Claims

The authors decompose geometric complementarity into three properties that need not coincide:

1. **Route coherence**: The complete selected set fits the token better than a matched alternative set.
2. **Candidate quality**: Under a fixed context, the selected expert fits the token better than the strongest unselected alternative.
3. **Positive geometric complementarity**: The selected expert becomes especially useful *because of* the specific experts with which it is grouped.

### Key Insight

> Prior work conflates these three properties. Identifying the interaction requires crossing candidates and contexts while holding each fixed in turn—a design the authors call the **prefix-controlled 2×2 factorial**.

---

## Methodology

### Models Analyzed

| Model | Routing Design | Depth | Routed Experts | Top-k | Shared | Evidence |
|-------|---------------|-------|----------------|-------|--------|----------|
| OLMoE-1B-7B | Routed-only | 16 | 64 | 8 | 0 | Geometry; Factorial; NLL |
| Mixtral-8x7B-v0.1 | Routed-only | 32 | 8 | 2 | 0 | Geometry; Factorial; NLL |
| DeepSeek-MoE-16B | Shared-expert | 28 | 64 | 6 | 2 | Geometry; Factorial; NLL |
| Qwen3-30B-A3B-Base | Routed-only | 48 | 128 | 8 | 0 | Geometry |
| Gemma4-26B-A4B | Shared-expert | 30 | 128 | 8 | 1 | Geometry |
| Qwen3.6-35B-A3B | Hybrid | 40 | 256 | 8 | 1 | Geometry |

### Expert Subspace Separation Index (ESSI)

The ESSI calibrates between-expert distances by local within-expert tangent dispersion. For rank-*p* bases *U* and *V*, the normalized chordal distance on the Grassmann manifold is:

$$
d_{\mathrm{ch}}(U, V) = \left(\frac{1}{p} \sum_{i=1}^{p} \sin^2 \theta_i\right)^{1/2} = \frac{\|UU^\top - VV^\top\|_F}{\sqrt{2p}}
$$

where $\theta_1, \ldots, \theta_p$ are the principal angles. The ESSI is then:

$$
\mathrm{ESSI} = \frac{\operatorname{mean}_{e < f}\, d_{\mathrm{ch}}(G_e, G_f)}{\max\left\{\sum_{e} \lambda_e \operatorname{mean}_{a \in \mathcal{A}_e} d_{\mathrm{ch}}(G_e, L_{e,a}),\ \epsilon\right\}}
$$

with $\epsilon = 10^{-12}$. An ESSI near one indicates that inter-expert separation is comparable to local within-expert variation.

### Prefix-Controlled 2×2 Factorial Design

The design crosses two candidates (selected expert *s* vs. rival *r*) with two contexts (actual prefix *S* vs. alternative prefix *W*):

- **Candidate advantage**: $A_{\text{actual}} = q_x(s \mid S) - q_x(r \mid S)$
- **Context effect**: $T_s = q_x(s \mid S) - \frac{1}{M}\sum_m q_x(s \mid W_m)$
- **Interaction (difference-in-differences)**: $D = A_{\text{actual}} - A_{\text{alt}} = T_s - T_r$

where $q_x(c \mid P)$ is the fractional novelty of candidate *c* given prefix *P*, defined via the unexplained energy fraction:

$$
\rho_x(P) = \frac{\|(I - B(P)B(P)^\top)z_x\|_2^2}{\|z_x\|_2^2}
$$

### Frozen-Route NLL Interventions

To assess functional value, the authors measure next-token NLL changes when replacing experts in frozen routes, plus a controlled Top-1/Top-2 training study with matched active parameters.

---

## Empirical Validation / Results

### 1. Expert Subspaces Overlap

- **ESSI remains near one** across all six models and 18 layers (range 0.776–1.060, median 0.969)
- This indicates between-expert separation is comparable to local within-expert variation—inconsistent with a hard directional partition

### 2. Routes Remain Coherent

- The alternative-to-actual residual ratio exceeds 1 in all 18 cells (range 1.012–1.901, median 1.283)
- Actual routes fit tokens better than load-matched alternatives, ruling out pure interchangeability

### 3. Factorial Results: Strong Candidates, Negative Interaction

| Metric | Finding |
|--------|---------|
| **Candidate advantage** $A_{\text{actual}} > 0$ | All 39 cells; 22/24 OLMoE/Mixtral CIs exclude zero |
| **Context effects** $T_s, T_r < 0$ | All estimates negative; actual prefix removes directions both candidates could add |
| **Interaction** $D < 0$ | All 39 cells; every 95% CI below zero |
| **Macro D (OLMoE+Mixtral)** | −0.05548 [−0.05780, −0.05340] |
| **Macro D (DeepSeek)** | −0.05770 [−0.06165, −0.05405] |

### 4. Naive Comparisons Reverse Conclusions

- When both candidate **and** context change: all 9 leader comparisons positive but all 39 later-expert comparisons negative
- When only the candidate changes (context fixed): all 39 comparisons favor the selected expert
- This sign reversal demonstrates that changing context can make a strong candidate appear weak

### 5. Functional Value Within Overlapping Geometry

- **Later-expert additions improve prediction**: 24/39 frozen-route additions reduce NLL; largest effects early (first addition recovers 0.0906 NLL at OLMoE layer 16)
- **Value is distributed**: Replacing the leader causes more damage in 7/9 configurations, but the aggregate later set matters more at OLMoE layers 8 and 16
- **Controlled training**: Top-2 (8 experts × width 512) beats Top-1 (4 experts × width 1,024) with $\Delta = 0.1016 \pm 0.0025$ (mean ± SD) across three seeds

### 6. Sensitivity Analyses

- Raw-gain and nearest-match intervals remain negative in all 24 OLMoE/Mixtral cells
- Strict caliper: all point estimates negative; 20/24 intervals exclude zero
- DeepSeek macro interaction negative under all three variants

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Separation of concepts**: The paper cleanly separates route coherence, candidate quality, and interaction effects—concepts previously conflated in the literature. The negative interaction is consistent with **geometric saturation** rather than route failure: the actual prefix already covers directions available to both candidates.

2. **Representation-dependent claims**: Complementarity claims must specify representation, rank, metric, and intervention. Rank-128 linear coverage of router inputs is distinct from nonlinear features, output directions, or logit effects.

3. **Rethinking redundancy**: Geometric overlap does not imply functional redundancy. Low linear novelty can coexist with predictive gain because experts apply parameterized nonlinear transformations.

### Practical Implications

1. **Pruning and compression**: Geometric similarity can screen candidates, but pruning/merging/skipping should be tested under the *retained* route because removal changes the context of other experts.

2. **Adaptive routing**: Separating subspaces need not improve prediction; adaptive-k routing could instead estimate another expert's output gain subject to load and compute constraints.

3. **Model interpretation**: Router rank is informative (leader replacement causes more damage in most cases), but leader prominence can coexist with substantial collective value among later experts.

### Key Caveats

- Conclusions concern a linear, rank-128 router-input metric
- Factorial analyses cover three architectures (OLMoE, Mixtral, DeepSeek)
- Training study is small-scale (six-layer MoE)
- DeepSeek's shared experts remain fixed outside routed sets

---

## Conclusion

This work makes complementarity *testable* rather than assumed. The key contributions are:

1. **ESSI** calibrates between-expert separation against within-expert variation
2. The **prefix-controlled 2×2 factorial** separately identifies candidate quality, contextual opportunity, and their interaction
3. **Functional interventions** measure predictive value of overlapping experts

The central finding—**coherent overlap**—shows that multi-expert benefit can arise from distinct computations within a shared token-relevant neighborhood without disjoint linear coverage. The actual context narrows rather than amplifies the selected candidate's geometric advantage, yet later experts remain functionally valuable.

### Future Directions

- Investigating whether semantic routing or expert collaboration reveals specialization invisible to subspace metrics
- Developing pruning criteria based on output-level counterfactuals rather than input-subspace overlap
- Exploring whether adaptive-k routing based on estimated output gains outperforms fixed Top-k
- Extending the framework to attention layers and other MoE variants

The separation of expert overlap, route coherence, candidate quality, contextual interaction, and functional value fundamentally changes the mechanistic picture of MoE routing: geometric similarity alone cannot determine redundancy or multi-expert benefit.

---

_Markdown view of https://picx.dev/p/vUWmrq, served by PicX — AI-generated visual whiteboard summaries of research papers._