# Multi-View MoE: Letting Every Router Shape Expert Computation

> Multi-View MoE reuses routing preferences to both select experts and shape their inputs, boosting downstream accuracy by 1.18 points without extra expert computation.

- **Source:** [arXiv](https://arxiv.org/abs/2610.40577)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/cTXjvJ
- **Whiteboard:** https://picx.dev/p/cTXjvJ/image

## Summary

# Multi-View MoE: Letting Every Router Shape Expert Computation

## Summary (Overview)

- **Core contribution**: The paper proposes Multi-View MoE, a framework that improves sparse mixture-of-experts (MoE) models by using multiple routing heads for both expert selection and input construction, all under a fixed expert budget.

- **Two key submodules**: (1) **Multi-Router** aggregates scores from independently initialized routing heads for better expert selection; (2) **Multi-View** further reuses routing preferences to mix learned latent views into each selected expert's input before computation.

- **Fixed computational cost**: The method preserves active-expert count, expert FLOPs, and all-to-all communication volume—each selected expert still performs exactly one forward pass.

- **Quantitative gains**: On a 24-layer 10B-A1B model trained on 708B tokens, full Multi-View MoE improves the nine-task downstream mean by 1.18 percentage points (33.07 → 34.25) over baseline; Multi-Router alone provides a 0.6-point gain.

- **Routing quality improvement**: Multi-Router improves the rank correlation between router scores and empirical expert-utility evaluations by a relative 8.1% globally, with approximately 58% relative improvement in the layer with the lowest single-router alignment.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Sparse mixture-of-experts (MoE) models expand parameter capacity while activating only a small subset of experts per token (Shazeer et al., 2017; Fedus et al., 2021). The routing mechanism determines which experts process each token, directly affecting model capability under a fixed active-expert budget.

**Key observation**: Dense evaluation of all experts reveals **weak agreement between single-router score rankings and rankings from empirical expert-utility evaluations**, especially in middle and deep layers (Figure 2 of the paper). The Spearman correlation between single-router scores and normalized expert-output norms is only 0.1245 globally.

**Self-reinforcing poor selections**: Unselected experts receive no direct task-loss gradient from a given token, so poor selections may become self-reinforcing during training. This motivates assessing experts from multiple scoring perspectives rather than relying on a single router.

### Limitations of Existing Approaches

| Approach | Limitation |
|----------|------------|
| Standard MoE | Single routing head; scores not reused for input construction |
| Load-balancing methods | Regulate utilization but provide no multiple assessments of experts |
| Activation-based routing (e.g., AoE) | Requires partial evaluation of all experts; adds cost |
| Multiple input representations | Would multiply expert computation |

### Theoretical Insight

The problem is to obtain **multiple assessments of experts** and use routing preferences to modulate their computation while preserving computational efficiency.

The paper observes a critical representational limitation: the summed score alone does not retain routers' relative preferences. Score vectors $(0.4, 0.1)$ and $(0.2, 0.3)$ both sum to 0.5 but express **opposite preferences** (Figure 1). Multi-View exploits this distinction by using the relative scores to blend views.

---

## Methodology

### Framework Overview

The backbone is a latent-space MoE with down-projection $P \in \mathbb{R}^{d \times D}$ supplying shared input $z = Ph$. The latent expert aggregation is:

$$y = \sum_{i \in \mathcal{S}} g_i E_i(z) \tag{1}$$

where $\mathcal{S}$ is the selected expert set, $g_i$ are output gates, and $E_i$ maps $\mathbb{R}^d \to \mathbb{R}^d$.

### Multi-Router: Score Aggregation Before Selection

Multi-Router uses $R$ routing heads $\{w^{(r)}\}$, $w^{(r)} \in \mathbb{R}^{D \times N}$, each producing sigmoid scores summed before top-$k$ selection:

$$\begin{aligned} \bar{s}_i &= \sum_{r=1}^{R} \sigma\big(w_i^{(r)\top} h\big), \\ \mathcal{S} &= \mathrm{TopK}_k(\bar{s} + b). \end{aligned} \tag{2}$$

Here $b \in \mathbb{R}^N$ is the expert-balancing bias, added only for selection. Output gates normalize the uncorrected score sums:

$$g_i = \alpha_i \frac{\bar{s}_i}{\sum_{j \in \mathcal{S}} \bar{s}_j}, \qquad i \in \mathcal{S} \tag{3}$$

**Critical design choice**: Summation happens **after** the sigmoid. Summing logits before the sigmoid collapses into a single linear scorer since $\sum_r w_i^{(r)\top} h = \left(\sum_r w_i^{(r)}\right)^{\top} h$. Post-sigmoid summation preserves routers as distinct nonlinear scoring functions.

### Multi-View: Router-Conditioned Expert Inputs

Multi-View pairs $R$ routing heads with $K$ latent views in one-to-one correspondence ($R = K$). Views are formed as $v^{(m)} = P_m h \in \mathbb{R}^d$, with view 1 using the backbone's projection and remaining views using independently initialized perturbations.

**View mixing weights** are derived by normalizing each selected expert's raw sigmoid scores across routing heads:

$$a_{m,i} = \frac{s_{m,i}}{\sum_{n=1}^{R} s_{n,i}}, \qquad \sum_{m=1}^{K} a_{m,i} = 1, \quad i \in \mathcal{S} \tag{4}$$

The mixed input is the convex combination:

$$\hat{v}_i = \sum_{m} a_{m,i} v^{(m)}$$

The resulting expert aggregation:

$$y = \sum_{i \in \mathcal{S}} g_i E_i\left(\sum_{m} a_{m,i} v^{(m)}\right) \tag{5}$$

**Key property**: For nonlinear $E_i$, mixing before evaluation is **not** equivalent to mixing outputs after separate evaluations:

$$E_i\left(\sum_{m} a_{m,i} v^{(m)}\right) \neq \sum_{m} a_{m,i} E_i(v^{(m)})$$

**Effective projection view**: The input uses a token- and expert-dependent effective projection:

$$P_{\mathrm{eff},i}(h) = \sum_{m} a_{m,i}(h) P_m, \qquad \hat{v}_i = P_{\mathrm{eff},i}(h) h$$

The projection basis $\{P_m\}$ is shared across experts; only mixing coefficients depend on $h$ and $i$.

### Initialization Strategy

To introduce additional projections gradually:
- View 1 retains backbone projection $P$
- Other views are ramped perturbations: $v^{(m)} = Ph + \gamma(t)\Delta P_m h$
- $\gamma$ increases from 0 to 1 early in training
- At $t = 0$, all view projections equal the backbone projection

### Algorithm 1: Forward Pass

```
Require: hidden state h; projections {P_m}; routing heads {w^(m)}; 
        experts {E_i}; active count k; balancing bias b; scales {α_i}

1:  v^(m) ← P_m h for m = 1, ..., K        ▷ K latent views
2:  s_{m,i} ← σ(w_i^(m)⊤ h)                ▷ per-router scores
3:  s̄_i ← Σ_m s_{m,i}                       ▷ summed score
4:  S ← TopK_k(s̄ + b)                       ▷ bias only for selection
5:  for i ∈ S do
6:      a_{m,i} ← s_{m,i}/Σ_n s_{n,i}       ▷ normalize across routers
7:      v̂_i ← Σ_m a_{m,i} v^(m)             ▷ per-expert view blend
8:      g_i ← α_i s̄_i/Σ_{j∈S} s̄_j           ▷ output gate (Eq. 3)
9:  end for
10: return y = Σ_{i∈S} g_i E_i(v̂_i)
```

---

## Empirical Validation / Results

### Setup

- 24-layer 10B-A1B backbone with top-8 routing and one always-active shared expert
- Trained on 708B tokens
- Three configurations share training data, token budget, backbone, and optimization settings

### Downstream Performance (Table 1)

At 708B training tokens, mean scores across nine tasks:

| Method | CMMLU | MMLU | Redux | Pro | Avg. (Exams) | HotpotQA | SimpleQA | PreciseWiki | Avg. (Knowledge) | GPQA | BBH | **Overall** |
|--------|-------|------|-------|-----|--------------|----------|-----------|-------------|-------------------|------|-----|-------------|
| Baseline (LatentMoE) | 53.71 | 57.48 | 59.03 | 27.28 | 49.38 | 22.00 | 3.35 | 8.64 | 11.33 | 30.00 | 36.10 | **33.07** |
| Multi-Router (R=2) | 53.77 | 57.50 | 58.90 | 27.71 | 49.47 | 22.13 | 3.85 | 9.16 | 11.71 | 32.86 | 37.31 | **33.69** |
| Multi-View (K=2) | 55.20 | 58.67 | 60.93 | 27.12 | 50.48 | 22.54 | 3.41 | 9.02 | 11.66 | 35.60 | 35.76 | **34.25** |
| Δ (MV − Baseline) | +1.49 | +1.19 | +1.90 | −0.16 | +1.10 | +0.54 | +0.06 | +0.38 | +0.33 | +5.60 | −0.34 | **+1.18** |

**Key findings**: Multi-View improves the mean by 1.18 percentage points over baseline and 0.56 over Multi-Router. The largest gains are on GPQA (+5.60 points) and MMLU-Redux (+1.90).

### Training Loss Convergence

Over the final 1,000 steps:
- Baseline: 2.25621
- Multi-Router: 2.25047 (reduction of $5.74 \times 10^{-3}$)
- Multi-View: 2.24883 (reduction of $7.38 \times 10^{-3}$)

### Routing Score–Utility Alignment

**Table 2: Global Spearman correlation ρ**

| Method | Spearman ρ |
|--------|------------|
| Single router | 0.1245 |
| Multi-Router | 0.1346 |

- Multi-Router improves rank correlation by a **relative 8.1%**
- Gains in 17 of 24 layers
- Largest relative gain at layer 20: 0.0625 → 0.0986 (**57.7% relative**)
- Largest absolute gain at layer 7: 0.0982 → 0.1388

### Controlled Mixing Interventions

At the final MoE layer, holding $\mathcal{S}$ and output gates $g_i$ fixed to numerical precision:

| Intervention | ΔNLL (nats/token) | 95% CI |
|-------------|-------------------|--------|
| Permute view proportions among experts | 0.01340 | [0.01256, 0.01435] |
| Uniform mixing (1/2, 1/2) | 0.00662 | [0.00620, 0.00709] |
| Permute proportions across tokens | 0.00223 | [0.00205, 0.00241] |

**Both views contribute**: View 1-only proportions → ΔNLL = 0.01881; view 2-only → ΔNLL = 0.02042.

### Training Ablations (Tables 3 and 4)

| Configuration | Δ Loss vs. Baseline |
|---------------|-------------------|
| Multi-Router (R=2, sigmoid-sum) | −0.0045 |
| Multi-Router (R=3) | −0.0051 |
| Fusion: max | −0.0037 |
| Fusion: geometric mean | −0.0037 |
| Fusion: product | −0.0034 |
| Learned per-expert weights (init 0.5) | −0.0048 |
| Coupled view-fed scoring | −0.0024 |
| **Detached view-fed scoring** | **+0.0090** |
| Observed run-pair difference | ±0.0012 |

**Critical finding**: Stopping gradient flow from mixing weights to views changes loss from −0.0024 to +0.0090 (increase of 0.0114), isolating the benefit of the gradient coupling path.

### Efficiency

- **Multi-Router parameter overhead**: ~0.06%
- **Multi-View total overhead**: ~0.19%
- **Computational cost preserved**: Each expert runs once; mixing before dispatch; same all-to-all volume
- **Additional cost scales as**: $D(d + N)$ for fixed $K$

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Routing preferences as a dual-purpose signal**: The paper establishes that routing scores can serve simultaneously for expert selection AND expert input construction, expanding the role of routers in MoE architectures.

2. **Non-equivalence of pre/post mixing**: The mathematical framing shows that mixing views before expert evaluation ($E_i(\sum_m a_{m,i}v^{(m)})$) is fundamentally different from mixing after separate evaluations ($\sum_m a_{m,i} E_i(v^{(m)})$), justifying the design choice.

3. **Post-sigmoid aggregation rationale**: Summing scores after sigmoid preserves routing heads as distinct nonlinear scoring functions, preventing collapse into a single linear scorer—a subtle but crucial design decision.

### Practical Implications

1. **Fixed-budget enhancement**: Multi-View MoE provides gains without increasing active-expert count, expert FLOPs, or communication volume—key practical constraints in distributed training.

2. **Scalable approach**: The number of projections depends on $K$ (routing heads), not on the number of experts $N$, and expert parameters are shared across views.

3. **Warm-start compatibility**: The ramped initialization procedure allows loading from existing checkpoints, facilitating adoption in deployed models.

4. **Self-stabilizing router diversity**: Routing heads develop highly differentiated preferences without explicit diversity penalties, suggesting emergent specialization across routing heads.

---

## Conclusion

### Main Takeaways

Multi-View MoE reuses routing preferences to both improve expert selection and shape expert computation. The framework addresses the observed weak agreement between single-router rankings and empirical expert utility by:

1. **Multi-Router submodule**: Combining independently initialized routing heads that learn highly differentiated expert preferences
2. **Multi-View extension**: Using relative router scores to blend latent views into expert inputs before dispatch

The 8.1% global improvement in score–utility rank correlation and 1.18 percentage point downstream gain demonstrate that routing preferences encode valuable signal beyond simple selection that can be productively reused for computation shaping.

### Future Directions

The paper's limitations suggest several promising directions:
- Scaling to larger backbones and longer training runs
- Exploring more than two routing heads/views ($R = K > 2$)
- Investigating whether the differentiated router preferences can be interpreted or controlled
- Extending the router-conditioned input framework to other MoE variants (e.g., fine-grained experts)

**Final insight**: "Routing preferences" serve as a rich signal that can inform both *which* experts process a token and *how* they process it, establishing a new paradigm where routers do more than just select—they instruct.

---

_Markdown view of https://picx.dev/p/cTXjvJ, served by PicX — AI-generated visual whiteboard summaries of research papers._