Multi-View MoE: Letting Every Router Shape Expert Computation

Summary (Overview)

• Core contribution: The paper proposes Multi-View MoE, a framework that improves sparse mixture-of-experts (MoE) models by using multiple routing heads for both expert selection and input construction, all under a fixed expert budget.

• Two key submodules: (1) Multi-Router aggregates scores from independently initialized routing heads for better expert selection; (2) Multi-View further reuses routing preferences to mix learned latent views into each selected expert's input before computation.

• Fixed computational cost: The method preserves active-expert count, expert FLOPs, and all-to-all communication volume—each selected expert still performs exactly one forward pass.

• Quantitative gains: On a 24-layer 10B-A1B model trained on 708B tokens, full Multi-View MoE improves the nine-task downstream mean by 1.18 percentage points (33.07 → 34.25) over baseline; Multi-Router alone provides a 0.6-point gain.

• Routing quality improvement: Multi-Router improves the rank correlation between router scores and empirical expert-utility evaluations by a relative 8.1% globally, with approximately 58% relative improvement in the layer with the lowest single-router alignment.

---

Introduction and Theoretical Foundation

Background and Motivation

Sparse mixture-of-experts (MoE) models expand parameter capacity while activating only a small subset of experts per token (Shazeer et al., 2017; Fedus et al., 2021). The routing mechanism determines which experts process each token, directly affecting model capability under a fixed active-expert budget.

Key observation: Dense evaluation of all experts reveals weak agreement between single-router score rankings and rankings from empirical expert-utility evaluations, especially in middle and deep layers (Figure 2 of the paper). The Spearman correlation between single-router scores and normalized expert-output norms is only 0.1245 globally.

Self-reinforcing poor selections: Unselected experts receive no direct task-loss gradient from a given token, so poor selections may become self-reinforcing during training. This motivates assessing experts from multiple scoring perspectives rather than relying on a single router.

Limitations of Existing Approaches

Theoretical Insight

The problem is to obtain multiple assessments of experts and use routing preferences to modulate their computation while preserving computational efficiency.

The paper observes a critical representational limitation: the summed score alone does not retain routers' relative preferences. Score vectors $(0.4, 0.1)$ and $(0.2, 0.3)$ both sum to 0.5 but express opposite preferences (Figure 1). Multi-View exploits this distinction by using the relative scores to blend views.

---

Methodology

Framework Overview

The backbone is a latent-space MoE with down-projection $P \in \mathbb{R}^{d \times D}$ supplying shared input $z = Ph$. The latent expert aggregation is:

$$y = \sum_{i \in \mathcal{S}} g_i E_i(z) \tag{1}$$

where $\mathcal{S}$ is the selected expert set, $g_i$ are output gates, and $E_i$ maps $\mathbb{R}^d \to \mathbb{R}^d$.

Multi-Router: Score Aggregation Before Selection

Multi-Router uses $R$ routing heads $\{w^{(r)}\}$, $w^{(r)} \in \mathbb{R}^{D \times N}$, each producing sigmoid scores summed before top-$k$ selection:

$$\begin{aligned} \bar{s}_i &= \sum_{r=1}^{R} \sigma\big(w_i^{(r)\top} h\big), \\ \mathcal{S} &= \mathrm{TopK}_k(\bar{s} + b). \end{aligned} \tag{2}$$

Here $b \in \mathbb{R}^N$ is the expert-balancing bias, added only for selection. Output gates normalize the uncorrected score sums:

$$g_i = \alpha_i \frac{\bar{s}_i}{\sum_{j \in \mathcal{S}} \bar{s}_j}, \qquad i \in \mathcal{S} \tag{3}$$

Critical design choice: Summation happens after the sigmoid. Summing logits before the sigmoid collapses into a single linear scorer since $\sum_r w_i^{(r)\top} h = \left(\sum_r w_i^{(r)}\right)^{\top} h$. Post-sigmoid summation preserves routers as distinct nonlinear scoring functions.

Multi-View: Router-Conditioned Expert Inputs

Multi-View pairs $R$ routing heads with $K$ latent views in one-to-one correspondence ($R = K$). Views are formed as $v^{(m)} = P_m h \in \mathbb{R}^d$, with view 1 using the backbone's projection and remaining views using independently initialized perturbations.

View mixing weights are derived by normalizing each selected expert's raw sigmoid scores across routing heads:

$$a_{m,i} = \frac{s_{m,i}}{\sum_{n=1}^{R} s_{n,i}}, \qquad \sum_{m=1}^{K} a_{m,i} = 1, \quad i \in \mathcal{S} \tag{4}$$

The mixed input is the convex combination:

$$\hat{v}_i = \sum_{m} a_{m,i} v^{(m)}$$

The resulting expert aggregation:

$$y = \sum_{i \in \mathcal{S}} g_i E_i\left(\sum_{m} a_{m,i} v^{(m)}\right) \tag{5}$$

Key property: For nonlinear $E_i$, mixing before evaluation is not equivalent to mixing outputs after separate evaluations:

$$E_i\left(\sum_{m} a_{m,i} v^{(m)}\right) \neq \sum_{m} a_{m,i} E_i(v^{(m)})$$

Effective projection view: The input uses a token- and expert-dependent effective projection:

$$P_{\mathrm{eff},i}(h) = \sum_{m} a_{m,i}(h) P_m, \qquad \hat{v}_i = P_{\mathrm{eff},i}(h) h$$

The projection basis $\{P_m\}$ is shared across experts; only mixing coefficients depend on $h$ and $i$.

Initialization Strategy

To introduce additional projections gradually:

• View 1 retains backbone projection $P$
• Other views are ramped perturbations: $v^{(m)} = Ph + \gamma(t)\Delta P_m h$
• $\gamma$ increases from 0 to 1 early in training
• At $t = 0$, all view projections equal the backbone projection

Algorithm 1: Forward Pass

Require: hidden state h; projections {P_m}; routing heads {w^(m)}; 
        experts {E_i}; active count k; balancing bias b; scales {α_i}

1:  v^(m) ← P_m h for m = 1, ..., K        ▷ K latent views
2:  s_{m,i} ← σ(w_i^(m)⊤ h)                ▷ per-router scores
3:  s̄_i ← Σ_m s_{m,i}                       ▷ summed score
4:  S ← TopK_k(s̄ + b)                       ▷ bias only for selection
5:  for i ∈ S do
6:      a_{m,i} ← s_{m,i}/Σ_n s_{n,i}       ▷ normalize across routers
7:      v̂_i ← Σ_m a_{m,i} v^(m)             ▷ per-expert view blend
8:      g_i ← α_i s̄_i/Σ_{j∈S} s̄_j           ▷ output gate (Eq. 3)
9:  end for
10: return y = Σ_{i∈S} g_i E_i(v̂_i)

---

Empirical Validation / Results

Setup

• 24-layer 10B-A1B backbone with top-8 routing and one always-active shared expert
• Trained on 708B tokens
• Three configurations share training data, token budget, backbone, and optimization settings

Downstream Performance (Table 1)

At 708B training tokens, mean scores across nine tasks:

Key findings: Multi-View improves the mean by 1.18 percentage points over baseline and 0.56 over Multi-Router. The largest gains are on GPQA (+5.60 points) and MMLU-Redux (+1.90).

Training Loss Convergence

Over the final 1,000 steps:

• Baseline: 2.25621
• Multi-Router: 2.25047 (reduction of $5.74 \times 10^{-3}$)
• Multi-View: 2.24883 (reduction of $7.38 \times 10^{-3}$)

Routing Score–Utility Alignment

Table 2: Global Spearman correlation ρ

• Multi-Router improves rank correlation by a relative 8.1%
• Gains in 17 of 24 layers
• Largest relative gain at layer 20: 0.0625 → 0.0986 (57.7% relative)
• Largest absolute gain at layer 7: 0.0982 → 0.1388

Controlled Mixing Interventions

At the final MoE layer, holding $\mathcal{S}$ and output gates $g_i$ fixed to numerical precision:

Both views contribute: View 1-only proportions → ΔNLL = 0.01881; view 2-only → ΔNLL = 0.02042.

Training Ablations (Tables 3 and 4)

Critical finding: Stopping gradient flow from mixing weights to views changes loss from −0.0024 to +0.0090 (increase of 0.0114), isolating the benefit of the gradient coupling path.

Efficiency

• Multi-Router parameter overhead: ~0.06%
• Multi-View total overhead: ~0.19%
• Computational cost preserved: Each expert runs once; mixing before dispatch; same all-to-all volume
• Additional cost scales as: $D(d + N)$ for fixed $K$

---

Theoretical and Practical Implications

Theoretical Contributions

1. Routing preferences as a dual-purpose signal: The paper establishes that routing scores can serve simultaneously for expert selection AND expert input construction, expanding the role of routers in MoE architectures.

2. Non-equivalence of pre/post mixing: The mathematical framing shows that mixing views before expert evaluation ($E_i(\sum_m a_{m,i}v^{(m)})$) is fundamentally different from mixing after separate evaluations ($\sum_m a_{m,i} E_i(v^{(m)})$), justifying the design choice.

3. Post-sigmoid aggregation rationale: Summing scores after sigmoid preserves routing heads as distinct nonlinear scoring functions, preventing collapse into a single linear scorer—a subtle but crucial design decision.

Practical Implications

1. Fixed-budget enhancement: Multi-View MoE provides gains without increasing active-expert count, expert FLOPs, or communication volume—key practical constraints in distributed training.

2. Scalable approach: The number of projections depends on $K$ (routing heads), not on the number of experts $N$, and expert parameters are shared across views.

3. Warm-start compatibility: The ramped initialization procedure allows loading from existing checkpoints, facilitating adoption in deployed models.

4. Self-stabilizing router diversity: Routing heads develop highly differentiated preferences without explicit diversity penalties, suggesting emergent specialization across routing heads.

---

Conclusion

Main Takeaways

Multi-View MoE reuses routing preferences to both improve expert selection and shape expert computation. The framework addresses the observed weak agreement between single-router rankings and empirical expert utility by:

1. Multi-Router submodule: Combining independently initialized routing heads that learn highly differentiated expert preferences
2. Multi-View extension: Using relative router scores to blend latent views into expert inputs before dispatch

The 8.1% global improvement in score–utility rank correlation and 1.18 percentage point downstream gain demonstrate that routing preferences encode valuable signal beyond simple selection that can be productively reused for computation shaping.

Future Directions

The paper's limitations suggest several promising directions:

• Scaling to larger backbones and longer training runs
• Exploring more than two routing heads/views ($R = K > 2$)
• Investigating whether the differentiated router preferences can be interpreted or controlled
• Extending the router-conditioned input framework to other MoE variants (e.g., fine-grained experts)

Final insight: "Routing preferences" serve as a rich signal that can inform both which experts process a token and how they process it, establishing a new paradigm where routers do more than just select—they instruct.