# MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling

> MASKerade learns binary masks over frozen pretrained feedforward weights to create MoE experts, beating all baselines on five vision-language benchmarks at matched activation budgets.

- **Source:** [arXiv](https://arxiv.org/abs/2610.07809)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/qIofUg
- **Whiteboard:** https://picx.dev/p/qIofUg/image

## Summary

## Summary (Overview)

- **MASKerade** is a novel dense-to-MoE upcycling method that constructs Mixture-of-Experts (MoE) layers by learning binary masks over **frozen** pretrained feed-forward network (FFN) weights, rather than training independent expert weight matrices.
- The method jointly optimizes a bank of expert masks and a token-level router, supporting neuron-structured, semi-structured (N:M), and unstructured mask granularities within the same routing architecture.
- The primary configuration uses **four 2:4 semi-structured experts with top-2 routing**, where two half-dense expert passes match the nominal FFN arithmetic of one dense pass.
- On five vision-language benchmarks (GQA, MME, POPE, ScienceQA, TextVQA) with Qwen and Gemma backbones, MASKerade achieves the **highest performance among all compared baselines**, including replication-based, partitioning-based, and pruning-based approaches.
- Ablations demonstrate that learned connectivity (masks) is more effective than fixed importance-based pruning (Wanda, magnitude), and that the router learns meaningful token-to-expert bindings with a shared connectivity core plus expert-specific connections.

---

## Introduction and Theoretical Foundation

### Background
Mixture-of-Experts (MoE) models decouple model capacity from per-token computation by routing each token through a small subset of expert sub-networks. Dense-to-MoE upcycling converts existing pretrained dense models into MoE architectures by reusing the investment of dense pretraining, making this approach accessible without training from scratch.

### Key Design Question
The central question MASKerade addresses is: **what constitutes an expert during upcycling?** Existing approaches:
- **Replication-based**: Copy the same FFN into multiple experts with independent weights (often with partial reinitialization, e.g., Drop-Upcycling).
- **Partitioning/pruning-based**: Construct smaller experts by partitioning neurons (e.g., MoEfication, LLaMA-MoE) or pruning with calibration data (e.g., DIVE).
- **Adapter-based**: Add expert-specific low-rank updates or adapters.

### Theoretical Foundation
MASKerade draws from two research threads:
1. **Supermask learning and learnable pruning**: Learning binary masks over frozen weights can change a network's function without updating the weights themselves (Ramanujan et al., 2020; Fang et al., 2024; Zhang et al., 2026).
2. **Sparse routing in MoE**: Token-level routers select experts based on input hidden states (Shazeer et al., 2017; Fedus et al., 2022).

MASKerade's key insight: *multiple masks can coexist over the same frozen weights, and a router can select among alternative sparse computations, replacing expert-specific weight adaptation with expert-specific connectivity learning.*

---

## Methodology

### 1. Experts as Frozen-Weight Subnetworks

Let $\theta_0$ denote the pretrained dense model and $\mathcal{T}$ the set of FFN layers to convert. For a converted layer $\ell$, the frozen FFN parameters are $W_\ell$. MASKerade defines each expert as:

$$
\widetilde{W}_{\ell,e}^{(a)} = W_{\ell}^{(a)} \odot M_{\ell,e}^{(a)}, \qquad M_{\ell,e}^{(a)} \in \{0, 1\}^{\text{shape}(W_{\ell}^{(a)})} \tag{1}
$$

where $a$ indexes the FFN projections (gate, up, down) and $\odot$ is elementwise multiplication. For gated FFNs with $W^{(g)}, W^{(u)} \in \mathbb{R}^{d_{\mathrm{ff}} \times d}$ and $W^{(d)} \in \mathbb{R}^{d \times d_{\mathrm{ff}}}$, expert $e$ computes:

$$
F_{\ell,e}(h) = \widetilde{W}_{\ell,e}^{(d)} \left[ \phi_{\ell}\left(\widetilde{W}_{\ell,e}^{(g)} h\right) \odot \left(\widetilde{W}_{\ell,e}^{(u)} h\right) \right] \tag{2}
$$

The pretrained parameters remain **frozen** throughout upcycling.

### 2. Token-Level Routing

A trainable linear router $R_\ell \in \mathbb{R}^{E \times d}$ maps each token's hidden state to probabilities:

$$p_{\ell,e}(h) = \text{softmax}(R_\ell h)_e$$

Deterministic top-$k$ selection gives the active expert set $\mathcal{S}_\ell(h) = \text{TopK}(p_\ell(h), k)$, and outputs are combined as:

$$
F_{\ell}^{\mathrm{MoE}}(h) = \sum_{e \in \mathcal{S}_\ell(h)} g_{\ell,e}(h) F_{\ell,e}(h), \qquad g_{\ell,e}(h) = \begin{cases} p_{\ell,e}(h), & k = 1, \\ \frac{p_{\ell,e}(h)}{\sum_{j \in \mathcal{S}_\ell(h)} p_{\ell,j}(h)}, & k \geq 2. \end{cases} \tag{3}
$$

### 3. Mask Parameterization and Granularity

Masks are derived from continuous trainable scores $S$ using a hard selection operator $\mathcal{H}$, so $M = \mathcal{H}(S)$ is binary. The retained fraction is denoted by $\rho$. Three families:

**Semi-structured (N:M)**: Each row is partitioned into contiguous groups $B$ of $M$ entries; the $N$ largest score magnitudes are kept:

$$
M_{\ell,e,i}^{(a)} = \mathbf{1}\left[ i \in \mathrm{TopN}_{j \in B} \left| S_{\ell,e,j}^{(a)} \right| \right], \quad \sum_{i \in B} M_{\ell,e,i}^{(a)} = N \tag{4}
$$

**Unstructured**: Select the $q = \lceil \rho P \rceil$ largest score magnitudes over all flattened positions of the projection tensor.

**Structured (neuron-level)**: Each expert learns a score vector over intermediate neurons; the mask is shared across gate/up rows and down columns:

$$
M_{\ell,e}^{(g)} = M_{\ell,e}^{(u)} = m_{\ell,e} \mathbf{1}_d^{\top}, \qquad M_{\ell,e}^{(d)} = \mathbf{1}_d m_{\ell,e}^{\top} \tag{5}
$$

### 4. Joint Mask and Router Training

**Score initialization**: Importance initialization uses pretrained weight magnitudes with expert-specific Gaussian noise:

$$
S_{\ell,e}^{(a)} = |W_{\ell}^{(a)}| + \epsilon_{\ell,e}^{(a)}, \qquad \epsilon_{\ell,e,i}^{(a)} \sim \mathcal{N}\left(0, \left[\alpha_{\mathrm{init}} \, \mathrm{std}(|W_{\ell}^{(a)}|)\right]^2\right) \tag{6}
$$

**Gradient estimator**: Hard masks are non-differentiable, so straight-through estimation propagates gradients:

$$
\frac{\partial \mathcal{L}}{\partial M_{\ell,e}^{(a)}} = W_{\ell}^{(a)} \odot \frac{\partial \mathcal{L}}{\partial \widetilde{W}_{\ell,e}^{(a)}} \tag{7}
$$

**Training objective**:

$$
\min_{\{S_\ell, R_\ell\}_{\ell \in \mathcal{I}}} \mathcal{L}_{\mathrm{SFT}}(\theta_0, S, R) + \lambda_{\mathrm{bal}} \sum_{\ell \in \mathcal{I}} \mathcal{L}_{\mathrm{bal}}^{(\ell)}, \qquad \theta_0 \text{ fixed} \tag{8}
$$

with probability-based load balancing:

$$
\bar{p}_{\ell,e} = \frac{1}{T}\sum_{t=1}^{T} p_{\ell,e}(h_t), \quad f_{\ell,e} = \frac{1}{T}\sum_{t=1}^{T} \mathbf{1}[e \in \mathcal{S}_\ell(h_t)], \quad \mathcal{L}_{\mathrm{bal}}^{(\ell)} = E \sum_{e=1}^{E} \bar{p}_{\ell,e} f_{\ell,e} \tag{9}
$$

---

## Empirical Validation / Results

### Main Results on Small Models (Qwen3-1.7B, Gemma3-1B)

**Table 1: Main comparison on small models** (key results, full table in paper)

| Method (Qwen3-1.7B) | Act. Params | GQA ↑ | MME ↑ | POPE ↑ | SQA ↑ | TextVQA ↑ |
|---|---|---|---|---|---|---|
| Dense MLP | 1.72B | 58.41 | 1300.4 | 85.01 | 61.16 | 49.19 |
| Drop-Upcycling | 2.25B | 60.77 | 1312.9 | 85.66 | 61.60 | 50.82 |
| MoE-LLaVA | 2.25B | 60.06 | 1315.3 | 85.19 | 62.07 | 51.36 |
| **MASKerade (2:4)** | **1.72B** | **62.00** | **1332.4** | **86.49** | **63.48** | **52.50** |

MASKerade achieves the **highest performance on all five benchmarks** while matching the nominal activated parameter count of the Dense MLP baseline (1.72B), which is **lower than most MoE baselines** (e.g., Drop-Upcycling and MoE-LLaVA use 2.25B).

### Main Results on Large Models (Qwen3.6-27B, Gemma4-31B)

On Qwen3.6-27B, MASKerade achieves: GQA 66.37, MME 1724.9, POPE 87.68, SQA 94.79, TextVQA 81.08 — again **best on all metrics** among 12+ baselines with only 26.90B activated parameters (vs. 35.45B for MoE-LLaVA and Drop-Upcycling).

### Mask-Based Comparison

**Table 3: Learned vs. fixed mask construction on Qwen3-1.7B** (excerpt)

| Method | GQA | MME | POPE | SQA | TextVQA |
|---|---|---|---|---|---|
| Random 2:4 | 43.36 | 871.8 | 82.15 | 40.75 | 28.22 |
| Magnitude 2:4 | 51.54 | 1127.6 | 85.38 | 58.34 | 43.15 |
| Wanda 2:4 | 51.97 | 1148.2 | 83.96 | 60.81 | 44.01 |
| **MASKerade Structural** | 58.81 | 1302.0 | 85.06 | 61.58 | 51.05 |
| **MASKerade Unstructured** | 62.06 | 1315.8 | 86.27 | 63.71 | 52.83 |
| **MASKerade Semi-structured 2:4** | **62.00** | **1332.4** | **86.49** | **63.48** | **52.50** |

Learning masks significantly outperforms fixed-mask construction based on random, magnitude, or Wanda criteria.

### Ablations

**Router learning matters** (Table 4): Trained routers outperform random routers by +2.0 GQA points; label-permuted routing (breaking the mask-to-expert binding) degrades performance further.

**Importance initialization is crucial** (Table 5): Importance initialization outperforms random initialization for all mask families, with especially large gains for 2:4 masks (62.08 vs. 54.44 GQA). The importance-permuted control shows this is due to location-specific information.

**More retention helps** (Table 6): Performance improves monotonically with per-expert retention from 25% → 75% for unstructured and semi-structured masks.

**Matched budget** (Table 7): At matched activation budgets (1.46B), MASKerade with 4e/top-1 (2:4) achieves 61.85 GQA, outperforming ToMoE (58.54), DIVE (57.87), and LLaMA-MoE (57.85) at the same budget.

**Inference speed** (Table 8): MASKerade's SparseLUT kernel is **21-25% faster** than the vanilla MoE implementation at the large model shapes (0.79× and 0.75× relative latency).

**Mask structure** (Figure 3): Learned 2:4 masks have ~0.59 Jaccard overlap — meaning experts are **overlapping but non-identical**. Semi-structured and unstructured masks share a large "latent shared expert" core (all-four intersection over half of retained entries), while structured masks have more expert-specific connectivity.

---

## Theoretical and Practical Implications

### Theoretical Implications
1. **Connectivity learning is sufficient**: MASKerade demonstrates that effective dense-to-MoE adaptation can be achieved *without updating pretrained weights* — only learning which connections to use. This challenges the assumption that expert diversity requires independent parameter updates.

2. **Mask granularity matters**: The comparison across granularities shows semi-structured and unstructured masks outperform structured neuron-level masks, suggesting that **fine-grained connectivity selection** is more expressive for expert specialization than coarse neuron selection.

3. **Shared core + expert-specific connections**: The learned masks contain a substantial shared connectivity core alongside expert-specific connections, suggesting a natural decomposition between shared computation and specialized computation.

4. **Routing is essential**: The router learns meaningful, source-dependent, layer-local selection patterns, and its learned binding to specific masks is crucial for performance.

### Practical Implications
1. **Parameter efficiency**: MASKerade matches dense activation budgets (1.72B for Qwen3-1.7B) while achieving higher performance than baselines using more parameters (2.25B).

2. **Memory efficiency**: Since the pretrained FFN weights remain frozen, MASKerade avoids storing multiple independent expert weight matrices — only the masks and router parameters need storage.

3. **Hardware acceleration of 2:4 sparsity**: The semi-structured configuration is compatible with NVIDIA GPU 2:4 sparse computation, enabling **latency reductions of 21-25%** on large models.

4. **Wide applicability**: The method works across backbones (Qwen and Gemma), model sizes (1B to 31B), and vision-language tasks, suggesting general utility for multimodal instruction tuning.

---

## Conclusion

MASKerade establishes that **learned binary masks over frozen pretrained FFN weights can serve as effective token-routed MoE experts**, achieving state-of-the-art performance on five vision-language benchmarks while matching dense-model activation budgets. The method supports multiple mask granularities, benefits strongly from importance-based initialization, and learns meaningful router-mask bindings that adapt expert utilization to input tokens.

Future work will focus on:
- Developing inference kernels that **reuse overlapping computations** across the shared connectivity core while preserving expert-specific nonlinear transformations.
- Evaluating end-to-end latency and memory trade-offs of dedicated sparse kernels.
- Extending the approach to additional architectures and task domains beyond vision-language instruction tuning.

---

_Markdown view of https://picx.dev/p/qIofUg, served by PicX — AI-generated visual whiteboard summaries of research papers._