Summary (Overview)

• MASKerade is a novel dense-to-MoE upcycling method that constructs Mixture-of-Experts (MoE) layers by learning binary masks over frozen pretrained feed-forward network (FFN) weights, rather than training independent expert weight matrices.
• The method jointly optimizes a bank of expert masks and a token-level router, supporting neuron-structured, semi-structured (N:M), and unstructured mask granularities within the same routing architecture.
• The primary configuration uses four 2:4 semi-structured experts with top-2 routing, where two half-dense expert passes match the nominal FFN arithmetic of one dense pass.
• On five vision-language benchmarks (GQA, MME, POPE, ScienceQA, TextVQA) with Qwen and Gemma backbones, MASKerade achieves the highest performance among all compared baselines, including replication-based, partitioning-based, and pruning-based approaches.
• Ablations demonstrate that learned connectivity (masks) is more effective than fixed importance-based pruning (Wanda, magnitude), and that the router learns meaningful token-to-expert bindings with a shared connectivity core plus expert-specific connections.

---

Introduction and Theoretical Foundation

Background

Mixture-of-Experts (MoE) models decouple model capacity from per-token computation by routing each token through a small subset of expert sub-networks. Dense-to-MoE upcycling converts existing pretrained dense models into MoE architectures by reusing the investment of dense pretraining, making this approach accessible without training from scratch.

Key Design Question

The central question MASKerade addresses is: what constitutes an expert during upcycling? Existing approaches:

• Replication-based: Copy the same FFN into multiple experts with independent weights (often with partial reinitialization, e.g., Drop-Upcycling).
• Partitioning/pruning-based: Construct smaller experts by partitioning neurons (e.g., MoEfication, LLaMA-MoE) or pruning with calibration data (e.g., DIVE).
• Adapter-based: Add expert-specific low-rank updates or adapters.

Theoretical Foundation

MASKerade draws from two research threads:

1. Supermask learning and learnable pruning: Learning binary masks over frozen weights can change a network's function without updating the weights themselves (Ramanujan et al., 2020; Fang et al., 2024; Zhang et al., 2026).
2. Sparse routing in MoE: Token-level routers select experts based on input hidden states (Shazeer et al., 2017; Fedus et al., 2022).

MASKerade's key insight: multiple masks can coexist over the same frozen weights, and a router can select among alternative sparse computations, replacing expert-specific weight adaptation with expert-specific connectivity learning.

---

Methodology

1. Experts as Frozen-Weight Subnetworks

Let $\theta_0$ denote the pretrained dense model and $\mathcal{T}$ the set of FFN layers to convert. For a converted layer $\ell$, the frozen FFN parameters are $W_\ell$. MASKerade defines each expert as:

$$\widetilde{W}_{\ell,e}^{(a)} = W_{\ell}^{(a)} \odot M_{\ell,e}^{(a)}, \qquad M_{\ell,e}^{(a)} \in \{0, 1\}^{\text{shape}(W_{\ell}^{(a)})} \tag{1}$$

where $a$ indexes the FFN projections (gate, up, down) and $\odot$ is elementwise multiplication. For gated FFNs with $W^{(g)}, W^{(u)} \in \mathbb{R}^{d_{\mathrm{ff}} \times d}$ and $W^{(d)} \in \mathbb{R}^{d \times d_{\mathrm{ff}}}$, expert $e$ computes:

$$F_{\ell,e}(h) = \widetilde{W}_{\ell,e}^{(d)} \left[ \phi_{\ell}\left(\widetilde{W}_{\ell,e}^{(g)} h\right) \odot \left(\widetilde{W}_{\ell,e}^{(u)} h\right) \right] \tag{2}$$

The pretrained parameters remain frozen throughout upcycling.

2. Token-Level Routing

A trainable linear router $R_\ell \in \mathbb{R}^{E \times d}$ maps each token's hidden state to probabilities:

$$p_{\ell,e}(h) = \text{softmax}(R_\ell h)_e$$

Deterministic top-$k$ selection gives the active expert set $\mathcal{S}_\ell(h) = \text{TopK}(p_\ell(h), k)$, and outputs are combined as:

$$F_{\ell}^{\mathrm{MoE}}(h) = \sum_{e \in \mathcal{S}_\ell(h)} g_{\ell,e}(h) F_{\ell,e}(h), \qquad g_{\ell,e}(h) = \begin{cases} p_{\ell,e}(h), & k = 1, \\ \frac{p_{\ell,e}(h)}{\sum_{j \in \mathcal{S}_\ell(h)} p_{\ell,j}(h)}, & k \geq 2. \end{cases} \tag{3}$$

3. Mask Parameterization and Granularity

Masks are derived from continuous trainable scores $S$ using a hard selection operator $\mathcal{H}$, so $M = \mathcal{H}(S)$ is binary. The retained fraction is denoted by $\rho$. Three families:

Semi-structured (N:M): Each row is partitioned into contiguous groups $B$ of $M$ entries; the $N$ largest score magnitudes are kept:

$$M_{\ell,e,i}^{(a)} = \mathbf{1}\left[ i \in \mathrm{TopN}_{j \in B} \left| S_{\ell,e,j}^{(a)} \right| \right], \quad \sum_{i \in B} M_{\ell,e,i}^{(a)} = N \tag{4}$$

Unstructured: Select the $q = \lceil \rho P \rceil$ largest score magnitudes over all flattened positions of the projection tensor.

Structured (neuron-level): Each expert learns a score vector over intermediate neurons; the mask is shared across gate/up rows and down columns:

$$M_{\ell,e}^{(g)} = M_{\ell,e}^{(u)} = m_{\ell,e} \mathbf{1}_d^{\top}, \qquad M_{\ell,e}^{(d)} = \mathbf{1}_d m_{\ell,e}^{\top} \tag{5}$$

4. Joint Mask and Router Training

Score initialization: Importance initialization uses pretrained weight magnitudes with expert-specific Gaussian noise:

$$S_{\ell,e}^{(a)} = |W_{\ell}^{(a)}| + \epsilon_{\ell,e}^{(a)}, \qquad \epsilon_{\ell,e,i}^{(a)} \sim \mathcal{N}\left(0, \left[\alpha_{\mathrm{init}} \, \mathrm{std}(|W_{\ell}^{(a)}|)\right]^2\right) \tag{6}$$

Gradient estimator: Hard masks are non-differentiable, so straight-through estimation propagates gradients:

$$\frac{\partial \mathcal{L}}{\partial M_{\ell,e}^{(a)}} = W_{\ell}^{(a)} \odot \frac{\partial \mathcal{L}}{\partial \widetilde{W}_{\ell,e}^{(a)}} \tag{7}$$

Training objective:

$$\min_{\{S_\ell, R_\ell\}_{\ell \in \mathcal{I}}} \mathcal{L}_{\mathrm{SFT}}(\theta_0, S, R) + \lambda_{\mathrm{bal}} \sum_{\ell \in \mathcal{I}} \mathcal{L}_{\mathrm{bal}}^{(\ell)}, \qquad \theta_0 \text{ fixed} \tag{8}$$

with probability-based load balancing:

$$\bar{p}_{\ell,e} = \frac{1}{T}\sum_{t=1}^{T} p_{\ell,e}(h_t), \quad f_{\ell,e} = \frac{1}{T}\sum_{t=1}^{T} \mathbf{1}[e \in \mathcal{S}_\ell(h_t)], \quad \mathcal{L}_{\mathrm{bal}}^{(\ell)} = E \sum_{e=1}^{E} \bar{p}_{\ell,e} f_{\ell,e} \tag{9}$$

---

Empirical Validation / Results

Main Results on Small Models (Qwen3-1.7B, Gemma3-1B)

Table 1: Main comparison on small models (key results, full table in paper)

MASKerade achieves the highest performance on all five benchmarks while matching the nominal activated parameter count of the Dense MLP baseline (1.72B), which is lower than most MoE baselines (e.g., Drop-Upcycling and MoE-LLaVA use 2.25B).

Main Results on Large Models (Qwen3.6-27B, Gemma4-31B)

On Qwen3.6-27B, MASKerade achieves: GQA 66.37, MME 1724.9, POPE 87.68, SQA 94.79, TextVQA 81.08 — again best on all metrics among 12+ baselines with only 26.90B activated parameters (vs. 35.45B for MoE-LLaVA and Drop-Upcycling).

Mask-Based Comparison

Table 3: Learned vs. fixed mask construction on Qwen3-1.7B (excerpt)

Learning masks significantly outperforms fixed-mask construction based on random, magnitude, or Wanda criteria.

Ablations

Router learning matters (Table 4): Trained routers outperform random routers by +2.0 GQA points; label-permuted routing (breaking the mask-to-expert binding) degrades performance further.

Importance initialization is crucial (Table 5): Importance initialization outperforms random initialization for all mask families, with especially large gains for 2:4 masks (62.08 vs. 54.44 GQA). The importance-permuted control shows this is due to location-specific information.

More retention helps (Table 6): Performance improves monotonically with per-expert retention from 25% → 75% for unstructured and semi-structured masks.

Matched budget (Table 7): At matched activation budgets (1.46B), MASKerade with 4e/top-1 (2:4) achieves 61.85 GQA, outperforming ToMoE (58.54), DIVE (57.87), and LLaMA-MoE (57.85) at the same budget.

Inference speed (Table 8): MASKerade's SparseLUT kernel is 21-25% faster than the vanilla MoE implementation at the large model shapes (0.79× and 0.75× relative latency).

Mask structure (Figure 3): Learned 2:4 masks have ~0.59 Jaccard overlap — meaning experts are overlapping but non-identical. Semi-structured and unstructured masks share a large "latent shared expert" core (all-four intersection over half of retained entries), while structured masks have more expert-specific connectivity.

---

Theoretical and Practical Implications

Theoretical Implications

1. Connectivity learning is sufficient: MASKerade demonstrates that effective dense-to-MoE adaptation can be achieved without updating pretrained weights — only learning which connections to use. This challenges the assumption that expert diversity requires independent parameter updates.

2. Mask granularity matters: The comparison across granularities shows semi-structured and unstructured masks outperform structured neuron-level masks, suggesting that fine-grained connectivity selection is more expressive for expert specialization than coarse neuron selection.

3. Shared core + expert-specific connections: The learned masks contain a substantial shared connectivity core alongside expert-specific connections, suggesting a natural decomposition between shared computation and specialized computation.

4. Routing is essential: The router learns meaningful, source-dependent, layer-local selection patterns, and its learned binding to specific masks is crucial for performance.

Practical Implications

1. Parameter efficiency: MASKerade matches dense activation budgets (1.72B for Qwen3-1.7B) while achieving higher performance than baselines using more parameters (2.25B).

2. Memory efficiency: Since the pretrained FFN weights remain frozen, MASKerade avoids storing multiple independent expert weight matrices — only the masks and router parameters need storage.

3. Hardware acceleration of 2:4 sparsity: The semi-structured configuration is compatible with NVIDIA GPU 2:4 sparse computation, enabling latency reductions of 21-25% on large models.

4. Wide applicability: The method works across backbones (Qwen and Gemma), model sizes (1B to 31B), and vision-language tasks, suggesting general utility for multimodal instruction tuning.

---

Conclusion

MASKerade establishes that learned binary masks over frozen pretrained FFN weights can serve as effective token-routed MoE experts, achieving state-of-the-art performance on five vision-language benchmarks while matching dense-model activation budgets. The method supports multiple mask granularities, benefits strongly from importance-based initialization, and learns meaningful router-mask bindings that adapt expert utilization to input tokens.

Future work will focus on:

• Developing inference kernels that reuse overlapping computations across the shared connectivity core while preserving expert-specific nonlinear transformations.
• Evaluating end-to-end latency and memory trade-offs of dedicated sparse kernels.
• Extending the approach to additional architectures and task domains beyond vision-language instruction tuning.