Summary (Overview)

• CIPHER-MoE is a plug-and-play algorithm for efficient and stable trillion-parameter Mixture-of-Experts (MoE) LLM training that mitigates workload imbalance without requiring additional hardware resources or complex runtime orchestration.
• The method applies affinity-aware Expert-to-Token filtering with explicit capacity control, keeping the router's token-side Top-K selection unchanged while bounding expert workloads.
• Theoretical analysis establishes a sufficient expert capacity bound for similarity-based token admission and a necessary capacity bound for natural-order admission, proving that similarity-based selection reduces required capacity.
• Empirical validation on models from 284B to 1.6T parameters (DeepSeek-V3, DeepSeek-V4-Flash, GLM-5, DeepSeek-V4-Pro) demonstrates 1.10×–1.94× training speedups and up to 64.9 percentage points Top-1 expert workload reduction while preserving training quality.
• Strict token dropping outperforms token–expert rerouting for domain-specific training, as rerouting introduces gradient interference that undermines expert specialization.

---

Introduction and Theoretical Foundation

Background and Motivation

Large language models (LLMs) have become foundational to modern AI systems, with scaling laws demonstrating that increasing model capacity substantially improves performance. However, naively scaling dense Transformers to trillion-parameter regimes incurs prohibitive computational and memory costs. Mixture-of-Experts (MoE) architectures address this by decoupling model capacity from per-token computation: they replace the dense feed-forward network (FFN) with multiple sparsely activated experts, dynamically routing each token to only a small subset of them.

The Workload Imbalance Problem

Training MoE models introduces substantial system-level challenges due to workload imbalance. The inherently non-uniform data distribution across domains and tasks induces highly skewed token routing, resulting in uneven workloads across experts. As shown in Figure 1, with domain-specific datasets, 10% of experts consume over 88% of routing workload in DeepSeek-V4-Flash. This imbalance causes:

• Degraded communication efficiency and hardware utilization
• Underloaded experts wasting resources while hot experts require additional capacity
• Excessive memory consumption on overloaded experts, potentially triggering out-of-memory (OOM) failures—the DeepSeek-V4-Pro baseline fails before the first training iteration with OOM

Limitations of Existing Approaches

Key Research Question

How to achieve efficient and stable training for trillion-parameter MoE LLMs without intricate system design or heuristic-based routing?

Theoretical Foundation: Bounded Expert Capacity

The core theoretical insight is that experts have a bounded learning capacity, making it feasible to drop tokens from expert training. The paper reveals the relationship between optimal token budget and training efficiency, proving that similarity-based selection can retain informative tokens with less expert capacity than natural-order processing.

---

Methodology

Preliminaries: MoE Routing

An MoE layer replaces the dense FFN with $E$ expert networks $\{f_e(\cdot; \theta_e)\}_{e=1}^{E}$ and a router that activates only $K \ll E$ experts per token. For $N$ token representations $\{\mathbf{x}_i\}_{i=1}^{N}$ with $\mathbf{x}_i \in \mathbb{R}^d$, the router computes token–expert affinity:

$$s_{i,e} = \mathbf{w}_e^{\top}\mathbf{x}_i, \qquad p_{i,e} = \frac{\exp(s_{i,e})}{\sum_{j=1}^{E}\exp(s_{i,j})}, \qquad \mathcal{S}_i = \mathrm{TopK}_{e \in \{1,\ldots,E\}}(p_{i,e}).\tag{1}$$

The routed MoE output is:

$$\mathbf{y}_i = \sum_{e=1}^{E} g_{i,e} f_e(\mathbf{x}_i; \theta_e), \quad \sum_{e=1}^{E} a_{i,e} = K.\tag{2}$$

CIPHER-MoE: Bidirectional Expert Workload Balancing

The method computes cosine similarity between token hidden states and expert gating vectors:

$$c_{i,e} = \frac{\mathbf{w}_e^{\top}\mathbf{x}_i}{\|\mathbf{w}_e\|_2 \|\mathbf{x}_i\|_2}.\tag{4}$$

For each expert $e$, CIPHER-MoE ranks candidate tokens by similarity, applies a threshold $\tau \in [-1, 1]$ to exclude low-similarity tokens, and enforces an expert capacity $C_e$:

$$\mathcal{T}_e^{\mathrm{S}} = \mathrm{Top}_{\min\{C_e, |\mathcal{P}_e^{\tau}|\}}\left(\{c_{i,e}: i \in \mathcal{P}_e^{\tau}\}\right), \quad b_{i,e}^{\mathrm{S}} = \mathbf{1}[i \in \mathcal{T}_e^{\mathrm{S}}].\tag{5}$$

Theoretical Result: Capacity Reduction Theorem

Theorem 1 (Similarity-based Capacity Reduction): Under assumptions A1–A3, with $d \geq 2$, define:

$$\ell = \log\frac{1}{\varepsilon - \eta}, \quad G(d, \delta) = -\frac{d-1}{2}\log(1-\delta^2), \quad B_e = \min\left\{N_e, \left\lceil 1 + \mu + \frac{\ell}{3} + \sqrt{2\mu\ell + \frac{\ell^2}{9}}\right\rceil\right\}.\tag{7}$$

Then:

$$C_{\text{CIPHER}}^{*} \leq B_e, \quad C_{\text{natural}}^{*} \geq \frac{(1-\varepsilon)a_0}{\beta E} e^{G(d,\delta)}.\tag{8}$$

This guarantees a factor-$\kappa$ capacity reduction ($C_{\text{CIPHER}}^{*} \leq C_{\text{natural}}^{*}/\kappa$ for $\kappa > 1$) whenever the natural-order lower bound is at least $\kappa B_e$.

Strict Token Drop vs. Token–Expert Reroute

Two strategies handle rejected assignments:

• Strict Token Drop: Removes rejected assignments, sets routing weight to zero, retains other accepted expert paths
• Token–Expert Reroute: Redirects rejected assignments to candidate experts satisfying similarity threshold with available capacity

Key finding: Strict token dropping outperforms rerouting for domain-specific training, as rerouting shifts token distributions and induces gradient interference that undermines expert specialization.

Training Objective

$$\mathcal{L}_{\mathrm{CIPHER}}^{\rho}(\Theta) = -\frac{1}{|\mathcal{U}|}\sum_{t \in \mathcal{U}} \log p_{\Theta}^{\rho}(z_t | z_{<t}) + \lambda_{\mathrm{loc}} \sum_{\ell \in \mathcal{M}} \mathcal{L}_{\mathrm{loc}}^{\rho,\ell}(\Theta),\tag{9}$$

where the locality loss (from LocMoE) is:

$$\mathcal{L}_{\mathrm{loc}}^{\rho,\ell} = \mathrm{KL}\left(D_c^{\rho,\ell} \| D_l^{\rho,\ell}\right).\tag{10}$$

---

Empirical Validation / Results

Training Efficiency and Memory Usage

CIPHER-MoE achieves 1.42× speedups on DeepSeek-V4-Flash compared to baselines, with memory limited to approximately 87.5% of device memory capacity. The computational overhead of strict and reroute policies accounts for only 1.57% and 3.85% of total training time, respectively.

Quality–Efficiency Tradeoff Across Models

Table 2: Domain-specific SFT evaluation results and training speedups

DeepSeek-V4-Pro (1.6T Parameters)

Table 3: DeepSeek-V4-Pro training efficiency and OR-domain evaluation

OR benchmark results:

CIPHER-MoE enables successful training where the baseline fails with OOM, and outperforms the fixed-router baseline on multiple benchmarks.

Expert Specialization Analysis

CIPHER-MoE promotes stronger expert differentiation and avoids homogenization. The CIPHER-strict variant exhibits the clearest separation in router adaptation, with the agreement between update dynamics and routed-representation geometry indicating a persistent division of labor rather than transient routing fluctuations.

---

Theoretical and Practical Implications

Theoretical Contributions

1. First capacity-reduction theorem for MoE training: Establishes that similarity-based token selection requires provably less expert capacity than natural-order processing to retain informative tokens, with the sufficient bound $B_e$ and natural-order lower bound given in Theorem 1.

2. Bounded learning capacity insight: Reveals that experts have a bounded learning capacity, making token dropping not only feasible but beneficial—contrary to the intuition that more computation always improves quality.

3. Strict drop vs. reroute analysis: Demonstrates theoretically and empirically that strict token dropping outperforms rerouting for domain-specific training, since cosine affinity alone does not constrain the direction of task gradients.

Practical Implications

• Scalability to trillion parameters: CIPHER-MoE successfully trains DeepSeek-V4-Pro (1.6T) where vanilla MoE training fails with OOM errors, and supports global batch sizes up to 8192.
• Plug-and-play deployment: Requires no expert replication, parameter migration, or complex runtime orchestration—only capacity-aware token selection integrated into existing MoE training pipelines.
• Broad applicability: Works across diverse model architectures (DeepSeek-V3, V4-Flash, V4-Pro, GLM-5) and domain-specific datasets (operations research, medical knowledge, social interactions).
• Quality preservation: Achieves optimal or near-optimal quality on domain-specific benchmarks while also preserving general knowledge capabilities (validated on robustness datasets in Appendix A.3).

---

Conclusion

CIPHER-MoE addresses the expert workload imbalance that limits the efficiency and stability of MoE training through a simple, theoretically grounded approach:

• Simplicity: Directly exploits bounded token budgets for MoE training, achieving high efficiency while preserving LLM training quality
• Theoretical Foundation: Derives a sufficient expert capacity for similarity-based admission and a necessary capacity for natural-order admission
• Scalability: Scales effectively across model sizes from 284B to 1.6T parameters, achieving 1.10×–1.94× measured training-time speedups

The work demonstrates that capacity-aware token selection provides a simple, efficient, stable, and scalable foundation for large-scale MoE training, eliminating the need for intricate system design or heuristic-based routing. Future directions may include extending the capacity analysis to other training phases (e.g., pretraining, RL) and exploring adaptive capacity scheduling across training stages. The source code will be released to facilitate further research and adoption.