MoE & Sparse ExpertsIssue 6Sep 12 – 19, 2026

The Cost of Routing Skew Relocated: From Memory Peaks to Exploration Gains

Highlights of This Issue

The main theme of this issue is that "the cost of routing skew is being relocated": three independent works on the training side, RL side, and inference side, from different perspectives, rewrite "non-uniform routing" from a burden to be eliminated into a property that can be absorbed or utilized.

On the training side, Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training for the first time unifies the four memory peaks of MoE long-context training (expert distribution, vocabulary projection, gradient checkpoint boundaries, optimizer states) into a "bounded streaming operator" framework, where each operator provides a fixed working set guarantee related to launch configuration. Its core increment, PipelinedLLEP, decouples the receiver buffer size from the routing matrix via a source-side token budget—routing skew no longer amplifies memory but stretches scheduling; strided chunking further eliminates local imbalance introduced by chunking. This contrasts with the negative result of Cacheable by Design in Issue 5 and the online placement of Director in Issue 4: this paper, while keeping routing decisions unchanged, dissolves the cost of routing skew from the memory side, and trains with 1M context on 120B–667B models, reporting complete routing statistics and dense baselines.

On the RL side, Expert-Space Exploration in MoE Reinforcement Learning flips routing perturbation from a "mismatch to be eliminated" into a "dimension worth actively exploring": during rollout, it adaptively perturbs logits based on router entropy, anchors on high-confidence experts, and samples from a candidate pool, while recording expert paths and replaying them during policy optimization to maintain training-inference consistency. Compared to PR² and ReLibra in Issue 2, which focus on eliminating mismatch, this paper combines perturbation-replay into a controlled exploration mechanism, improving Pass@1 by 3.2 points over GRPO on Qwen3-30B-A3B, and covering three routing structures: top-K, top-1, and shared experts. On the inference side, ACE advances the contribution estimation for expert skipping to a purely parameter-driven dual-perspective conservative intersection criterion, requiring no calibration data, and at a 50% skip rate reduces PPL by 7.96% on Qwen3.6-35B compared to the strongest baseline—contrasting with the global k-reduction of k1/k2 in Training-Free Halving from Issue 5, this is a token-wise adaptive counterpart.

Additionally, Data Scarcity and Model Sparsity is the first systematic study of data repetition's effect on MoE overfitting, finding that MoE is more sensitive to repeated data than dense models, determined by total parameters rather than activated parameters, with mechanisms of early routing ossification and expert specialization increasing with repetition rate; Towards a Statistical Understanding of Mixture-of-Experts is the first to formalize MoE as a localized aggregation class and provide a unified nonparametric theoretical framework, decomposing risk into approximation error, expert learning error, and router estimation cost. These two works provide new baselines for the sparsity-granularity trade-off and expert specialization from empirical and theoretical perspectives, respectively.

Community and Updates

On the evidence side for expert specialization, Do Domain-specific Experts exist in MoE-based LLMs? systematically examines whether domain experts exist across ten MoE models ranging from 3.8B to 120B, and proposes training-free domain-specific expert activation (DSMoE), which surpasses strong baselines including SFT at zero additional inference cost—its conclusion that "training-free steering outperforms SFT" provides multi-scale positive evidence for the causal proof threshold of How Modular Is a Frontier Mixture-of-Experts from Issue 1. Meanwhile, Cognaptus's analysis reframes the benefits of sparse routing as inspectability rather than efficiency, and explicitly calls for completing causal intervention validation and OOD safety guarantees in expert specialization claims—consistent with this direction's insistence on proof standards.

Open Questions

  1. PipelinedLLEP makes routing skew only stretch scheduling without amplifying memory—if the memory cost of skew is eliminated, does it weaken the motivation for load balancing losses (such as Quantile Balancing in Kimi K3 from Issue 1) on the training side? Where is the boundary between "skew cost can be absorbed by scheduling" and "skew still needs balancing"?

  2. ESRL treats routing perturbation as an exploration dimension, while Router Prior Bias and Training-Free Halving from Issue 5 emphasize that routing structure needs to be preserved—does actively perturbing routing conflict with preserving routing structure? Is there a quantifiable safe interval between exploration intensity and structure preservation?

  3. ACE uses purely parameter-based statistical estimation for expert contribution, but the threshold still requires a one-time unlabeled forward mapping budget—how does this budget compare to the cost of routing replay methods (such as ReLibra from Issue 2)? Can parameter-driven contribution estimation be unified with routing replay?

  4. Data Scarcity and Model Sparsity finds that data repetition induces early routing ossification—is this ossification precisely the root of training-inference routing drift (the staleness that PR² from Issue 2 focuses on)? Can data mixing ratios serve as an intervention for routing stability?

Papers in this issue

  1. Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training

    Four bounded-streaming operators cut MoE memory peaks by up to 86.6%, enabling exact 1M-token training 8–32× longer than FSDP2 with up to 10.4× throughput.

    Editor's note

    Must-read. For the first time, unifies the four memory peaks of MoE long-context training (expert distribution, vocabulary projection, gradient checkpoint boundaries, optimizer states) into a "bounded streaming operator" framework, where each operator provides a fixed working set guarantee related to launch configuration. The core increment, PipelinedLLEP, decouples the receiver buffer size from the routing matrix via a source-side token budget, making routing skew only stretch scheduling rather than amplify memory, and uses strided chunking to eliminate local imbalance introduced by chunking. Compared to the negative result of Cacheable by Design in Issue 5 and the online placement of Director in Issue 4, this paper dissolves the cost of routing skew from the memory side while keeping routing decisions unchanged, training with 1M context on 120B–667B models and reporting complete routing statistics and dense baselines.

  2. Expert-Space Exploration in MoE Reinforcement Learning (ESRL)

    Expert-Space Exploration RL (ESRL) boosts Mixture-of-Experts LLM reasoning by explicitly perturbing router logits to explore alternative expert paths, improving Pass@1/Pass@8 by up to 3.2/4.5 points across math, science, and code benchmarks.

    Editor's note

    Must-read. For the first time, treats the expert routing space itself as an explicit exploration dimension in RL: during rollout, it adaptively perturbs logits based on router entropy, anchors on high-confidence experts, and samples from a candidate pool, while recording expert paths and replaying them during policy optimization to maintain training-inference consistency. Compared to PR² and ReLibra in Issue 2, which focus on "eliminating mismatch," this paper combines perturbation-replay into a controlled exploration mechanism, improving Pass@1 by 3.2 points over GRPO on Qwen3-30B-A3B, and covering three routing structures: top-K, top-1, and shared experts, with analysis of expert utilization and training dynamics.

  3. ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs

    ACE enables training-free, calibration-free expert skipping in MoE LLMs via dual-view scoring, yielding up to 2.25x prefill and 1.41x decoding speedups with superior quality.

    Editor's note

    Worth reading. For the first time, advances the contribution estimation for expert skipping to a purely parameter-driven dual-perspective conservative intersection criterion (global spectral proxy GSP + router-conditioned refinement RCR), requiring no calibration data or additional training, and at a 50% skip rate reduces PPL by 7.96% on Qwen3.6-35B compared to the strongest baseline. Compared to the global k-reduction of k1/k2 in Training-Free Halving from Issue 5, this paper skips low-contribution slots token-wise while preserving top-1, using the spectral norm upper bound of SwiGLU multiplicative coupling to provide provable conservatism, offering a new alternative path for training-inference routing consistency in inference-time sparsification.

  4. Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

    Mixture-of-Experts models overfit repeated data faster than dense Transformers, losing their advantage by 32x repetition, but dropout and output masking can restore it even at 64x.

    Editor's note

    Worth reading. The first systematic study of data repetition's effect on MoE overfitting: on a grid of 80M–1B activated parameters (total 8.5B), it finds that MoE is more sensitive to repeated data than dense models, and the degree of overfitting is determined by total parameters rather than activated parameters, with mechanisms of early routing ossification and expert specialization increasing with repetition rate. Compared to the sparsity-granularity decoupling in Slicing and Dicing from Issue 3, this paper adds the axis of data repetition, providing a reusable diagnostic protocol for routing stability and expert specialization, and reports comparisons with regularization mitigations like dropout.

  5. Towards a Statistical Understanding of Mixture-of-Experts

    Mixture-of-experts performance hinges on a three-way tradeoff between approximation error, expert learning cost, and router estimation complexity, with shared experts and Top-K routing providing statistical gains only under specific geometric conditions.

    Editor's note

    Worth reading. The first to formalize MoE as a localized aggregation class and provide a unified nonparametric statistical framework: it derives oracle risk bounds for dense softmax and Top-K routing, decomposing risk into approximation error, expert learning error, and router estimation cost, and proves that Top-K sparsity does not lose universal approximation capability with a fixed number of experts. Compared to the empirical scaling laws in Slicing and Dicing from Issue 3, this paper provides a theoretical baseline for the sparsity-granularity trade-off and the statistical role of shared experts, and the explicit separation of router estimation cost provides a theoretical benchmark for evaluating routing replay methods.

  6. ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts

    ProbMoE replaces deterministic top-k expert selection with exact probabilistic routing, improving expert utilization and enabling adaptive computation without sacrificing sparse execution.

    Editor's note

    Worth reading. The first to formalize MoE routing as probabilistic inference over cardinality-constrained subsets, using the SIMPLE estimator to sample k-expert subsets during training and backpropagate through exact marginal probabilities, unifying exact-k and dynamic-k routing. Compared to the soft anchoring of Router Prior Bias from Issue 5, this paper directly changes the gradient estimation method of the router, without relying on auxiliary losses or priors, providing lower gradient variance through exact marginals, and reports routing distribution diversity and dynamic cardinality usage analysis on OLMoE/Qwen.

  7. Physically Partitioned KVCache Format for CPU-GPU Load Balancing in MoE Inference (InplaceKVCache/WriteScope)

    WriteScope embeds CPU-GPU KV-Cache placement into the data format at write time, achieving 1.5-2.5x speedups for single-GPU long-context MoE inference.

    Editor's note

    Worth reading. The first to fix the physical residency of KV cache (pin state and layout continuity) as a format attribute at write time, proposing a four-region layout that enables CPU-GPU load balancing to dynamically adjust without data migration, and using a roofline online policy to dynamically adjust CPU share. Compared to the negative result of Cacheable by Design from Issue 5 (training locality infeasible), this paper achieves cacheability through format-level write-time routing, moving load balancing from the scheduling layer down to the data format layer, and reports complete routing statistics with dense baselines at the 1M token aggregation scale.

  8. Dynamic HBM Repartitioning for Multi-Turn MoE Serving (VAMP)

    VAMP dynamically repurposes expert-weight HBM for KV cache via CUDA VMM, cutting TTFT p90 23.6x and boosting throughput 20.7% on MoE agentic workloads.

    Editor's note

    Worth reading. The first to dynamically repartition the HBM boundary between expert weights and KV cache in MoE serving, using CUDA VMM page remapping to convert expert weight regions into KV cache capacity, and using a three-way cost model (expert offloading, cache eviction, request preemption) to select the action with the lowest estimated cost. Compared to the pre-routing load prediction of FreeBalance from Issue 3 and the asynchronous expert collection of MoE-Prefill from Issue 2, this paper unifies expert memory and KV cache into a shared pool, providing dynamic capacity expansion for the prefix-cache cliff of multi-turn agentic workloads, and reports routing statistics and dense baselines on Qwen3-Next-80B.