MoE & Sparse ExpertsIssue 14Nov 7 – 14, 2026

In-Switch Replication and Eviction Rules: A System-Side Reassessment of Expert Parallel Serving

Highlights

The main thread of this issue runs along two parallel lines: "system-side optimization of expert parallel serving and load balancing" and "new evidence on the sources of routing drift." On the serving side, ISER-MoE is the first to use NVLink switch in-network multicast (NVLS) for per-call replication of hot MoE experts: by "fixing alignment groups rather than fixed replicas," it resolves the contradiction between fixed multicast members and per-layer changing demands, uses SAMI multi-port broadcast to distribute injections to group member ports, and hides replication with a shared expert computation window, achieving a 17.4% end-to-end throughput improvement on GB300 (DEP8). Compared to the point-to-point replication of MegaFlux in Issue 8 and UltraEP in Issue 4, this is a paradigm shift in the replication path itself. Meanwhile, Fewer Expert Transfers Without Changing Routing demonstrates under real CPU-to-GPU copy measurements that Belady-style farthest-future eviction without changing routing reduces copies by 2.1x compared to LRU with zero quality loss, and disproves the common assumption that "fewer routing switches mean fewer copies"—offering a complementary "eviction rules vs. scheduling" perspective to the runahead scheduling in Issue 13.

The second line is pretraining-side evidence on the sources of routing drift. MoE Pretraining: Parallelism Layout Changes the Model, Not Quality is the first to show through controlled experiments that the parallelism layout (DP/EP combination) itself can cause routing drift close to seed-level differences: the layout does not change quality, but at the end of training, prediction differences reach about 80% of a new seed, with over 90% of routing decisions differing, and the root cause is traced to near-ties in the router (AUC 0.97–1.00). This supplements the pretraining-side drift source for the discussion of FP4 RL and inference-time routing consistency in Issue 13, and also suggests that layout should be a key recorded variable in routing-related experiments.

On the specialization evidence side, Understanding MoE Beyond Expert Boundaries is the first to localize domain specialization to the neuron level rather than the expert level: the RAMP metric (routing-weighted activation magnitude × output projection norm) shows that specialized neurons are distributed across experts and correlate with routing preferences, and that neuron-level importance continuously reorganizes during pretraining while expert-level importance remains stable—directly responding to the debate over "whether routing preferences reflect domain specialization," and raising new questions about the intervention granularity of pruning/deactivation methods.

Community and Updates

Logical Skew Is Not Physical Contention directly disproves the implicit assumption driving expert-aware schedulers—"routing skew ⇒ multi-source physical incast"—using production-grade EP=4 workloads (GLM-4.7-Flash, Qwen3.6-35B-A3B, Nemotron-3-Nano): the p95 of both expert-level fan-in and target rank-level fan-in is 1, and the max is 2; logical fragmentation does not equal physical fan-in. This is a negative-control measurement that should be performed before designing any expert placement or redundant expert rebalancing, and it directly provides the workload-side background for ISER-MoE's "fixed groups rather than fixed replicas" in this issue.

Expert-Aware Causal Tracing of Factual Recall in Sparse MoE LLMs uses causal tracing to find that block-level fact recovery does not imply a single expert is responsible: Qwen3-30B-A3B-Base localizes to a positively specific resident expert (L44E069), while Mixtral's individual experts are not specific and require a coalition of routed experts to recover layer-level signals—providing a two-model comparison at the 30B scale for the standard of causal evidence of expert specialization. Your Experts Have Partners. There's HOPE for Keeping Them Together uses the pairwise structure of expert co-activation as a pruning criterion, improving by up to 6.1% over first-order baselines (REAP, etc.) on agentic coding tasks at a 50% pruning rate, directly challenging the first-order assumption that "expert contributions are mutually independent."

On the engineering side, Parallelism for Long-Context Serving: GLM-5.3-Flash on MI300X combines DP8+EP8 with local KV placement on AMD MI300X for long-context serving with 288 routed experts, reporting KV cache capacity ceilings and throughput comparisons across concurrency levels—TP8 dominates in single-request latency, while DP8+EP8 leads by up to 2.04x at concurrency above 128 in cache-constrained long-session scenarios (96k/32k).

Open Questions

  1. MoE Parallelism Layout proves that the pretraining layout causes routing drift close to seed-level via near-ties—is this pretraining-side drift and the training-inference inconsistency caused by quantization asymmetry in FP4 RL from Issue 13 the same source or orthogonal mechanisms? Are near-ties also the common starting point for the failure of inference-time routing replay methods?

  2. The Belady eviction in Fewer Expert Transfers does not change routing, and its 16-token lookahead assumes expert access is temporally predictable—when the access distribution is dynamically reordered by RL rollouts or long contexts, how robust is the farthest-future prediction? Does stacking the eviction rule with the runahead scheduling from Issue 13 still yield independent gains?

  3. The fixed alignment groups in ISER-MoE compress load to 1.002–1.020 of the per-step optimum under static routing statistics—when routing fluctuates sharply between tokens (agentic or long-context), can group-level replication and HALO-Q's step-level decisions keep up? What determines the benefit boundary between in-switch replication and the point-to-point pipelining of MegaFlux in Issue 8?

  4. RAMP proves that neuron-level specialization is distributed across experts and expert-level importance is stable—how does this unify with the routing solidification in Issue 6's Data Scarcity and the superposition specialization in Issue 9's RouterInterp? If the specialization unit is neurons rather than experts, do expert-level pruning/deactivation methods (such as HOPE's pairwise pruning in this issue) often choose the wrong intervention granularity?

Papers in this issue

  1. ISER-MoE: Dynamic In-Switch Replication of Hot Experts for Load Balancing in Expert-Parallel

    ISER-MoE uses in-switch multicast to replicate hot experts per invocation, removing 98-99% of MoE rank imbalance and boosting prefill throughput by 17.4%, exceeding perfectly balanced routing.

    Editor's note

    Must-read. The first to use NVLink switch in-network multicast (NVLS) for per-call weight replication of hot MoE experts, resolving the contradiction between fixed multicast members and per-layer changing demands via "fixed alignment groups rather than fixed replicas," and introducing SAMI multi-port broadcast to distribute injections to group member ports, hiding replication with a shared expert computation window. Compared to the point-to-point replication of MegaFlux in Issue 8 and UltraEP in Issue 4, this paper is the first to do per-call replication inside the switch, reporting end-to-end gains, full routing statistics, and dense baselines on GB300/Rubin hardware.

  2. Fewer Expert Transfers Without Changing Routing: Expert Eviction for MoE Offloading

    Evicting the expert needed farthest in the future copies 2.1x fewer experts than LRU with zero quality loss, making expert eviction a better optimization target than learned routing.

    Editor's note

    Must-read. Directly compares eviction policies with routing-changing policies under real CPU-to-GPU copy measurements, proving that Belady-style farthest-future eviction without changing routing reduces copies by 2.1x compared to LRU with zero quality loss, with 16-token lookahead recovering 96% of the gains, and disproving the common assumption that "fewer routing switches mean fewer copies." Compared to the runahead scheduling in Issue 13's Accelerating Offloading-Based MoE Inference, this paper focuses on the eviction rule itself without changing routing decisions, reporting full routing statistics and dense baselines.

  3. MoE Pretraining:Parallelism Layout Changes the Model, Not Quality

    Parallelism layout leaves MoE model quality unchanged but alters the trained model itself, with routing near-ties amplifying tiny numerical differences into seed-level divergence.

    Editor's note

    Must-read. The first to show through controlled experiments that the parallelism layout (DP/EP combination) itself can cause routing drift close to seed-level differences: the layout does not change quality, but at the end of training, prediction differences reach about 80% of a new seed, with over 90% of routing decisions differing, and the root cause is traced to router near-ties (AUC 0.97–1.00), separating numerical effects from definitional effects. Compared to Issue 13's focus on inference-time/RL routing consistency, this paper is the first to establish a causal link between layout and routing drift in the pretraining phase, suggesting that layout should be a key recorded variable for routing replay and interpretability work.

  4. A Per-Step Load-Balancing Engine for Expert-Parallel MoE Serving

    Relief's all-integer per-step load-balancing for MoE serving recovers up to 37.3 points of utilization at 0.02% silicon overhead, near-optimal within 1%.

    Editor's note

    Must-read. The first to advance per-layer load balancing for expert parallel serving from software solvers to hardware engines: the Relief bidirectional pairwise relaxation algorithm approximates the per-step optimum within a few steps using integer arithmetic, and implements three deployments (CUDA kernel, FPGA, on-chip fixed-function block), reporting effective utilization of 93%–99.5% relative to the per-step optimum at 8–64 GPU scales. Compared to Issue 13's FastAFD AFD gain analysis, this paper focuses on token splitting among replicas in expert parallelism, solving decision latency with hardware implementation, reporting full routing statistics and dense baselines.

  5. Understanding Mixture-of-Experts Language Models Beyond Expert Boundaries

    Neuron-level domain specialization in MoE models spans mixed experts, and RAMP-based pruning preserves 91-98% performance at 75% retention while cutting memory 36-43%.

    Editor's note

    Worth reading. The first to localize MoE domain specialization to the neuron level rather than the expert level: the RAMP metric (routing-weighted activation magnitude × output projection norm) shows that specialized neurons are distributed across experts and correlate with routing preferences, and that neuron-level importance continuously reorganizes during pretraining while expert-level importance remains stable. Compared to Issue 9's RouterInterp SAE feature micro-domain proxies, this paper directly uses neurons as the unit, reporting full routing statistics and dense baselines across six models, and its training dynamics analysis is valuable for routing stability research.

  6. vExpert: Virtualizing Expert Storage for Adaptive Load Balancing in Distributed MoE Inference

    vExpert decouples expert residency from GPU HBM by virtualizing full expert redundancy in host DRAM, improving load balance by 55% and throughput by 1.2x over state-of-the-art baselines.

    Editor's note

    Worth reading. The first to advance DRAM offloading from partial residency to fully redundant virtualized expert storage, and provides the first system performance model jointly modeling H2D latency, PCIe contention, and load imbalance, relaxing per-layer H2D constraints with group-level budget. Compared to Issue 12's Rimuru elastic expert cache, the increment here is in storage abstraction and performance modeling rather than unified memory management, reporting load balancing ratios and dense baselines on DeepSeek-V3.1.