MoE & Sparse ExpertsIssue 15Nov 14 – 21, 2026

Expert-Parallel Service Sinking: One-Sided Communication and Bubble Harvesting

Highlights

The main thread of this issue is "sinking expert-parallel services into communication primitives and idle GPU resources." One-Sided Expert-Parallel All-to-All is the first to implement EP all-to-all dispatch/combine as fully one-sided communication: the source rank can push tokens into the target rank's rank-major workspace without prior metadata exchange, GEMM reads directly by slot index, and provides two synchronization protocols, Fence and Sentinel, for high throughput and low latency respectively, achieving 1.2–3.0× speedup over DeepEP on Blackwell for small-batch decoding. Meanwhile, Harvesting Expert-Parallel Bubbles for Fine-Tuning is the first to treat the residual idle GPU intervals (bubbles) after load balancing as harvestable resources: it predicts the bubble duration per GPU layer-by-layer from router token allocation, schedules fine-tuning work blocks into these windows, and uses a compiler to derive a fine-tuning runtime that shares weights with serving from the serving forward pass. Together, these push "expert-parallel serving" from load balancing into two more fundamental dimensions: communication primitives and idle resource utilization.

The second line is recovery under extreme pruning. Reasoning Again with a Quarter of the Experts is the first to systematically isolate pruning-induced verbatim loop failures on 2-bit quantized inference MoE with 25% expert retention (below the previous 37.5% threshold), and establishes a "retry first, then train" recovery baseline: decoding-time stop-and-redraw as a strong baseline that weight methods must beat, then proposes a B method of low-rank correction plus recovering two experts per layer, achieving a significant +9.2 pp improvement on pre-registered untouched problems. Compared to SafeStart's expert reduction safety cost audit in Issue 13 and the pruning study in Issue 12 (You Only Need 2/3), this paper attributes loop failures to pruning rather than quantization (controlled comparison), and is the first to use retry strategies as a control baseline for weight recovery methods, while also reporting negative results (training-independent weight edits do not transfer).

On the specialization evidence side, What MoE Routing Controls uses causal intervention (applying fixed logit bias to target expert paths) to systematically compare the controllability of format goals versus concept goals in MoE routing: format paths can change generated text, concept paths only change next-token probabilities without affecting text, and localizes differences to intermediate layers and concept strength—providing a new path-level bias intervention method for causal evidence of expert specialization.

Community and Updates

vLLM's near-zero-downtime Elastic EP uses background asynchronous EPLB and NIXL EP to reduce reconfiguration downtime for elastic expert parallelism from about 100s to 0.82s: NIXL EP adds/removes ranks in place, reuses communication buffers and captured CUDA graphs, so retained workers do not need to recapture after reconfiguration. This is an engineering complement to FastEEP's elastic scaling in Issue 10, advancing "online rebalancing" from static deployment to dynamic elastic deployment. vLLM PR #67591 makes EPLB work with modular TRT-LLM FP8 block-scale experts: on GLM-5.3 DEP8, it improves balancedness from 0.58 to ~0.95 and throughput by +11.6%, providing production-stack validation of online redundant expert rebalancing on FP8.

Open Questions

  1. BubbleTea's bubble prediction relies on router token allocation and tile counts—how robust is bubble duration prediction when routing fluctuates sharply between tokens (agentic or long-context)? Does bubble harvesting still have independent gains when combined with in-switch replication from ISER-MoE in Issue 14?

  2. NVLinkOneSided's one-sided communication relies on direct read/write within the NVLink domain; cross-domain RDMA paths are not yet implemented—how can one-sided primitives extend to multi-domain hierarchical all-to-all? Can its Sentinel data self-synchronization protocol serve as a low-latency synchronization baseline for FP4 RL routing consistency evaluation in Issue 13?

  3. Reasoning Again's retry baseline holds at 25% retention on a single model (Ornith-1.5-122B)—what are the boundaries of retry strategies and weight recovery methods under more extreme sparsity, other reasoning MoEs, or long contexts? Is the observation that "retry gains outweigh weight method gains" universal?

  4. EchoMoE's cross-layer residual predictability relies on reference layer caching—when routing causes drastic changes in expert input distributions, how faithful is the compressed residual? Is there interaction between communication compression and routing replay/train-inference consistency?

Papers in this issue

  1. BUBBLETEA predicts expert-parallel bubbles during MoE inference and schedules LoRA fine-tuning inside them, achieving 1.9-6.2x faster training throughput than state-of-the-art colocation systems.

    Editor's note

    Must-read. First to treat the residual idle GPU intervals (bubbles) after expert-parallel load balancing as harvestable resources: predicts bubble duration per GPU layer-by-layer from router token allocation, schedules fine-tuning work blocks into these windows, and uses a compiler to derive a fine-tuning runtime sharing weights with serving from the serving forward pass. Compared to in-switch replication in ISER-MoE (Issue 14) and AFD decomposition in FastAFD (Issue 13), this paper explicitly models and utilizes "idle after load balancing," with router-aware planning recovering 42–45% of bubble capacity (vs. ~1.3% for GPU-agnostic planning), maintaining p90 latency within 10% of serving-only in 79% of 63 configurations, and reporting full routing statistics and dense baselines.

  2. NVLinkOneSided eliminates metadata exchange in MoE all-to-all communication, making dispatch and combine 1.2-3.0x faster than DeepEP on Blackwell GPUs, with the largest gains at small decoding batches.

    Editor's note

    Must-read. First to implement EP all-to-all dispatch/combine as fully one-sided communication: source rank pushes tokens into target rank's rank-major workspace without prior metadata exchange, GEMM reads directly by slot index, and provides Fence and Sentinel synchronization protocols for high throughput and low latency respectively. Compared to in-switch replication in ISER-MoE (Issue 14) and AFD decomposition in FastAFD (Issue 13), this paper focuses on the communication primitive itself, replacing metadata exchange with data self-synchronization, achieving 1.2–3.0× speedup over DeepEP on Blackwell for small-batch decoding, and reporting full routing statistics and dense baselines.

  3. EchoMoE cuts MoE expert-parallel RDMA traffic by 58% via cross-layer activation compression, improving inference throughput 22.6% with only a 3.8 percentage point performance drop.

    Editor's note

    Worth reading. First to use cross-layer residual stream predictability for EP communication compression: uses early-layer activations as reference, transmits only compressed residuals instead of full-width activations, and absorbs compression/decompression matrices into expert projections via linear folding to avoid explicit reconstruction. Compared to in-switch replication in ISER-MoE (Issue 14), this is a new increment in the communication compression path, retrofittable to existing models (8,500× fewer training tokens), improving inference throughput by up to 2.38× on bandwidth-limited Ethernet, and reporting full routing statistics and dense baselines.

  4. Pruning, not quantization, causes reasoning loops in a 25%-retention 122B MoE model, and low-rank corrections plus restored experts recover 9.2 percentage points over a weight-free baseline.

    Editor's note

    Must-read. First to systematically isolate pruning-induced verbatim loop failures on 2-bit quantized inference MoE with 25% expert retention (below the previous 37.5% threshold), and establishes a "retry first, then train" recovery baseline: decoding-time stop-and-redraw as a strong baseline that weight methods must beat, then proposes a B method of low-rank correction plus recovering two experts per layer, achieving a significant +9.2 pp improvement on pre-registered untouched problems. Compared to SafeStart's expert reduction safety cost audit (Issue 13) and the pruning study in Issue 12 (You Only Need 2/3), this paper attributes loop failures to pruning rather than quantization (controlled comparison), and is the first to use retry strategies as a control baseline for weight recovery methods, while also reporting negative results (training-independent weight edits do not transfer).

  5. Adding a constant bias to router logits to force tokens onto cross-layer expert paths causally steers MoE output format, but concept paths shift probabilities without changing generated text.

    Editor's note

    Worth reading. First to use causal intervention (applying fixed logit bias to target expert paths) to systematically compare the controllability of format goals versus concept goals in MoE routing: format paths can change generated text, concept paths only change next-token probabilities without affecting text, and localizes differences to intermediate layers and concept strength. Compared to SAE feature micro-domain proxies in RouterInterp (Issue 9) and behavior-associated expert suppression in Routes of Reasoning (Issue 13), this paper uses path-level bias intervention to distinguish the causal efficacy differences between format and concept goals, providing a new dimension for causal evidence of expert specialization.

  6. Framing MoE load balancing as PID control reveals the pure-integral controller PID(I) achieves the lowest expert imbalance across distribution shifts without modifying weights or objectives.

    Editor's note

    Worth reading. First to unify MoE load balancing into a PID feedback control framework, proving that auxiliary loss is the proportional term, ALF-LB is the relay integral term, and arguing that the pure integral term is the only one that both eliminates persistent imbalance and averages out step-wise inter-row load fluctuations; validates PID(I) over Aux-loss and ALF-LB under distribution shift (curriculum boundaries/continued pretraining). Compared to the PID control perspective in ID Balancing (Issue 8), the increment here is explicitly providing spectral analysis of the three terms (error spectrum, increment spectrum, flip rate) and recovery behavior under distribution shift, rather than just stability analysis.