vExpert: Virtualizing Expert Storage for Adaptive Load Balancing in Distributed MoE Inference

Summary (Overview)

• Core Problem: Distributed Mixture-of-Experts (MoE) inference suffers from severe dynamic load imbalance due to the conflict between static expert placement on GPUs and the inherent routing dynamism of MoE models.
• Key Insight: The paper proposes a paradigm shift from partial residency (DRAM offloading) to full redundancy through a novel storage abstraction called vExpert (virtual Expert), which decouples expert residency from GPU HBM constraints by leveraging underutilized host DRAM.
• Main Contributions: (1) The first systematic performance model jointly capturing H2D latency, PCIe contention, and load imbalance; (2) a lightweight allocator that dynamically maps physical experts to virtual slots; (3) a disaggregated Coordinator–Worker memory manager that reduces system-level memory overheads in multi-process environments.
• Results: On DeepSeek-V3.1 within an 8-node H100 cluster, vExpert improves load balance ratio by up to 55% and achieves 1.2× end-to-end speedup over state-of-the-art baselines.
• Key Innovation: The system achieves just-in-time expert availability via a layer-wise pre-allocating pipeline that masks H2D transfer latency behind computation of preceding layers.

Introduction and Theoretical Foundation

Background: The MoE Load Imbalance Problem

Mixture-of-Experts (MoE) architectures have become the de facto standard for efficient large language model inference. Expert Parallelism (EP) is universally adopted for distributed inference, sharding experts across multiple GPUs. However, static expert placement contrasts sharply with the inherent dynamism of MoE routing—activated experts fluctuate drastically depending on input, leading to severe load imbalance where GPUs hosting hot experts become bottlenecks.

Limitations of Existing Approaches

Existing systems mitigate imbalance by replicating redundant experts, but this exposes a fatal flaw: the inherent conflict between redundancy capacity and memory constraints. With GPU HBM heavily occupied by model parameters and KV caches, redundancy is restricted to merely 1–2 experts per GPU, stripping the system of elasticity needed for rapid load drifts.

Key Motivations for Host DRAM Utilization

The paper identifies three observations motivating a tiered expert storage hierarchy:

1. Abundant capacity: Underutilized TB-scale host DRAM provides ample space for full-model redundancy
2. Competitive bandwidth: Aggregated H2D unidirectional bandwidth (e.g., 512GB/s via PCIe Gen5) is competitive with Device-to-Device interconnects (e.g., 450GB/s via NVLink 4.0)
3. Topology symmetry: Host memory offers global visibility and uniform access by all intra-node GPUs

Two Overlooked Challenges

The paper identifies why existing DRAM offloading fails in distributed inference:

1. Coupled performance impacts: H2D transfers compete with inter-node All-to-All (A2A) communication for node-level PCIe resources, causing contention. Load balance exhibits dynamism both spatially (layers) and temporally (iterations), making constant H2D traffic sub-optimal.

2. System-level memory overhead: Achieving peak H2D bandwidth requires page-locked weights, forcing each GPU process to maintain page table mappings—consuming GBs of HBM for a 671B model. Additionally, OS-level shared memory quotas (typically 50% of physical RAM) limit accessible host capacity.

Methodology

vExpert Storage Hierarchy

The system establishes a two-tier expert storage hierarchy:

• L1 (Device HBM): Serves as a dynamic cache for weights required by the current iteration and layer, maintaining a minimal set of vExpert Slots (HBM buffers serving as H2D transfer destinations)
• L2 (Host DRAM): Maintains a full-coverage physical expert pool of the model

Pre-Allocating Pipeline

vExperts are managed via a layer-wise pre-allocating pipeline to mask H2D latency:

During execution of layer $i$, the system asynchronously initiates H2D transfers for vExperts required by layer $i+1$.

This allows GPUs to maintain only buffers sufficient for two layers (current and next), minimizing memory footprint.

Performance Model

The baseline per-layer latency is defined as:

$$T _ { i } (L _ { i }) = T _ { i, \text { dispatch }} (L _ { i }) + T _ { i, \text { moe }} (L _ { i }) + T _ { i, \text { combine }} (L _ { i })\tag{1}$$

where $L _ { i } = \frac { \mathrm { A v g T o k e n s P e r G P U } } { \mathrm { M a x T o k e n s P e r G P U } } \in [ 0 , 1 ]$ is the load balance ratio.

With virtual experts, the enhanced latency model becomes:

$$T _ {i} ^ {*} (L _ {i}, V) = T _ {i, \mathrm{a2a}} (L _ {i}) + T _ {i, \mathrm{moe}} (L _ {i}) - \underbrace {T _ {i , \mathrm{a2a}} \cdot \gamma_ {1} (L _ {i} , V)} _ {\text { communication gain }} - \underbrace {T _ {i , \mathrm{moe}} \cdot \gamma_ {2} (L _ {i} , V)} _ {\text { computation gain }} + \underbrace {T _ {\mathrm{H2D}} (V) \cdot \eta} _ {\text { H2D overhead }}\tag{2}$$

Key model components include:

Communication gain:

$$\gamma_ {1} (L _ {i}, V) \approx 1 - \frac {B / (L _ {i} + \Delta_ {L})}{B / L _ {i}} = \frac {\Delta (L _ {i} , V)}{L _ {i} + \Delta (L _ {i} , V)}\tag{3}$$

Load balance improvement (empirically fitted):

$$\Delta (L _ {i}, V) = \alpha \cdot L _ {i} + \beta \cdot V + k\tag{4}$$

with fitted parameters $\alpha = -0.75$, $\beta = 0.53$, $k = 0.1$.

H2D latency: $T _ { \mathrm { H 2 D } } ( V ) = \frac { V \cdot M _ { \mathrm { e x p e r t } } } { \mathrm { B W _ { H 2 D } } }$, achieving $\mathrm { B W _ { H 2 D } \approx 2 6 \mathrm { G B / s } }$ per GPU on H100 servers.

Contention factor: $\eta \approx 0.19$, reflecting that contention degrades H2D by 40% but only mildly impacts All2All due to multistage network fabric traversal.

Optimization Problem

$$V ^ {*} = \underset {V \in \mathbb {Z} _ {\geq 0}} {\operatorname{argmin}} T _ {i} ^ {*} (L _ {i}, V) \quad \text { s.t. } \quad T _ {\mathrm{H2D}} (V) \leq T _ {i} \implies V \leq V _ {\max}\tag{5}$$

Since the search space is small (typically $V _ { \mathrm { m a x } } < 3$), a greedy traversal strategy is employed.

Coordinator–Worker Disaggregated Memory Management

The architecture designates Rank 0 as coordinator (allocates and pins the expert pool in host memory, uses CUDA IPC to import worker GPU buffers) and Ranks 1–7 as workers (allocate local GPU ping-pong buffers, export handles via cudaIpcGetMemHandle, focus exclusively on kernel execution).

Empirical Validation / Results

Experimental Setup

• Hardware: 8-node cluster, each node with 8×H100-NVLink GPUs, 1.8 TB DDR5-4800 RAM, 4×400 Gbps InfiniBand
• Model: 671B DeepSeek-V3.1 (default), KIMI-K2 (cross-model validation)
• Workloads: Synthetic workload from ShareGPT, LongBench, LiveCodeBench
• Baselines: Vanilla SGLang (B0), EPLB (B1), EPLB+redundant expert (B2), Offload

Key Results

Load Balance Effectiveness:

• Vanilla baseline: balance ratio 0.43
• B1/B2: 0.44 and 0.49
• vExpert: 0.76 (55% improvement over best baseline)

End-to-End Throughput:

• 28.4% improvement at batch size 256
• 12.4% improvement at batch size 128
• Offload baseline degraded 10% due to constant H2D transfers

Ablation Study Results

Allocator Efficiency (Figure 7c):

• 41 virtual expert slots assigned to 35 imbalanced layers
• Layers with 1 slot: balance ratio improved from 0.438 → 0.763
• Layers with 2 slots: balance ratio improved from 0.347 → 0.768
• Allocator overhead: < 2%

Memory Manager Impact (Table 1):

Enabling the manager yields an 8× reduction in GPU memory overhead.

Scalability: At EP64, vExpert improves balance ratio to 0.76 and throughput by 20% over B2.

Model Robustness (Table 2): With ±20% parameter perturbations, throughput degradation remains below 3% in all cases.

Cross-Model Generalizability (Table 3, KIMI-K2):

Theoretical and Practical Implications

Theoretical Contributions

1. First systematic performance model for H2D transfers in distributed MoE inference, jointly capturing load imbalance, PCIe contention, and transfer latency
2. Paradigm shift from partial residency to full redundancy, decoupling expert redundancy capacity from HBM constraints
3. Empirical characterization of load balance improvement as a linear function of initial imbalance and vExpert count

Practical Implications

1. Elastic load balancing: The system adapts to dynamic workloads with near-zero overhead (10ms allocation every 10 iterations, < 1ms per-iteration overhead)
2. Memory efficiency: Coordinator–Worker architecture reduces GPU memory overhead by 8× and bypasses OS shared memory limits
3. Deployment flexibility: Parameters can be re-calibrated in under 30 minutes for new models; microbenchmarks suffice for new hardware platforms
4. Robustness: Discrete integer decisions make the system resilient to parameter estimation errors

Conclusion

vExpert pioneeringly resolves challenges for exploiting host resources in distributed MoE inference by:

1. Building a systematic performance model that quantifies the coupled impact of H2D latency, PCIe contention, and load imbalance
2. Designing a lightweight allocator that dynamically maps physical experts to virtual slots
3. Implementing a disaggregated memory manager that reduces system-level overheads

The system achieves up to 20% throughput improvement and highly stable load balance under dynamic workloads.

Future Directions

• Prefill phase integration: Complements GPU-based load-balancing methods (e.g., EPLB) since longer per-forward latency amortizes expert reordering overheads
• Emerging architectures: Supernode architectures (e.g., GB200 NVL72 with NVLink-C2C and coherent CPU-GPU paths) may reshape the host–device landscape, making host-backed expert virtualization even more promising
• Adaptation to advancing topology: Extending vExpert to new hardware hierarchies is identified as important future work