Summary (Overview)

• Problem: Distributed Mixture-of-Experts (MoE) inference suffers from persistent execution-time imbalance across GPUs due to the interaction of routing-induced workload skew and hardware performance variability. Under synchronized expert parallelism, the slowest GPU determines layer latency, creating stragglers that inflate tail latency and reduce throughput.

• Key Insight: Prior work optimizes token balance assuming homogeneous hardware, but even balanced token assignments leave hardware-induced stragglers unaddressed. The authors propose co-optimizing workload skew and hardware asymmetry: assign high-load experts to faster GPUs and low-load experts to slower GPUs to balance execution time rather than token count.

• Proposed Solution: ViBE (Variability-Informed Binning of Experts) is a hardware-aware expert placement framework combining per-GPU performance modeling, expert activation profiling, execution-time-aware placement, and drift-triggered recalibration. It minimizes predicted layer latency span rather than token count alone.

• Results: ViBE improves SLO attainment by up to 15% over EPLB, reduces P90 TTFT by up to 45%, lowers per-layer barrier idle time by 41%, and reduces synchronized MoE execution latency by 35% across two models (DeepSeek-V3, Qwen-3 235B) and two AMD GPU platforms.

• Key Finding: Hardware variability is stress-dependent—it manifests when GPUs operate near power limits (prefill, high QPS). The impact of variability increases at scale, making variability-aware placement essential for efficient, high-utilization LLM serving.

---

Introduction and Theoretical Foundation

Background

Mixture-of-Experts (MoE) architectures allow large language models to scale by decoupling total model capacity from per-token computation. Instead of activating all parameters for every token, MoE models route each token to a small subset of experts, allowing parameter count to grow without proportional increase in compute.

In distributed inference, MoE systems rely on expert parallelism (EP) to place expert weights across GPUs since the full expert set cannot fit in a single device's memory. With EP, each MoE layer requires all-to-all communication and synchronization, and layer latency is determined by the GPU that finishes last.

The Core Challenge

The paper identifies three key challenges:

Challenge 1: Routing skew creates shifting stragglers. Token routing produces uneven, layer-varying expert loads. In layer 11 of DeepSeek-V3 during prefill, the busiest GPU processes over 24% of tokens while the least-loaded GPU handles less than 10%. The bottleneck GPU shifts across layers during a single execution.

Challenge 2: Hardware variability creates execution-time imbalance even with balanced token load. Nominally identical GPUs exhibit effective throughput differences due to process variation, power limits, and thermal conditions. This variability is activated by workload intensity—when GPUs operate near their power envelope, DVFS enforces power budgets, and device-specific power characteristics determine achievable frequency. The paper reports up to 7% kernel execution time variation even under balanced token loads.

Challenge 3: Placement becomes stale under workload-performance drift. Token routing ratios alone don't capture the workload stress that triggers hardware variability. Fixed-interval recalibration (as in EPLB) incurs reshuffling overhead even when placement remains appropriate, while failing to respond promptly when drift is significant.

Theoretical Foundation

The key theoretical insight is that execution time depends on the interaction of two factors that evolve differently:

1. Input-dependent expert load: Token routing produces uneven, layer-varying expert activation patterns
2. Device-specific performance: GPU throughput depends on device-specific operating characteristics and workload intensity

Under expert parallelism, layer latency is governed by the tail of the execution time distribution:

$$T = \max_{g} f_g(n_g)$$

where $f_g(n)$ is the latency function for GPU $g$ under token load $n$. Prior work (EPLB) implicitly assumes $f_g(n) = n$, so it cannot compensate for hardware throughput differences.

The paper shows that MoE layers are the dominant target for optimization: MoE FFN kernels account for 49% of prefill and 20% of decode time in DeepSeek-V3. Furthermore, MoE layers operate at TDP limit for 82.8% of execution time (vs. 34.8% for attention), reducing GPU frequency by 38% on average (vs. 10% for attention), making MoE layers the dominant source of execution-time imbalance.

---

Methodology

ViBE Framework Overview

ViBE consists of four components (Figure 7 in the paper):

1. Device-specific performance models: Maps token load to latency for each GPU
2. Expert activation profiling: Estimates workload skew for each layer
3. Execution-time-aware placement: Aligns predicted per-layer completion times across GPUs
4. Drift-aware incremental updates: Modifies placement only when workload/performance drift exceeds a threshold

Design Principles

Design Principle 1: Optimize for aligned completion time. Expert placement should not optimize for equal token counts alone but for aligned execution time across GPUs.

Design Principle 2: Minimize the slowest-GPU penalty. Equalizing token counts does not eliminate stragglers; the placement objective should minimize execution-time outliers.

Device-Specific Performance Models

Each GPU is warmed to steady-state operating conditions and profiled independently using the fused MoE kernel across a range of token counts, producing a function $f_g(n)$ mapping token load $n$ to expected kernel latency. The relationship between token load and expected latency per GPU does not vary over time, so profiling can be done once per GPU and retained.

Token Load Profiling

Expert activation is profiled over a representative input set, producing an activation matrix $W \in R^{L \times E}$, where each entry $w_e^{(l)}$ captures the relative load of expert $e$ at layer $l$.

Variability-Informed Expert Placement

The placement problem is formalized as minimizing execution time. Given a placement $\{S_g\}$, the token load on GPU $g$ is $n_g = \sum_{e \in S_g} w_e$, and the layer latency is determined by the slowest device:

$$T = \max_{g} f_g(n_g)$$

The objective minimizes $\sum_l T^{(l)}$ across all $L$ MoE layers.

Since the optimal partition problem is NP-hard (unrelated-machines makespan minimization), ViBE applies a per-layer greedy heuristic. For each GPU, a speed estimate is derived:

$$s_g = 1/f_g(n_{\text{ref}})$$

where $n_{\text{ref}}$ is the mean per-expert token load. A token target is then defined:

$$\tau_g = N \cdot \frac{s_g}{\sum_h s_h}$$

where $N = \sum_e w_e$ is the layer's total token load. Experts are assigned in descending order of $w_e$ to the GPU farthest below its target $\tau_g$, enabling faster GPUs to absorb high-load experts.

Drift-Aware Recalibration

Drift detection: ViBE tracks the cosine similarity between current and reference per-layer expert load vectors. After each rearrangement, it records the mean load vector $\hat{w}$ averaged over a window of 100 samples. Every $H$ forward passes (default $H=10$), ViBE computes:

$$d = 1 - \frac{\mathbf{w} \cdot \hat{\mathbf{w}}}{\|\mathbf{w}\| \|\hat{\mathbf{w}}\|}$$

When any layer's $d$ exceeds threshold $\delta_{\text{cos}}$ (default 0.05), recalibration triggers.

Incremental placement update: Rather than re-solving from scratch, ViBE starts from the current placement and applies minimal cross-GPU expert swaps. The process terminates when all GPUs fall within tolerance $\epsilon$ (default 0.03):

$$\max_g f_g(n_g) \leq (1 + \epsilon) \cdot \frac{1}{|G|} \sum_g f_g(n_g)$$

The incremental solver typically converges in 5–30 swaps per layer (vs. 200+ for full rebalancing), reducing transfer volume by over an order of magnitude.

---

Empirical Validation / Results

Experimental Setup

• Hardware: 8× AMD Instinct™MI325X GPUs (256 GB HBM3e, 1000 W TDP), plus validation on AMD Instinct™MI300X
• Software: vLLM v0.14.2, PyTorch v2.9.0, ROCm v7.0, AITER kernel backend
• Models: DeepSeek-V3 (256 experts, FP8), Qwen-3 235B (128 experts, FP8)
• Datasets: ShareGPT (variable-length), Sonnet (fixed 1024-in/128-out)
• Policies compared: vLLM (contiguous), EPLB (token-balanced), ViBE (execution-time-balanced)

SLO Attainment Results

Kernel Time and Hardware Telemetry

• Latency gap reduction: EPLB reduces median latency gap by 63.9% vs. vLLM; ViBE adds another 19.6% reduction
• Average MoE latency: ViBE improves by 49.3% vs. vLLM and 27.9% vs. EPLB
• Clock frequency uniformity: ViBE narrows inter-GPU frequency spread and compresses intra-GPU distribution, indicating more consistent utilization

Tail Latency Results

• DeepSeek-V3 on Sonnet: P90 TTFT reduced by up to 45%, P99 TTFT by up to 30%
• Qwen-3: P90 TTFT reduced by up to 10%, P99 TTFT by up to 30%

Dynamic Workload Adaptation

• SG→SN transition: Adaptive ViBE extends 90% SLO attainment from 1.68 to 1.80 QPS/GPU (vs. EPLB: 1.51 → 1.63)
• SN→SG transition: Adaptive ViBE pushes 90% crossover from 6.7 to 7.4 QPS/GPU (vs. EPLB: 6.5 → 7.0)
• Recalibration events cause transient TTFT spikes but recover within seconds

Sensitivity to Variability Distribution

• Mild variability (MI300X): ViBE provides consistent TTFT improvement across all QPS
• Skewed variability (13% deviation): ViBE assigns fewer tokens to degraded GPU, widening performance gap over EPLB at high QPS

Rack-Scale Projection

ViBE's advantage depends on two competing factors at larger EP group sizes: larger groups accumulate more performance spread (increasing benefit), but each GPU holds fewer experts (reducing placement flexibility). The sweet spot is at 16–32 GPUs; beyond 64 GPUs, all algorithms converge as per-GPU expert count collapses.

---

Theoretical and Practical Implications

Theoretical Implications

1. Challenging the token-balance assumption: The paper demonstrates that "balancing token counts is sufficient to balance performance" is a flawed assumption in distributed MoE serving. Execution time is shaped by both routing skew and hardware variability, and these factors interact non-trivially.

2. Variability as a resource: Rather than attempting to eliminate hardware variability, ViBE treats it as a lever—assigning high-load experts to faster devices transforms variability into an opportunity to reduce stragglers.

3. Stress-dependent variability: The paper establishes that hardware variability is a fixed property, but its performance impact is activated by workload intensity. This explains why variability-aware placement matters most at high utilization and during prefill.

4. Scale-dependent benefits: The impact of hardware variability increases at scale, making variability-aware placement increasingly important for large-scale deployments.

Practical Implications

1. SLO attainment: ViBE consistently improves SLO compliance (up to 15% higher sustainable throughput at 90% goodput), directly translating to lower serving costs since operators pay for provisioned GPU capacity.

2. Tail latency reduction: P90/P99 TTFT reductions of 10–45% improve user-perceived interactivity without sacrificing throughput.

3. Recalibration efficiency: Drift-triggered recalibration with incremental updates reduces weight-transfer overhead by over an order of magnitude (5–30 swaps vs. 200+ expert reassignments per layer).

4. General applicability: ViBE works across different models, GPU platforms, and variability regimes, requiring only per-GPU profiling and expert activation statistics.

---

Conclusion

Main Takeaways

1. Token balancing is insufficient: Under expert parallelism, where each MoE layer completes at the pace of the slowest device, routing skew and hardware variability combine to create persistent stragglers that inflate tail latency and limit sustainable throughput.

2. Co-optimization works: By shifting the optimization target from token balance to execution-time balance, ViBE uses variability as a lever to reduce stragglers rather than treating it as a source of inefficiency.

3. Hardware variability is a first-order constraint: The results show that hardware variability is not a secondary artifact of deployment but a primary factor in large-scale LLM serving efficiency.

4. Drift-aware adaptation is essential: Workload characteristics and effective GPU throughput shift across serving conditions; recalibration must be stress-aware and workload-aware to remain effective without excessive overhead.

Future Directions

• Non-uniform expert allocation: The paper leaves varying the number of experts per GPU to future work (both EPLB and ViBE can support this).
• Reducing migration overhead: Incremental or speculative placement updates could further minimize latency disruptions during recalibration events.
• Co-design at extreme EP degrees: Variability-aware TP grouping or selective expert duplication for systems with 64+ GPUs per EP group.
• Rack-scale systems: Emerging systems with 70+ GPUs in a single scale-up domain present new opportunities for variability-aware placement.