Summary of "DIRECTOR: Proactive Expert Placement for Fine-Grained MoE Serving"

This paper presents DIRECTOR, a system for optimizing expert placement in Mixture-of-Experts (MoE) model serving. The key innovation is using a proactive, prediction-driven approach to place experts on GPUs, rather than reactive or static methods. The system reduces end-to-end serving latency by predicting which experts will be needed and placing them strategically to minimize communication and computation costs.

---

Summary (Overview)

• Problem: Efficiently placing experts across GPUs is critical for Mixture-of-Experts (MoE) model performance. Existing static or reactive placement strategies fail under dynamic, diverse request patterns.
• Key Contribution: Proposes a proactive framework (DIRECTOR) that predicts future expert usage and places experts accordingly, using a lightweight predictor and a provably near-optimal placement algorithm.
• Proposed Solution: Comprises two main components:
  1. A lightweight expert-usage predictor for fine-grained MoE models that achieves high accuracy with negligible overhead.
  2. A (1+ε)-approximation placement algorithm that solves the NP-hard expert placement problem in polynomial time, with theoretical guarantees.
• Key Results: Extensive experiments show DIRECTOR reduces end-to-end latency up to 60.9% compared to existing state-of-the-art baselines (e.g., DeepSeek-V2 and Qwen3 models), while the prediction overhead remains below 3% of per-token generation latency.

---

Introduction and Theoretical Foundation

Background and Motivation

• Mixture-of-Experts (MoE) models scale model parameters efficiently by routing each input token to only a subset of experts (top-k routing).
• Expert Parallelism (EP) is the standard method for serving MoE models across multiple GPUs; it distributes experts across devices, typically requiring tokens to be communicated between GPUs during all-to-all exchange.
• The Core Challenge: Expert placement—deciding which expert is stored on which GPU—significantly impacts both communication and computation latency. A poor placement leads to excessive inter-GPU traffic and load imbalance.
• Limitations of Existing Work:
  • Static/Optimal Placement (offline): Computes a placement from historical traces and keeps it fixed. This fails when request patterns change dynamically.
  • Reactive Placement (online): Adjusts placement based on observed, recent traffic. This suffers from a "lag" effect)Skip, continually reacting to past/current demand rather than anticipating future needs.

Proposed Framework: DIRECTOR

1. Prediction Module: A lightweight predictor forecasts future expert usage per layer.

2. Optimization Module: A fast, scalable optimizer that computes a near-optimal expert placement based on the predicted usage, balancing communication and computation costs across GPUs.

---

Theoretical Foundation and Algorithm

Problem Formulation

• The goal is to minimize total inference latency, which is the sum of communication cost (dependent on which experts are on which GPU) and computation cost (dependent on load balance across GPUs).
• The loss of a token ( x ) is given by:
  • ( x = w_Q^l x + \sum_{i=1}^{N_l} \pi_i(x) \cdot \alpha_i(x) \cdot (w_E^l)^T x )
  • where π_i(x) is the gating probability, α_i(x) is the binary on/off decision, and w_E^l is the expert embedding matrix.
• Optimization Problem: The objective is to find the placement ( P ) that minimizes total latency:
  • Objective: min TotalLatency = sum of (Token Dispatch + Expert Computation + Token Gather) across all GPUs.
  • Constraint: Each expert must be assigned within GPU capacity limits.
  • NP-Hardness: This problem is proven to be NP-hard (reducible from the Generalized Assignment Problem).

Key Theoretical Results

• The paper provides a binary search framework over the latency budget (β).
• It introduces a potential function Φ(x) to guide the rounding of a fractional solution (from an LP relaxation) to an integer solution.
  • Φ(x) — The potential, defined as the sum over all fractional variables (clamped to boundaries with +ε), letting the optimization process track the reduction of the solution space.
• Main Theoretical Guarantee: The randomized rounding algorithm produces a placement that is within (1+ε) (e.g., 1.05) of the optimal latency with high probability, running in polynomial time.

Key Concepts and Contributions

1. Algorithmic Contribution (Near-Optimal Placement):

   • The problem of optimizing expert placement is NP-hard.
   • DIRECTOR's algorithm provides a mathematical guarantee: it achieves a solution that is within O(ε) of the optimal, with a time complexity polynomial in problem size, i.e., a (1+є)-approximation.

2. Practical Implementation for Scalability:

   • To avoid adding latency to the critical request path, DIRECTOR runs expert placement offline, in parallel with ongoing generation.

3. Enhanced Predictor Design:

   • Motivation: Naive predictors (e.g., via gate logits) fail on fine-grained MoE models (accuracy as low as 61% for models like Mixtral-8×22B and DeepSeekMoE-16B).
   • DIRECTOR's predictor incorporates hierarchical semantic tokens to better capture workload patternscars, improving prediction accuracy and making the proactive approach viable.

Key Concepts and Definitions

• Expert Placement Problem: Assignment of experts (specialized sub-networks) to GPUs in a distributed MoE system.
• Communication Costs: Costs of sending tokens from source GPUs to GPUs hosting the required experts.
• **Computation Costs:**Load balancing across GPUs, quantified by the maximum token-processing load.
• Latency Objective: $L = L_{comm} + L_{comp}$, where $L_{comm}$ depends on the total byte traffic and $L_{comp}$ relates to the maximum load on any GPU.

Theoretical Contributions

The algorithm provides a provable (1+ε)-approximation guarantee for the NP-hard placement problemcase, meaning it finds a solution that is at most (1+ε) times worse than the theoretical optimumable optimum. This is achieved via a randomized rounding of a linear programming relaxation.

---

Algorithm Design

Algorithm 1: DIRECTOR Framework

The workflow is executed for each epoch (or preemption-free window):

1. Input: Current request queue ( X ).
2. Prediction: Predict the per-expert request counts for the upcoming epoch.
3. Placement: Compute the optimal expert placement using the optimization algorithm (e.g., the randomized-rounding method).
4. Update (Optional): Trigger migration if the predicted load metrics (e.g., distribution of requests) change significantly enough to justify the overhead.

Solution to the Optimization Problem

1. Linear Program (LP) Relaxation: The NP-hard integer programming problem is relaxed to a linear program, which can be solved in polynomial time.
2. Rounding: The fractional solution from the LP is converted into an integer solution via:
   • Dependent Rounding: Rounds variables in accordance with the problem's structure (conditional dependence).
   • PQ-tree-based Rejection Sampling (RS) or dependent rounding to fix infeasible assignments.

All steps are polynomial in the number of variables and constraints.

---

Experimental Results

Experimental Setup

• Models: M3ixtral-8×7B, DeepSeekMoE-16B, DeepSeek-V2-Lite, Qwen3-30B-A3B.
• Datasets: 1K prompts from LiveCodeBench (code), MATH-500 (math), TextVQA (visual question answering).

Results

End-to-End Latency

• DIRECTOR achieves the best performance across all models.
• Average latency reduction: Up to 60.9% over the DeepSeek-V2 baseline and 60.66% over the strongest baseline (ReMo).
• Consistent improvement: Reduces latency by 21–57% compared to the next-best baseline (e.g., ReMo, FasterMoE) across diverse workloads and prompt types.
• Hyperparameter Robustness: DIRECTOR maintains its performance advantage across different batch sizes and message-token limits, demonstrating robustness to workload changes.

Table: Summary of Key Results (Illustrative)

---

Algorithm Design: The Prediction-and-Optimization Pipeline

1. Online Predictor: Hierarchical RNN with Gating

• Why: Existing LLM predictors are too heavy. Naive predictors fail on fine-grained models (low accuracy).
• Approach: A lightweight Hierarchical Recurrent Neural Network (HRNN) with gating mechanisms.
  • The model handles multi-granular semantics by learning multiple levels of temporal abstraction (e.g., word, sentence, codeline) and using a gate to fuse them.
  • Uses outputs from the LLM's earlier layers (a hidden state, ( h^l )) as enriched input features.
• Performance: Achieves 88.74% prediction accuracy and <3% relative overhead on fine-grained MoE (e.g., DeepSeek-V2-Lite), vastly outperforming naive predictors (60%).

2. Optimization Algorithm: Randomized Rounding with Capacity Constraints

• Core Challenge: The placement problem is an NP-hard quadratic assignment problem. Complexity is O(n^2 * m) where n is experts and m is GPUs.
• Relaxation & Rounding:
  1. Relaxation: Formulate the problem as an Integer Linear Program (ILP) and relax it to a Linear Program (LP) by allowing fractional assignments.
  2. Randomized Rounding with Alteration for Capacity Constraints: Solve the LP, then use randomized rounding to assign experts.
  3. Theoretical Guarantee: Bound the probability of exceeding a GPU’s capacity by making the expected load strictly below the capacity. Adjusting parameters in the objective/capacity constraints yields a ( (1+\epsilon) )-approximation for both communication and computation costs. This guarantees near-optimal results (Theorems and Corollaries in paper show detailed bounds).

Why it works

Since the predictor is fast (and accurate), the system can repeatedly re-optimize placement. Since the optimizer runs in polynomial time and has an optimality guarantee, the overall system can react quickly (within milliseconds) to shifting traffic patterns. The theoretical backbone ensures that while the algorithm is fast, it does not compromise on solution quality, consistently maintaining strong performance.

---

Experimental Results

Experimental Setup

• Models: Mixtral-8×7B, DeepSeekMoE-16B, DeepSeek-V2-Lite, Qwen3-30B-A3B.
• Datasets: LiveCodeBench (code), MATH-500 (math), WildChat (conversation).
• Baselines:
  • State-of-the-art reactive algorithm (Plug-and-Play).
  • Static placement (optimal from historical offline).
  • Default placement (DeepSpeed).

Key Metrics & Results

(A) End-to-End Latency

• Effect: Significantly lowers latency.
• Magnitude: 6.9%-60.9% lower latency compared to the strongest reactive baseline across all models.
• Example: On Mixtral-8×7B, latency dropped from 895ms (baseline) to 349ms.

(B) Communication Cost

• Effect: Reduces per-token communication volume by 31-47% compared to baselines, directly addressing the aforementioned all-to-all bottleneck.

Training and Optimization Details

• Training set: Utilized multi-task datasets (e.g., MATH, LiveCodeBench, Wikipedia) to simulate diverse workloads. Evaluated on up to 8 A100 GPUs.

Ablation Studies and Key Findings

• Contribution Breakdown (on DeepSeek-V2-Lite):

  Component	Latency (ms)
  No Placement Optimization	325
  + Predictor Only	277
  + Predictor + Optimizer	179 (Speedup)
  The optimizer yields a larger performance gain than the predictor, highlighting communication vs. computation trade-offs.

• Key Findings:

  • Effectiveness: DIRECTOR consistently outperforms all baselines across all models and datasets.
  • Theoretical Rigor: The (1+ε)-approximation guarantee provides a worst-case bound, ensuring reliable performance.
  • Practical Impact: The hybrid prediction-and-placement approach offers a practical solution for production MoE systems where workloads are dynamic.
  • Ablation: The optimization algorithm contributes the majority of the performance gain (e.g., 43% of improvement on DeepSeek-V2-Lite) compared to the predictor. The optimal placement alone provides significant gains over naïve strategies (e.g., 2x better than hash-based placement).

Conclusion

The paper presents a complete, theoretically-grounded, and practical solution (DIRECTOR) to the problem of dynamic expert placement in MoE serving. It combines a novel, efficient predictor with a rigorous optimization algorithm to deliver robust performance improvements. The results, benchmarked on state-of-the-art MoE models including DeepSeek-V2 and Qwen3, demonstrate the system's readiness for production deployment in dynamic, multi-tenant serving environments.