Summary (Overview)

• Gimbal is a coordinated cross-level scheduling system for Mixture-of-Experts (MoE) large language model (LLM) serving that jointly optimizes frontend data-parallel (DP) engine scheduling and backend expert placement.
• It introduces a fine-grained DP-engine scheduler that uses online backend pressure signals (KV-cache usage, remaining prefill work, queue backlog, MoE expert pressure) instead of coarse request counters.
• It extends expert load balancing with source-aware routing statistics (source-DP-to-expert activation matrices) and a heuristic calibrated against a mixed-integer nonlinear program (MINLP) to jointly optimize expert load, communication cost, and migration stability.
• Evaluation on a 4×H100 testbed with Qwen3-30B-A3B and BurstGPT workloads shows Gimbal reduces average TTFT by 42.9% and average TPOT by 33.3% compared to vLLM, while improving high-load throughput by 3.0%.
• An ablation study confirms that coordinated cross-level scheduling outperforms simply enabling DP-side and EP-side optimizations independently.

---

Introduction and Theoretical Foundation

Background and Motivation

Large Language Models (LLMs) have advanced rapidly, but scaling dense Transformer models increases computational demands, inference latency, and deployment costs. Sparse architectures, particularly Mixture-of-Experts (MoE) models, address this by activating only a small subset of experts per token, scaling model capacity while keeping per-token inference costs manageable. DeepSeek-series models demonstrate that sparse MoE models achieve competitive capability with substantially reduced activated computation.

Key Challenges in MoE Serving

MoE inference systems combine three parallelism paradigms:

• Data Parallelism (DP): replicates inference engines to improve throughput
• Expert Parallelism (EP): distributes experts across GPUs
• Tensor Parallelism (TP): partitions dense model computation across GPUs

This layered design introduces coordination challenges:

1. Coarse request scheduling causes inter-engine imbalance: Request-count-based dispatching assumes equal execution cost, but real workloads (e.g., GPT-4 traces) exhibit highly skewed input lengths. A 2K-token request consumes 187.5 MiB of KV cache vs. 18.75 MiB for a 200-token request.

2. Sparse routing causes intra-model imbalance: Token traffic is not uniformly distributed across experts. Profiling shows 83.4% of Layer-23 traffic from DP0 and 66.5% of Layer-36 traffic from DP1 route to remote DP groups under current placement, creating cross-DP communication overhead.

3. Inter-engine and intra-model imbalance are coupled: DP-engine dispatch determines where tokens enter the all-to-all communication path, while expert placement determines whether selected experts are local or remote. Expert migration costs average 1.08s for first rearrangement and 0.72s for subsequent ones.

Limitations of Current Systems

Existing systems make scheduling decisions in isolation:

• Frontend schedulers (vLLM, SGlang, FastServe) use request-count-based routing or phase-aware policies but don't expose fine-grained engine pressure
• Backend expert balancers (EPLB, MoETuner) use aggregate expert activation counts but don't distinguish which DP source generates traffic
• Sem-MoE uses offline profiling that becomes stale as workloads change

---

Methodology

System Architecture

Gimbal coordinates two main components through a feedback loop:

1. Fine-grained DP-engine Scheduler (Section 4)
2. Source-aware Expert Load Balancer (Section 5)

Backend engine traces and expert pressure signals flow upward to the scheduler, while expert migration actions flow downward to the EP region.

Fine-grained DP-engine Scheduling

Runtime Trace Collection: Each engine periodically reports:

• Remaining prefill tokens from running requests
• Waiting prefill tokens in local queue
• KV-cache usage
• Backend MoE expert pressure

Pressure-Aware Engine Selection (Algorithm 1):

• If KV-cache usage is high (>90%) and the gap across engines exceeds 10%, select the engine with lowest KV usage
• Otherwise, compute a pressure score for each engine: $$score_i = pre_i^{rem} + wait_i + comp_i + P_{kv}(kv_i) + P_{moe}(moe_i)$$
• A lightweight compensation term ($comp_i$) accounts for recently dispatched requests before trace updates

Intra-Engine Ordering (Algorithm 2): SJF-style queue ordering with aging, using prefill token count as job-size estimate. Requests waiting longer than threshold $\theta_{age} = 5$ seconds are promoted to high priority.

Source-aware Expert Load Balancing

Online Traffic Profiling: Two statistics are collected along the MoE dispatch path:

• $B_{l,e}$: aggregate expert activation counts (computation load)
• $A_{l,s,e}$: layer-wise source-DP-to-expert activation matrix (source-aware routing profile)

Placement Objective: The MINLP formulation optimizes three goals:

Load balance:

$$C_{\mathrm{load}} = \sum_{l} \sum_{g} \left(L_{l,g} - \bar{L}_{l}\right)^{2}$$

Source-aware communication:

$$C_{\mathrm{comm}} = \sum_{l} \sum_{s} \sum_{e} \sum_{g} A_{l,s,e} D_{s,g} x_{l,e,g}$$

Migration cost:

$$C_{\mathrm{mig}} = \sum_{l} \sum_{e} \sum_{g} M_{l,e,g} \left| x_{l,e,g} - x_{l,e,g}^{0}\right|$$

Overall objective:

$$\min_{x} C_{\mathrm{load}} + C_{\mathrm{comm}} + C_{\mathrm{mig}}$$

Online Heuristic: Since solving the MINLP takes ~15 seconds for a 48-layer model (prohibitive for online serving), Gimbal uses a greedy heuristic with local placement score:

$$S_{l,e,g} = \alpha C_{l,e,g}^{\mathrm{comm}} + \beta C_{l,e,g}^{\mathrm{load}} + \gamma C_{l,e,g}^{\mathrm{mig}}$$

Calibrated parameters: $(\alpha, \beta, \gamma) = (1.0, 0.0025, 1.0)$, preserving >80% of MINLP placement decisions with communication within 0.6% of the offline reference.

---

Empirical Validation / Results

Experimental Setup

• Testbed: 4×NVIDIA HGX H100 80GB SXM5 GPUs, NVLink-connected, 2×32-core Intel Xeon Platinum 8462Y+
• Deployment: DP=2, TP=2, EP=4
• Model: Qwen3-30B-A3B (MoE)
• Workloads: BurstGPT traces reshaped into five distributions (Random, Central, Descending, Two-end, Average)
• Baselines: vLLM (production-grade), MoE-Tuner (offline affinity-based), Sem-MoE (oracle variant with perfect routing knowledge)

Key Results

Performance improvements grow with load: TTFT reduction over vLLM increases from 33.1% at RPS=2 to 48.3% at RPS=4.

Ablation Study

The coordinated design (feeding backend MoE pressure to DP scheduler) provides an additional 16.5% TTFT and 6.5% TPOT reduction over independent optimizations.

MINLP Calibration Impact

Calibrating the greedy expert-placement policy against the offline MINLP reference reduces:

• Average TTFT by 10.8%
• Average TPOT by 9.2%

compared to the uncalibrated greedy policy.

Runtime Behavior at RPS=4

• Cross-engine prompt-throughput gap: reduced from 1485.66 tokens/s (vLLM) to 768.43 tokens/s (Gimbal-DP)
• Average running requests per engine: reduced from 87.56 (vLLM) to 71.53 (Gimbal-EP)

System Overhead

The optimized source-aware matrix collection path (with Triton kernel fusion) achieves lower latency than the original vLLM baseline, eliminating overhead from the profiling path.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Coordinated cross-level scheduling: Demonstrates that DP-engine dispatch and MoE expert placement are tightly coupled in MoE serving, and optimizing them jointly (with feedback) outperforms independent optimization.

2. Source-aware placement formulation: The MINLP formulation captures the three-way trade-off among expert load balance, source-dependent communication cost, and migration stability, providing a principled framework for expert placement.

3. Fine-grained pressure signals: Shows that token-level metrics (remaining prefill work, KV-cache usage, MoE expert pressure) are more accurate indicators of engine load than request counts.

Practical Implications

1. Deployment-ready: Implemented on vLLM with only ~1.7K lines of Python and Triton code, demonstrating feasibility in production systems.

2. Generalizability: The mechanisms rely on general serving signals (KV-cache usage, prefill work, expert activation counts, source-DP-to-expert routing statistics) commonly exposed by MoE serving systems, not model-specific internals.

3. Scalability: The coordinated approach is expected to provide greater benefits in large multi-node clusters where communication and placement costs become increasingly important.

4. Complementary to disaggregation: Gimbal's pressure signals could serve as routing inputs in prefill/decode disaggregated deployments (e.g., Splitwise, DistServe, Mooncake).

---

Conclusion

Gimbal presents a coordinated cross-level scheduling system for MoE-based LLM serving that closes the loop between DP-engine scheduling and MoE expert placement. Key takeaways:

1. Fine-grained DP scheduling using online backend pressure signals (KV-cache usage, remaining prefill work, queue backlog, MoE expert pressure) effectively mitigates request-side imbalance.

2. Source-aware expert placement using online source-DP-to-expert routing statistics reduces both expert hotspots and cross-DP communication costs.

3. Coordinated feedback (feeding backend MoE pressure back to the DP scheduler) is critical: it prevents expert-side hotspots from propagating into DP-engine imbalance.

4. Empirical results demonstrate consistent improvements: 42.9% lower TTFT, 33.3% lower TPOT, and 3.0% higher throughput compared to vLLM.

Future directions include:

• Evaluating on additional MoE models
• Extending to multi-node deployments
• Exploring more adaptive expert-placement policies
• Extending the feedback loop to attention/expert disaggregated architectures