# Coordinated Scheduling for MoE LLM Serving (Gimbal)

> Gimbal's coordinated cross-level scheduling, jointly optimizing DP-engine dispatch and source-aware expert placement, cuts MoE LLM serving TTFT by 42.9% and TPOT by 33.3% versus vLLM.

- **Source:** [arXiv](https://arxiv.org/abs/2606.15177)
- **Published:** 2026-08-29
- **Permalink:** https://picx.dev/p/yO0CNl
- **Whiteboard:** https://picx.dev/p/yO0CNl/image

## Summary

## Summary (Overview)

- **Gimbal** is a coordinated cross-level scheduling system for Mixture-of-Experts (MoE) large language model (LLM) serving that jointly optimizes frontend data-parallel (DP) engine scheduling and backend expert placement.
- It introduces a **fine-grained DP-engine scheduler** that uses online backend pressure signals (KV-cache usage, remaining prefill work, queue backlog, MoE expert pressure) instead of coarse request counters.
- It extends expert load balancing with **source-aware routing statistics** (source-DP-to-expert activation matrices) and a heuristic calibrated against a mixed-integer nonlinear program (MINLP) to jointly optimize expert load, communication cost, and migration stability.
- Evaluation on a 4×H100 testbed with Qwen3-30B-A3B and BurstGPT workloads shows Gimbal reduces average TTFT by **42.9%** and average TPOT by **33.3%** compared to vLLM, while improving high-load throughput by **3.0%**.
- An ablation study confirms that **coordinated** cross-level scheduling outperforms simply enabling DP-side and EP-side optimizations independently.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Large Language Models (LLMs) have advanced rapidly, but scaling dense Transformer models increases computational demands, inference latency, and deployment costs. Sparse architectures, particularly **Mixture-of-Experts (MoE) models**, address this by activating only a small subset of experts per token, scaling model capacity while keeping per-token inference costs manageable. DeepSeek-series models demonstrate that sparse MoE models achieve competitive capability with substantially reduced activated computation.

### Key Challenges in MoE Serving

MoE inference systems combine three parallelism paradigms:
- **Data Parallelism (DP)**: replicates inference engines to improve throughput
- **Expert Parallelism (EP)**: distributes experts across GPUs
- **Tensor Parallelism (TP)**: partitions dense model computation across GPUs

This layered design introduces coordination challenges:

1. **Coarse request scheduling causes inter-engine imbalance**: Request-count-based dispatching assumes equal execution cost, but real workloads (e.g., GPT-4 traces) exhibit highly skewed input lengths. A 2K-token request consumes 187.5 MiB of KV cache vs. 18.75 MiB for a 200-token request.

2. **Sparse routing causes intra-model imbalance**: Token traffic is not uniformly distributed across experts. Profiling shows 83.4% of Layer-23 traffic from DP0 and 66.5% of Layer-36 traffic from DP1 route to remote DP groups under current placement, creating cross-DP communication overhead.

3. **Inter-engine and intra-model imbalance are coupled**: DP-engine dispatch determines where tokens enter the all-to-all communication path, while expert placement determines whether selected experts are local or remote. Expert migration costs average 1.08s for first rearrangement and 0.72s for subsequent ones.

### Limitations of Current Systems

Existing systems make scheduling decisions in isolation:
- Frontend schedulers (vLLM, SGlang, FastServe) use request-count-based routing or phase-aware policies but don't expose fine-grained engine pressure
- Backend expert balancers (EPLB, MoETuner) use aggregate expert activation counts but don't distinguish which DP source generates traffic
- Sem-MoE uses offline profiling that becomes stale as workloads change

---

## Methodology

### System Architecture

Gimbal coordinates two main components through a feedback loop:

1. **Fine-grained DP-engine Scheduler** (Section 4)
2. **Source-aware Expert Load Balancer** (Section 5)

Backend engine traces and expert pressure signals flow upward to the scheduler, while expert migration actions flow downward to the EP region.

### Fine-grained DP-engine Scheduling

**Runtime Trace Collection**: Each engine periodically reports:
- Remaining prefill tokens from running requests
- Waiting prefill tokens in local queue
- KV-cache usage
- Backend MoE expert pressure

**Pressure-Aware Engine Selection** (Algorithm 1):
- If KV-cache usage is high (>90%) and the gap across engines exceeds 10%, select the engine with lowest KV usage
- Otherwise, compute a pressure score for each engine:
  $$score_i = pre_i^{rem} + wait_i + comp_i + P_{kv}(kv_i) + P_{moe}(moe_i)$$
- A lightweight compensation term ($comp_i$) accounts for recently dispatched requests before trace updates

**Intra-Engine Ordering** (Algorithm 2): SJF-style queue ordering with aging, using prefill token count as job-size estimate. Requests waiting longer than threshold $\theta_{age} = 5$ seconds are promoted to high priority.

### Source-aware Expert Load Balancing

**Online Traffic Profiling**: Two statistics are collected along the MoE dispatch path:
- $B_{l,e}$: aggregate expert activation counts (computation load)
- $A_{l,s,e}$: layer-wise source-DP-to-expert activation matrix (source-aware routing profile)

**Placement Objective**: The MINLP formulation optimizes three goals:

Load balance:
$$C_{\mathrm{load}} = \sum_{l} \sum_{g} \left(L_{l,g} - \bar{L}_{l}\right)^{2}$$

Source-aware communication:
$$C_{\mathrm{comm}} = \sum_{l} \sum_{s} \sum_{e} \sum_{g} A_{l,s,e} D_{s,g} x_{l,e,g}$$

Migration cost:
$$C_{\mathrm{mig}} = \sum_{l} \sum_{e} \sum_{g} M_{l,e,g} \left| x_{l,e,g} - x_{l,e,g}^{0}\right|$$

Overall objective:
$$\min_{x} C_{\mathrm{load}} + C_{\mathrm{comm}} + C_{\mathrm{mig}}$$

**Online Heuristic**: Since solving the MINLP takes ~15 seconds for a 48-layer model (prohibitive for online serving), Gimbal uses a greedy heuristic with local placement score:
$$S_{l,e,g} = \alpha C_{l,e,g}^{\mathrm{comm}} + \beta C_{l,e,g}^{\mathrm{load}} + \gamma C_{l,e,g}^{\mathrm{mig}}$$

Calibrated parameters: $(\alpha, \beta, \gamma) = (1.0, 0.0025, 1.0)$, preserving >80% of MINLP placement decisions with communication within 0.6% of the offline reference.

---

## Empirical Validation / Results

### Experimental Setup

- **Testbed**: 4×NVIDIA HGX H100 80GB SXM5 GPUs, NVLink-connected, 2×32-core Intel Xeon Platinum 8462Y+
- **Deployment**: DP=2, TP=2, EP=4
- **Model**: Qwen3-30B-A3B (MoE)
- **Workloads**: BurstGPT traces reshaped into five distributions (Random, Central, Descending, Two-end, Average)
- **Baselines**: vLLM (production-grade), MoE-Tuner (offline affinity-based), Sem-MoE (oracle variant with perfect routing knowledge)

### Key Results

| Metric | vs. vLLM | vs. MoE-Tuner | vs. Sem-MoE |
|--------|----------|---------------|-------------|
| Mean TTFT | -42.9% | -47.0% | -34.7% |
| Mean TPOT | -33.3% | -36.2% | -29.5% |
| P99 TTFT | -44.3% | -48.6% | -33.8% |
| End-to-end latency | -32.0% | -34.7% | -28.5% |
| High-load throughput | +3.0% | — | — |

Performance improvements grow with load: TTFT reduction over vLLM increases from 33.1% at RPS=2 to 48.3% at RPS=4.

### Ablation Study

| Configuration | TTFT reduction | TPOT reduction |
|---------------|---------------|----------------|
| Gimbal-DP only | 25.1% | 13.4% |
| Gimbal-EP only | 26.2% | 22.7% |
| Gimbal-All (No collaboration) | 29.8% | 27.3% |
| **Gimbal-All (Coordinated)** | **41.4%** | **32.0%** |

The coordinated design (feeding backend MoE pressure to DP scheduler) provides an additional 16.5% TTFT and 6.5% TPOT reduction over independent optimizations.

### MINLP Calibration Impact

Calibrating the greedy expert-placement policy against the offline MINLP reference reduces:
- Average TTFT by 10.8%
- Average TPOT by 9.2%

compared to the uncalibrated greedy policy.

### Runtime Behavior at RPS=4

- Cross-engine prompt-throughput gap: reduced from 1485.66 tokens/s (vLLM) to 768.43 tokens/s (Gimbal-DP)
- Average running requests per engine: reduced from 87.56 (vLLM) to 71.53 (Gimbal-EP)

### System Overhead

The optimized source-aware matrix collection path (with Triton kernel fusion) achieves **lower latency than the original vLLM baseline**, eliminating overhead from the profiling path.

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Coordinated cross-level scheduling**: Demonstrates that DP-engine dispatch and MoE expert placement are tightly coupled in MoE serving, and optimizing them jointly (with feedback) outperforms independent optimization.

2. **Source-aware placement formulation**: The MINLP formulation captures the three-way trade-off among expert load balance, source-dependent communication cost, and migration stability, providing a principled framework for expert placement.

3. **Fine-grained pressure signals**: Shows that token-level metrics (remaining prefill work, KV-cache usage, MoE expert pressure) are more accurate indicators of engine load than request counts.

### Practical Implications

1. **Deployment-ready**: Implemented on vLLM with only ~1.7K lines of Python and Triton code, demonstrating feasibility in production systems.

2. **Generalizability**: The mechanisms rely on general serving signals (KV-cache usage, prefill work, expert activation counts, source-DP-to-expert routing statistics) commonly exposed by MoE serving systems, not model-specific internals.

3. **Scalability**: The coordinated approach is expected to provide greater benefits in large multi-node clusters where communication and placement costs become increasingly important.

4. **Complementary to disaggregation**: Gimbal's pressure signals could serve as routing inputs in prefill/decode disaggregated deployments (e.g., Splitwise, DistServe, Mooncake).

---

## Conclusion

Gimbal presents a coordinated cross-level scheduling system for MoE-based LLM serving that closes the loop between DP-engine scheduling and MoE expert placement. Key takeaways:

1. **Fine-grained DP scheduling** using online backend pressure signals (KV-cache usage, remaining prefill work, queue backlog, MoE expert pressure) effectively mitigates request-side imbalance.

2. **Source-aware expert placement** using online source-DP-to-expert routing statistics reduces both expert hotspots and cross-DP communication costs.

3. **Coordinated feedback** (feeding backend MoE pressure back to the DP scheduler) is critical: it prevents expert-side hotspots from propagating into DP-engine imbalance.

4. **Empirical results** demonstrate consistent improvements: 42.9% lower TTFT, 33.3% lower TPOT, and 3.0% higher throughput compared to vLLM.

**Future directions** include:
- Evaluating on additional MoE models
- Extending to multi-node deployments
- Exploring more adaptive expert-placement policies
- Extending the feedback loop to attention/expert disaggregated architectures

---

_Markdown view of https://picx.dev/p/yO0CNl, served by PicX — AI-generated visual whiteboard summaries of research papers._