# FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE Serving

> FastEEP enables 4.5x–11.7x faster elastic expert-parallel scaling for MoE serving by reusing workers, virtual addresses, and communicators instead of rebuilding instances.

- **Source:** [arXiv](https://arxiv.org/abs/2610.18451)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/hhnAkm
- **Whiteboard:** https://picx.dev/p/hhnAkm/image

## Summary

## Summary (Overview)

- **FastEEP** is a fast intra-instance elastic scaling system for Mixture-of-Experts (MoE) serving, enabling dynamic adjustment of Expert Parallelism (EP) degree at runtime without full instance teardown and reconstruction.
- The core principle is **Partial Reuse**: separating stable structural state (workers, communication topology, virtual addresses) from active state that changes with EP size, enabling coordinated resource reclamation and reconfiguration.
- FastEEP achieves **4.5×–11.7× faster scaling** than vLLM Elastic EP (geometric mean 7.0×) and **22.0× over restart**, with scaling completed in 4.2–13.4 s across six models.
- The system addresses three key challenges: immutable communication groups, memory reclamation destroying execution contexts, and safe worker retention/reuse.
- FastEEP is implemented on vLLM (~5K lines of code) and preserves steady-state throughput while enabling fine-grained, reversible scaling.

## Introduction and Theoretical Foundation

### Background and Motivation

Large-scale Expert Parallelism (EP) has become the standard deployment configuration for MoE models in production. Under EP, experts are partitioned across multiple GPUs, tokens are dispatched to their routed experts, and outputs are aggregated. While a large EP degree reduces per-GPU weight memory footprint (freeing HBM for KV caches), it compromises scaling elasticity.

Real-world LLM workloads exhibit extreme short-term traffic volatility:
- In the BurstGPT trace (1.4M requests), **59% of ten-second windows have no incoming requests**, while busy periods show a **25× swing** between median and 99th-percentile arrival rates.
- The Azure conversation trace (27M requests) shows traffic fluctuating by **1.7×** with transient spikes.

This creates a severe dilemma: provisioning for peak traffic causes idle overheads during troughs, while provisioning for median traffic violates SLOs during bursts.

### Key Insight: Partial State Reuse

FastEEP divides serving state into **persistent** (reusable) and **reconfigurable** components:

| State | Cost | EP-dep. | Today | Reused |
|---|---|---|---|---|
| Worker process, executor, kernels | 24 s startup | no | destroyed | retained |
| Communication domains, RDMA buffers | 13 s setup | members only | rebuilt | subdomain derived |
| CUDA context | 0.32 GiB | no | destroyed | retained |
| CUDA Graphs | 36 s, 1.47 GiB | no | recaptured | replayed |
| Weights, KV cache, workspaces | 9.62 / 71.04 GiB | sizes only | reloaded | pages refilled |
| Expert placement, request routing | - | yes | rebuilt | rewritten in place |

### Three Rigid Bindings That Force Rebuild

1. **Immutable Communicator Binding**: Collective communication groups are constructed for fixed rank sets; dropping/adding a rank forces destruction and re-initialization.
2. **Static Memory & Address Binding**: Captured CUDA graphs and registered RDMA regions bind to hardcoded virtual addresses; deallocation invalidates execution paths.
3. **Tightly Coupled Worker Execution**: A worker concurrently manages local requests, KV caches, and hosted experts; it cannot pause without triggering remote hangs.

## Methodology

### Design-1: Reusing Communicators Through Flexible Group Splitting

FastEEP creates a **parent NCCL communicator** over the maximum worker set and retains its rank identities and bootstrap state. Each resize derives a subgroup containing only active workers:

- **Collective communication**: Uses NCCL communicator split, where active ranks receive the same color and form a subcommunicator; inactive ranks remain outside.
- **All-to-All communication**: Uses a NIXL mask to hide inactive ranks while connections among active peers remain in place.
- Both views (NCCL subgroup and NIXL mask) are published together with the expert mapping at a safe point.

### Design-2: Reusing GPU Context with Fine-Grained Offloading

Using CUDA Virtual Memory Management (VMM), FastEEP creates separate CuMem pools for shared weights, expert weights, KV cache, and CUDA Graph buffers:

- **Scale-down**: Unmaps and releases physical pages while retaining virtual addresses, tensor metadata, and object references.
- **Scale-up**: Maps new pages at the same addresses, preserving kernel arguments and tensor pointers.

**Weight restoration policies**:
- Non-expert weights (replicated): discarded and restored from active ranks over NVLink.
- Expert weights: transferred via device-to-device path based on control-plane computed placement.
- KV cache: three policies—discard and recompute, migrate asynchronously, or swap to pinned host memory.

**CUDA Graph preservation**: Capture-time allocations route to a dedicated graph pool; physical pages are unmapped while virtual addresses are preserved, enabling direct replay without recapture.

### Design-3: Safe Worker Retention and Reuse Through Coordinated Sleep

The shadow worker state retains execution identity without consuming GPU compute:
- Process, CPU threads, executor, virtual addresses, and control state persist.
- Physical memory for weights, KV cache, graph buffers, and communication buffers is unmapped or offloaded.
- Shadow workers bypass forward pass, launching no attention, expert, or communication kernels.

**Scale-down algorithm**:
1. Mark target as draining; remove from active DP set.
2. Recompute expert-to-physical-slot mapping; copy missing experts to active workers.
3. Publish mapping and NIXL mask together.
4. Offload weights, KV cache, and CUDA Graph buffers; release HBM.
5. Derive new subgroup; update NIXL mask and endpoints; switch views at safe point.

**Scale-up algorithm**:
1. Select shadow workers to wake; remap physical pages.
2. Reload CUDA Graph and communication buffers from CPU.
3. Copy weights and KV blocks from active ranks to original addresses.
4. Derive expanded subgroup; register NIXL buffers; establish endpoints.
5. Atomically publish expert mapping, communication view, and request routing.

## Empirical Validation / Results

### End-to-End Scaling Latency

Across all models and six transitions among {4, 8, 12, 16}:

| System | Scaling Time | Speedup vs. Elastic EP | Speedup vs. Restart |
|---|---|---|---|
| FastEEP | 4.2–13.4 s | 4.5×–11.7× (7.0× GM) | 22.0× |
| FullAsync variant | 3.2–9.8 s | 9.3× GM | 29.3× |
| vLLM Elastic EP | 36.4–92.0 s | - | - |
| Restart | 126–293 s | - | - |

### Phase Breakdown (Qwen3-30B, EP 8→4)

**vLLM Elastic EP scale-down (47 s)**: standby setup 10 s, expert resharding 9 s, compilation/warmup 11 s, graph capture 15 s.

**FastEEP scale-down (4.21 s)**: resize (expert layout + communication view) 2.81 s, buffer offloading 0.92 s, NIXL buffer destruction 0.24 s.

**FastEEP scale-up (7.15 s)**: worker wake 1.09 s, weight refill 0.85 s, NIXL connectivity restore 2.50 s, resize 2.55 s.

### Scaling While Serving

For an 8 → 4 → 8 cycle under load (concurrency 128→64→128):

| Transition | FastEEP | Elastic EP | Restart |
|---|---|---|---|
| 8 → 4 | 4.2 s | 31 s | 141 s |
| 4 → 8 | 7.3 s | 44 s | 149 s |

### Performance Across EP Sizes

- At concurrency 4, increasing EP from 4 to 16 yields **0.88×–0.99×** the EP-4 throughput (wider EP underutilized at low load).
- At concurrency 256, the same change improves throughput by **1.85×–2.30×** depending on request shape.
- Prefill-heavy requests benefit most; decode-heavy requests exhibit flatter scaling trends.
- Increasing EP expands available KV cache, valuable for high-concurrency, long-context workloads.

### Multi-Model Resource Sharing

Two colocated services (Qwen3-30B-A3B and DeepSeek-V2-Lite) can release and reassign GPUs without instance reconstruction. Driver-reported free memory rose from 34,191 to 64,613 MiB after an 8→4 scale-down. Cross-service scaling imposes an ordering dependency: released pages return to the node allocator, so dependent transitions must be ordered.

## Theoretical and Practical Implications

### Space-Time Trade-off

Fast scaling trades memory for reconstruction time. FastEEP retains only metadata needed for reuse while releasing or offloading data resources unused by the active configuration—enabling fast transitions without the memory cost of an entire inactive configuration.

### GPU Resource Reclamation

Persistence does not imply GPU residency. After each resize, FastEEP offloads inactive GPU data buffers to CPU memory (2–4 GB, excluding KV cache) while retaining only metadata. Initializing at maximum world size does not require corresponding GPU resources to remain occupied throughout execution.

### Design Principles

The three rigid bindings (immutable communicators, static memory addresses, tightly coupled worker execution) stem from software conventions rather than hardware constraints. FastEEP demonstrates that systematically breaking all three enables low-latency intra-instance EP scaling on live workloads.

### Practical Impact

- Enables fine-grained scaling granularity down to single-GPU level.
- Supports MaaS platforms, RL rollouts, and spot workloads contending for shared GPUs.
- Open source implementation submitted to vLLM as a PR.
- Design scales to larger clusters via distributed state offloading (decentralized approach).

## Conclusion

FastEEP operationalizes **partial state reuse** for intra-instance EP scaling in MoE serving. By preserving workers, stable virtual addresses, and startup communication topology while selectively reclaiming and reconfiguring GPU resources, FastEEP:

- Reduces scaling latency by **4.5×–11.7×** over vLLM Elastic EP and **22.0×** over restart.
- Enables scaling in **4.2–13.4 s** across six models and six transitions.
- Preserves steady-state throughput (no added work to inference path).
- Enables colocated services to release and reassign GPUs without reconstruction.

**Limitations**: FastEEP scales only within the maximum topology initialized at startup; scale-up requires sufficient available GPU memory; larger topology changes still require reconstruction.

**Future directions**: Extending to larger clusters via distributed state offloading, integration with autoscaling policies, and support for prefill/decode disaggregation deployments.

---

_Markdown view of https://picx.dev/p/hhnAkm, served by PicX — AI-generated visual whiteboard summaries of research papers._