# MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

> MoE-Prefill decouples expert placement from activation routing, asynchronously gathering expert weights to eliminate AllToAll and achieve 1.35–1.37× throughput over baselines.

- **Source:** [arXiv](https://arxiv.org/abs/2605.02960)
- **Published:** 2026-08-22
- **Permalink:** https://picx.dev/p/BrYNX3
- **Whiteboard:** https://picx.dev/p/BrYNX3/image

## Summary

## Summary

- **Problem**: Prefill-only workloads (classification, ranking, verification) dominate production LLM traffic (65.3% of input tokens) but existing distributed MoE serving strategies (tensor, expert, pipeline parallelism) introduce significant redundant computation, communication, and synchronization overheads, degrading efficiency to below 16% MFU.
- **Core Insight:** These overheads stem from coupling expert placement with synchronous activation routing—a design inherited from decoding-era serving. Large-batch prefill's long, compute-bound forward passes create a per-layer window wide enough to stream expert weights in the background.
- **Proposed Solution**: MoE-Prefill, a prefill-only serving system with two tiers:
  - **Backend (AsyncEP)**: Gathers experts by weight rather than routing by activation, replacing per-layer AllToAll with asynchronous weight AllGather fully overlapped with computation.
  - **Frontend**: Co-enforces a physically-derived saturation threshold through prefix-aware routing and true-FLOPs load tracking.
- **Results**: On Qwen3-235B-A22B across four hardware/precision configurations, MoE-Prefill delivers **1.35–1.37× throughput** over the strongest baseline on real-world workloads and up to **1.59×** on long-context synthetic workloads, sustaining **29.8–36.2% per-GPU MFU**.
- **Key Contribution**: Expands the deployable hardware envelope from "≥4 GPUs required" to "1–8 GPUs" (a 4× broader range), treating expert weights as schedulable resources rather than static parameters.

## Introduction and Theoretical Foundation

### Background
- LLMs are increasingly used for **discriminative tasks** (classification [52,61], moderation [29], recommendation [69], factual verification [44]) where the answer is determined by logits of a single forward pass with no autoregressive decoding.
- Production measurements show prefill-only workloads account for **65.3% of all input tokens** served (Fig. 1).
- Prefill-only workloads differ from generative serving along three axes:
  1. **Throughput-oriented** rather than latency-sensitive
  2. **Extremely large batch sizes** yielding long compute-bound forward passes
  3. **Abundant prefix sharing** (system prompts, user profiles, document headers)

### The MoE Challenge
State-of-the-art open-weight LLMs (Qwen3, DeepSeek-V3, Mixtral, gpt-oss) adopt MoE architecture where total parameter counts grow far faster than per-GPU HBM, forcing distributed execution. This introduces three redundancies:

1. **Redundant computation**: Sub-saturated per-device GEMMs and routing-imbalance stragglers
2. **Redundant memory**: Full-resident expert weights, k-way activation expansion, duplicated prefix KV caches
3. **Redundant communication**: Two synchronous AllToAll operations per MoE layer under expert parallelism

### Theoretical Foundation
The paper identifies that these overheads stem from a **decoding-era design choice**: coupling expert placement with synchronous activation routing. The key insight is that in prefill-only serving:

> "If expert weights are streamed into the window in the background, per-layer AllToAll can be removed from the critical path entirely. Concretely, experts are gathered by **weight rather than routed by activation**, so every GPU holds the complete expert set for the current layer, and dispatch becomes a local operation."

### Key Notation
- $B$: batch size, $S$: sequence length, $L$: layers, $H$: hidden size
- $N_{kv}$: KV heads, $d_h$: head dimension, $h$: MoE intermediate size
- $k$: top-k routing fan-out, $b$: bytes/element (BF16: $b=2$, FP8: $b=1$)
- $P$: parallel degree, $t_{EP}$: slowest EP data transfer per layer
- $F_{GPU}$: GPU peak FLOP rate

## Methodology

### System Architecture

MoE-Prefill is a **two-tier serving system**:

**Frontend (§7):**
1. **Normalization**: Transforms each request into prefill-only form (single-token classification, single-choice selection, multi-selection decomposed into binary siblings)
2. **Scheduling rules**:
   - Co-locate same-prefix requests to maximize KV reuse (F1)
   - Measure load in true FLOPs after prefix-sharing credit (F2)
   - Stop admitting new requests once accumulated FLOPs reach saturation threshold (F3)

**Backend (AsyncEP, §6):**
- Runs each batch as pure DP attention with asynchronous expert parallelism
- Expert weights for upcoming layers gathered in background via D2D AllGather over NVLink
- Optional CPU-DRAM offloading (H2D PCIe prefetch) for further-ahead layers
- KV-cache-free mode disables KV storage for workloads without prefix reuse

### The Saturation Threshold

The saturation threshold $T$ governs the overlap:

$$T = t_{EP} \times F_{GPU} \times \gamma$$

where $\gamma \geq 1$ (e.g., 1.2) absorbs transient jitter. This is a physical quantity computed once at startup from hardware and model configuration.

### Execution Model

**D2D-only AsyncExec (Fig. 7a):**
- Each GPU holds full attention weights + 1/N of expert weights partitioned by expert index
- All GPUs replicate the complete expert set for the first MoE layer
- After computing attention locally, each GPU evaluates the current MoE layer entirely on-device while initiating a background D2D AllGather over NVLink to collect the next layer's complete expert weights

**With offloading (Fig. 7b):**
- Each GPU retains 1/N expert weights for only a few upcoming layers
- Complete expert weights for all layers live in CPU pinned memory as offloaded backing store
- Two-channel pipeline: D2D AllGather (NVLink) for next layer + H2D PCIe prefetch for further-ahead layers

### Frontend Scheduling

**Prefix-Aware Routing:**
- Routes each request to the GPU whose resident prefix KV cache shares the longest common prefix
- Block-granularity matching (16 tokens/block) aligned with PagedAttention/RadixAttention

**Compute-Aware Tracking:**
- Cost increment when request $r$ with prefix length $P_r$ and suffix length $S_r$ is assigned to GPU with $M_r$ prefix tokens already cached:

$$\Delta_r = C_{pfx}(P_r - M_r) + C_{sfx}(S_r, P_r)$$

where $C_{sfx}(S,P) = C_{FFN}(S) + C_{self}(S) + C_{cross}(S,P)$ combines linear feed-forward, $O(S^2)$ suffix self-attention, and $O(S \cdot P)$ cross-attention.

## Empirical Validation / Results

### Evaluation Setup

- **Models**: Qwen3-235B-A22B (128 experts, top-8 routing, ~22B activated parameters)
- **Hardware**: 8×A100 (80GB, BF16), 8×H100 (80GB, BF16 and FP8), 8×H200 (141GB, FP8)
- **Baselines**: DP+EP, DP+TP, TP+EP, TP+TP, PP+PP, plus PrefillOnly-augmented variants
- **Workload**: Aggregated real-world workload from six benchmarks (Table 2)

### Key Results

**End-to-End Throughput (Fig. 9):**
- MoE-Prefill achieves the highest throughput in every configuration cell
- **1.37×** on 8×A100 (BF16), **1.36×** on 8×H100 (BF16), **1.35×** on 8×H100 (FP8), **1.37×** on 8×H200 (FP8) over strongest baseline
- Baselines show sub-linear scaling (several plateau/regress past 4 GPUs), while MoE-Prefill scales near-linearly

**Frontend-Backend Co-Design (Fig. 10):**
- At 8 GPUs, frontend adds: **+18%** (A100 BF16), **+17%** (H100 BF16), **+18%** (H100 FP8), **+16%** (H200 FP8) over backend-only
- Contribution grows with parallel degree

**Synthetic No-Prefix-Reuse Workloads (Fig. 11):**
- Backend alone (DP+AsyncExec) achieves **1.30×** (short), **1.30×** (medium), **1.52×** (long), **1.49×** (ultra-long) at 8 GPUs
- Improvement grows with context length

**Memory Scalability & MFU (Fig. 12):**
- Deployable envelope widens from "≥4 GPUs" to "1–8 GPUs" (4× broader)
- Per-GPU MFU: **29.8–36.2%** across all configurations
- Baselines' best 8-GPU cell (TP+TP at 128K, 20.09%) falls below MoE-Prefill's worst cell (29.84%)

**Accuracy (Tables 3-4):**
- Prefill-only logit scoring within ±3.6 pp of autoregressive decoding on 7/9 tasks
- Decoding-mode contribution isolated to ~1 pp

## Theoretical and Practical Implications

1. **Decoupling expert placement from activation routing**: The paper demonstrates that expert weights should be treated as schedulable resources rather than static parameters, applicable whenever per-layer compute window covers transfer latency (including long-context reasoning and speculative-decode verification).

2. **Frontend-backend co-design**: The saturation threshold T provides a clean interface between scheduling policy and execution engine, enabling load balancing to emerge structurally rather than through explicit optimization.

3. **Hardware flexibility**: The hybrid offloading approach turns HBM capacity into a deployment knob, enabling large-MoE serving on commodity hardware.

4. **Communication reduction**: Replacing per-layer AllToAll with background weight AllGather removes the primary bottleneck in distributed MoE serving, achieving near-zero on-path communication.

5. **Complementarity with existing systems**: MoE-Prefill is complementary to prefill/decode disaggregation and can compose with other optimizations like BlendServe's resource-profile overlap.

## Conclusion

MoE-Prefill addresses the three structural redundancies in distributed MoE serving by inverting the coupling between expert placement and activation routing. Through AsyncExec (asynchronous expert parallelism) and a co-designed frontend enforcing a saturation threshold, it achieves:

- **1.35–1.37× throughput** over baseline on real-world workloads
- **1.59×** on long-context synthetic workloads
- **29.8–36.2% MFU** across 1–8 GPUs
- **4× broader deployable hardware envelope**

**Future directions**: The authors suggest the core insight—expert weights as schedulable resources—applies broadly, including to long-context reasoning and speculative-decode verification. The saturation threshold T could be extended to dynamic calibration for workload drift, and the frontend-backend co-design pattern could be applied to other serving systems.

---

_Markdown view of https://picx.dev/p/BrYNX3, served by PicX — AI-generated visual whiteboard summaries of research papers._