Summary

• Problem: Prefill-only workloads (classification, ranking, verification) dominate production LLM traffic (65.3% of input tokens) but existing distributed MoE serving strategies (tensor, expert, pipeline parallelism) introduce significant redundant computation, communication, and synchronization overheads, degrading efficiency to below 16% MFU.
• Core Insight: These overheads stem from coupling expert placement with synchronous activation routing—a design inherited from decoding-era serving. Large-batch prefill's long, compute-bound forward passes create a per-layer window wide enough to stream expert weights in the background.
• Proposed Solution: MoE-Prefill, a prefill-only serving system with two tiers:
  • Backend (AsyncEP): Gathers experts by weight rather than routing by activation, replacing per-layer AllToAll with asynchronous weight AllGather fully overlapped with computation.
  • Frontend: Co-enforces a physically-derived saturation threshold through prefix-aware routing and true-FLOPs load tracking.
• Results: On Qwen3-235B-A22B across four hardware/precision configurations, MoE-Prefill delivers 1.35–1.37× throughput over the strongest baseline on real-world workloads and up to 1.59× on long-context synthetic workloads, sustaining 29.8–36.2% per-GPU MFU.
• Key Contribution: Expands the deployable hardware envelope from "≥4 GPUs required" to "1–8 GPUs" (a 4× broader range), treating expert weights as schedulable resources rather than static parameters.

Introduction and Theoretical Foundation

Background

• LLMs are increasingly used for discriminative tasks (classification [52,61], moderation [29], recommendation [69], factual verification [44]) where the answer is determined by logits of a single forward pass with no autoregressive decoding.
• Production measurements show prefill-only workloads account for 65.3% of all input tokens served (Fig. 1).
• Prefill-only workloads differ from generative serving along three axes:
  1. Throughput-oriented rather than latency-sensitive
  2. Extremely large batch sizes yielding long compute-bound forward passes
  3. Abundant prefix sharing (system prompts, user profiles, document headers)

The MoE Challenge

State-of-the-art open-weight LLMs (Qwen3, DeepSeek-V3, Mixtral, gpt-oss) adopt MoE architecture where total parameter counts grow far faster than per-GPU HBM, forcing distributed execution. This introduces three redundancies:

1. Redundant computation: Sub-saturated per-device GEMMs and routing-imbalance stragglers
2. Redundant memory: Full-resident expert weights, k-way activation expansion, duplicated prefix KV caches
3. Redundant communication: Two synchronous AllToAll operations per MoE layer under expert parallelism

Theoretical Foundation

The paper identifies that these overheads stem from a decoding-era design choice: coupling expert placement with synchronous activation routing. The key insight is that in prefill-only serving:

"If expert weights are streamed into the window in the background, per-layer AllToAll can be removed from the critical path entirely. Concretely, experts are gathered by weight rather than routed by activation, so every GPU holds the complete expert set for the current layer, and dispatch becomes a local operation."

Key Notation

• $B$: batch size, $S$: sequence length, $L$: layers, $H$: hidden size
• $N_{kv}$: KV heads, $d_h$: head dimension, $h$: MoE intermediate size
• $k$: top-k routing fan-out, $b$: bytes/element (BF16: $b=2$, FP8: $b=1$)
• $P$: parallel degree, $t_{EP}$: slowest EP data transfer per layer
• $F_{GPU}$: GPU peak FLOP rate

Methodology

System Architecture

MoE-Prefill is a two-tier serving system:

Frontend (§7):

1. Normalization: Transforms each request into prefill-only form (single-token classification, single-choice selection, multi-selection decomposed into binary siblings)
2. Scheduling rules:
   • Co-locate same-prefix requests to maximize KV reuse (F1)
   • Measure load in true FLOPs after prefix-sharing credit (F2)
   • Stop admitting new requests once accumulated FLOPs reach saturation threshold (F3)

Backend (AsyncEP, §6):

• Runs each batch as pure DP attention with asynchronous expert parallelism
• Expert weights for upcoming layers gathered in background via D2D AllGather over NVLink
• Optional CPU-DRAM offloading (H2D PCIe prefetch) for further-ahead layers
• KV-cache-free mode disables KV storage for workloads without prefix reuse

The Saturation Threshold

The saturation threshold $T$ governs the overlap:

$$T = t_{EP} \times F_{GPU} \times \gamma$$

where $\gamma \geq 1$ (e.g., 1.2) absorbs transient jitter. This is a physical quantity computed once at startup from hardware and model configuration.

Execution Model

D2D-only AsyncExec (Fig. 7a):

• Each GPU holds full attention weights + 1/N of expert weights partitioned by expert index
• All GPUs replicate the complete expert set for the first MoE layer
• After computing attention locally, each GPU evaluates the current MoE layer entirely on-device while initiating a background D2D AllGather over NVLink to collect the next layer's complete expert weights

With offloading (Fig. 7b):

• Each GPU retains 1/N expert weights for only a few upcoming layers
• Complete expert weights for all layers live in CPU pinned memory as offloaded backing store
• Two-channel pipeline: D2D AllGather (NVLink) for next layer + H2D PCIe prefetch for further-ahead layers

Frontend Scheduling

Prefix-Aware Routing:

• Routes each request to the GPU whose resident prefix KV cache shares the longest common prefix
• Block-granularity matching (16 tokens/block) aligned with PagedAttention/RadixAttention

Compute-Aware Tracking:

• Cost increment when request $r$ with prefix length $P_r$ and suffix length $S_r$ is assigned to GPU with $M_r$ prefix tokens already cached:

$$\Delta_r = C_{pfx}(P_r - M_r) + C_{sfx}(S_r, P_r)$$

where $C_{sfx}(S,P) = C_{FFN}(S) + C_{self}(S) + C_{cross}(S,P)$ combines linear feed-forward, $O(S^2)$ suffix self-attention, and $O(S \cdot P)$ cross-attention.

Empirical Validation / Results

Evaluation Setup

• Models: Qwen3-235B-A22B (128 experts, top-8 routing, ~22B activated parameters)
• Hardware: 8×A100 (80GB, BF16), 8×H100 (80GB, BF16 and FP8), 8×H200 (141GB, FP8)
• Baselines: DP+EP, DP+TP, TP+EP, TP+TP, PP+PP, plus PrefillOnly-augmented variants
• Workload: Aggregated real-world workload from six benchmarks (Table 2)

Key Results

End-to-End Throughput (Fig. 9):

• MoE-Prefill achieves the highest throughput in every configuration cell
• 1.37× on 8×A100 (BF16), 1.36× on 8×H100 (BF16), 1.35× on 8×H100 (FP8), 1.37× on 8×H200 (FP8) over strongest baseline
• Baselines show sub-linear scaling (several plateau/regress past 4 GPUs), while MoE-Prefill scales near-linearly

Frontend-Backend Co-Design (Fig. 10):

• At 8 GPUs, frontend adds: +18% (A100 BF16), +17% (H100 BF16), +18% (H100 FP8), +16% (H200 FP8) over backend-only
• Contribution grows with parallel degree

Synthetic No-Prefix-Reuse Workloads (Fig. 11):

• Backend alone (DP+AsyncExec) achieves 1.30× (short), 1.30× (medium), 1.52× (long), 1.49× (ultra-long) at 8 GPUs
• Improvement grows with context length

Memory Scalability & MFU (Fig. 12):

• Deployable envelope widens from "≥4 GPUs" to "1–8 GPUs" (4× broader)
• Per-GPU MFU: 29.8–36.2% across all configurations
• Baselines' best 8-GPU cell (TP+TP at 128K, 20.09%) falls below MoE-Prefill's worst cell (29.84%)

Accuracy (Tables 3-4):

• Prefill-only logit scoring within ±3.6 pp of autoregressive decoding on 7/9 tasks
• Decoding-mode contribution isolated to ~1 pp

Theoretical and Practical Implications

1. Decoupling expert placement from activation routing: The paper demonstrates that expert weights should be treated as schedulable resources rather than static parameters, applicable whenever per-layer compute window covers transfer latency (including long-context reasoning and speculative-decode verification).

2. Frontend-backend co-design: The saturation threshold T provides a clean interface between scheduling policy and execution engine, enabling load balancing to emerge structurally rather than through explicit optimization.

3. Hardware flexibility: The hybrid offloading approach turns HBM capacity into a deployment knob, enabling large-MoE serving on commodity hardware.

4. Communication reduction: Replacing per-layer AllToAll with background weight AllGather removes the primary bottleneck in distributed MoE serving, achieving near-zero on-path communication.

5. Complementarity with existing systems: MoE-Prefill is complementary to prefill/decode disaggregation and can compose with other optimizations like BlendServe's resource-profile overlap.

Conclusion

MoE-Prefill addresses the three structural redundancies in distributed MoE serving by inverting the coupling between expert placement and activation routing. Through AsyncExec (asynchronous expert parallelism) and a co-designed frontend enforcing a saturation threshold, it achieves:

• 1.35–1.37× throughput over baseline on real-world workloads
• 1.59× on long-context synthetic workloads
• 29.8–36.2% MFU across 1–8 GPUs
• 4× broader deployable hardware envelope

Future directions: The authors suggest the core insight—expert weights as schedulable resources—applies broadly, including to long-context reasoning and speculative-decode verification. The saturation threshold T could be extended to dynamic calibration for workload drift, and the frontend-backend co-design pattern could be applied to other serving systems.