Summary (Overview)

• FastAFD is the first attention–FFN disaggregation (AFD) inference engine built specifically for GB200 NVL72 rack-scale NVLink systems, with kernels, microbatching, and coordination tuned for the rack.
• Key empirical finding: Against per-workload-tuned vLLM wide expert parallelism (wide EP), AFD achieves throughput advantages of up to 1.17× on Qwen3-235B and 1.16× on Qwen3.5-397B, while matching parity (0.99–1.02×) on MiniMax-M2.5.
• Analytical model: A per-layer overlap model predicts AFD throughput within 2.9% mean error using only profiles of colocated vLLM deployments—no FastAFD runs needed—enabling operators to decide before deploying.
• Speedup decomposition: AFD's speedup factors into a batch gain (larger batches from freed HBM), an overlap gain (MoE hidden behind attention), and a disaggregation cost (FFN GPUs serving no requests): $S_{\text{all}} = \text{batch gain} \times \text{overlap gain} \times \text{disaggregation cost}$.
• Forward-looking insight: Heterogeneous FFN placement on SRAM-based accelerators (e.g., Groq LPX paired with Vera Rubin NVL72) could raise AFD's speedup to an ideal 2.48× with speculative decoding.

---

Introduction and Theoretical Foundation

Background

Mixture-of-experts (MoE) LLM decoding couples stateful attention (which reads growing KV caches from fixed per-GPU HBM) with stateless expert FFNs (which stream shared expert weights across requests). Two competing designs exist for serving these models on rack-scale systems:

1. Colocated wide EP: Every GPU stores a full copy of attention weights alongside a shard of routed experts, with experts sharded across 8–64 GPUs (EP8–EP64) over NVLink. This frees HBM for KV caches while each step executes attention, NVLink all-to-all routing, and expert FFN serially on the same GPU.

2. Attention–FFN Disaggregation (AFD): Separates MoE inference into an attention pool (holding KV caches) and a dedicated FFN pool at a chosen attention-to-FFN (A:F) node ratio. Attention GPUs hold no expert weights, maximizing per-GPU decode batch.

Motivation

The paper addresses a critical gap: prior AFD results were obtained on older hardware against narrow colocated baselines (TP/EP4), or via analytical models that disagree with each other. The GB200 NVL72 rack (72 GPUs, 1.8 TB/s NVLink per GPU) strengthens both designs simultaneously:

• NVLink enables rack-scale AFD and heterogeneous racks built around it.
• The same fabric lets colocated serving shard experts across the rack (EP8–EP64), reclaiming most of the HBM that AFD frees.

Additionally, speculative decoding (standard in production) changes the question: verifying $q$ draft tokens per step reuses KV-cache reads but multiplies routed expert work by $q$, which lands on the critical path under wide EP but on a partially idle FFN pool under AFD.

Theoretical Foundation

The core theoretical contribution is the speedup decomposition:

$$S_{\text{all}} = \frac{b_A}{b_V} \times \frac{T_V}{T_A} \times \frac{r}{r+1}\tag{2}$$

Where:

• $r = N_A/N_F$: number of attention nodes per FFN node
• $b_A$, $b_V$: per-GPU decode batches of FastAFD and colocated wide EP
• $T_A$, $T_V$: decode step times of FastAFD and colocated wide EP
• Batch gain $b_A/b_V$: from removing expert weights from attention GPUs
• Overlap gain $T_V/T_A$: from taking MoE off the critical path
• Disaggregation cost $r/(r+1)$: from FFN GPUs holding no KV cache and serving no requests

---

Methodology

System Design (FastAFD)

Five key design choices differentiate FastAFD:

1. MoE-boundary split with matched kernels: Built on Mini-SGLang, keeping norms, QKV/O projections, and top-K routing on attention GPUs, using identical TensorRT-LLM FMHA attention kernels and block-FP8 expert precision as the vLLM baseline.

2. GPU-initiated, one-sided communication: Attention GPUs publish FP8 tokens and routing metadata into pre-allocated symmetric NVLink buffers (costing 24 SMs per attention GPU for the fused send/receive kernel), replacing host posts/polls.

3. Persistent fused MegaMoE kernel: FFN GPUs run a single kernel per layer fusing NVLink TMA pull (N2M), block-FP8 grouped GEMMs, activation, and push-back (M2N).

4. Two ping-pong microbatches ($k=2$): Splitting each attention GPU's batch into two microbatches hides remote MegaMoE execution behind local attention. NVL72's bandwidth covers the round trip with $k=2$, whereas slower interconnects require $k \geq 3$.

5. Whole-step CUDA graph capture with a one-step-ahead CPU coordinator: The host prepares step $t+1$ (batch composition, padding, buffer slots, KV-cache frees) while step $t$ runs.

Per-Layer Overlap Model

The predicted step time is:

$$\widehat{T}_A(r) = T_{\text{draft}} + 2\sum_{\ell=1}^{L} \max\{T_{a,\ell}, T_{e,\ell}(r)\}\tag{3}$$

Where:

• $T_{\text{draft}}$: draft-and-accept time outside the layer loop (zero in standard decoding)
• $T_{a,\ell}$: attention stage time in layer $\ell$ (KV-cache reads, norms, QKV/O projections, top-K routing)
• $T_{e,\ell}(r)$: FFN stage time in layer $\ell$ (NVLink dispatch, expert-weight reads, grouped-GEMM compute over routed tokens from $r$ attention nodes)

Experimental Setup

• Hardware: Single GB200 NVL72 rack (18 four-GPU nodes, 72-GPU NVLink domain)
• Models (chosen to span three architectural axes):
  • MiniMax-M2.5: 10B active of 229B, 256 experts (light-expert, full attention)
  • Qwen3-235B: 22B active of 235B, 128 experts (heavy-expert, full attention)
  • Qwen3.5-397B: 17B active of 397B, 512 experts (ultra-sparse, hybrid linear-plus-full attention)
• Workloads: Fixed prompts of 4K, 8K, or 16K tokens; steady-state closed-loop decode throughput; prefill excluded
• Baselines: vLLM wide EP tuned per workload (EP4 through EP64); FastAFD with A:F ratios from 1:1 to 17:1
• Metrics: All-GPU speedup $S_{\text{all}}$ (charging all GPUs), attention-GPU speedup $S_{\text{att}} = \frac{r+1}{r}S_{\text{all}}$, and $S_{\text{EP4}}$ against single-node EP4

---

Empirical Validation / Results

Standard Decoding Results

Table 1. Standard decoding (d = 0): context-tuned wide EP erases most of FastAFD's gain over EP4.

▲/▼: largest/smallest in each column; †: still rising at the 18-node rack limit.

Key findings:

• FastAFD achieves 1.30–2.36× over single-node EP4, but context-tuned wide EP compresses this to 0.99–1.17× all-GPU speedup.
• Attention-GPU speedup $S_{\text{att}}$ reaches 1.08–1.39× across all workloads.
• Wide EP peaks at interior widths (EP16–EP32), not EP64.

Speculative Decoding Results

Table 2. Speculative decoding (d = 3) at least halves FastAFD's best A:F ratio and raises $S_{att}$ on every workload.

Key findings:

• Speculative decoding ($q=4$ verified tokens per step) contracts optimal A:F ratios by more than half.
• $S_{\text{all}}$ on MiniMax-M2.5 rises from parity to 1.08×; Qwen3 holds at 1.10–1.13×.
• $S_{\text{att}}$ rises to 1.22–1.70× across all workloads.

Model Validation

• Equation (3) validated against all 88 measured FastAFD deployments (standard + speculative decoding).
• Mean error of 2.9% across 65 deployments up to each workload's optimal ratio (3.0% standard, 2.6% speculative); never exceeds 7.4%.
• Model-selected ratio retains 95–100% of measured peak throughput in standard decoding and 97–100% in speculative decoding.

Ablation Results

• Fused MegaMoE vs. modular DeepEP+DeepGEMM: cuts step latency by 50% (Qwen3) and 44% (MiniMax-M2.5) at 7:1 ratio; avoids 12+ GiB extra buffer memory per attention GPU.
• Microbatch count: $k=2$ achieves lowest step latency at both 3:1 and 7:1.
• One-step-ahead coordinator: disabling it adds 14 ms of synchronous host overhead per step, reducing throughput by 32%.
• MoE-boundary vs. CAD: FastAFD achieves 1.28–1.48× advantage over Core Attention Disaggregation, which saturates at 2:1–4:1 ratios (vs. FastAFD's 9:1–17:1).

---

Theoretical and Practical Implications

Win Condition

The coverage boundary and win condition are given by:

$$\frac{b_A}{b_V} \cdot \frac{T_V}{T_A(r)} > 1 + \frac{1}{r}, \quad \text{or} \quad \frac{T_V}{T_A(r)} > 1 + \frac{1}{r} \text{ for } b_A \approx b_V\tag{4}$$

• The FFN stage in layer $\ell$ stays hidden whenever $T_{e,\ell}(r) \leq T_{a,\ell}$ (coverage boundary).
• General form: batch gain × overlap gain must outweigh the disaggregation cost.
• Matched-batch form: when wide EP already reclaims most KV-cache memory, the overlap gain alone must exceed the disaggregation cost.

Model-Specific Insights

• MiniMax-M2.5 (light experts): sits at parity because neither gain repays even a small cost.
• Qwen3 (heavy experts): batch gain similar to MiniMax-M2.5 but with a real overlap gain, enough to win.
• Qwen3.5 (hybrid attention): leads on batch alone (1.4–1.6×) despite a slower step, until advantage shrinks to parity at 16K.

Provisioning Trajectories

• Full-attention MoEs: Optimal A:F ratio $r^*$ fans out with context (9:1 → 17:1 for MiniMax-M2.5; 5:1 → 11:1 for Qwen3), consuming only 6–13% of the GPU fleet.
• Hybrid Qwen3.5: Maintains flat 5:1 ratio across 4K–16K due to linear-attention layers with fixed-size recurrent state updates.
• Speculative decoding: Shifts $r^*$ to 2:1–8:1, requiring 11–33% of the rack for FFN nodes.

Heterogeneous FFN Placement

Placing the FFN pool on dedicated SRAM accelerators (e.g., Groq LPX paired with Vera Rubin NVL72):

• Removes the disaggregation cost: Per-GPU speedups rise from 0.99–1.17× to 1.08–1.39× (standard) and from 1.02–1.13× to 1.22–1.70× (speculative).
• Zero-FFN-cost ceiling (with SRAM-resident expert weights): speedups reach 1.20–1.76× (standard) and 1.38–2.48× (speculative), eliminating both the second pass of dense projections and uncovered FFN tail latency.

---

Conclusion

The paper answers "when does AFD pay on rack-scale NVLink?" through measurement and modeling:

1. AFD is a provisioning decision, not an architectural win: On GB200 NVL72, AFD ranges from parity to 1.17× over per-workload-tuned wide EP when every FFN GPU is counted.

2. The speedup decomposition (batch gain × overlap gain × disaggregation cost) explains which models benefit: heavy-expert models (Qwen3) win via overlap gain; hybrid models (Qwen3.5) win via batch gain at shorter contexts; light-expert models (MiniMax-M2.5) sit at parity.

3. The analytical model predicts AFD throughput within 3% using only colocated profiles, enabling deployment decisions without running AFD.

4. Two levers tilt the balance toward AFD:

   • Speculative decoding: Multiplies MoE work, enlarging the overlap gain (adds up to 8% to AFD's advantage on light-expert models).
   • Off-GPU expert accelerators: Heterogeneous placement removes the disaggregation cost, raising ideal speedup to 2.48× with speculative decoding.

FastAFD is open source at https://github.com/hao-ai-lab/FastAFD.