# FastAFD: When Does Attention-FFN Disaggregation Pay on Rack-Scale NVLink?

> FastAFD shows attention-FFN disaggregation beats per-workload-tuned vLLM wide EP by up to 1.17x on GB200 NVL72, while an analytical model predicts its speedups within 2.9% mean error.

- **Source:** [arXiv](https://arxiv.org/abs/2611.07149)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/rwQIIl
- **Whiteboard:** https://picx.dev/p/rwQIIl/image

## Summary

## Summary (Overview)

- **FastAFD** is the first attention–FFN disaggregation (AFD) inference engine built specifically for GB200 NVL72 rack-scale NVLink systems, with kernels, microbatching, and coordination tuned for the rack.
- **Key empirical finding**: Against per-workload-tuned vLLM wide expert parallelism (wide EP), AFD achieves throughput advantages of up to **1.17×** on Qwen3-235B and **1.16×** on Qwen3.5-397B, while matching parity (0.99–1.02×) on MiniMax-M2.5.
- **Analytical model**: A per-layer overlap model predicts AFD throughput within **2.9% mean error** using only profiles of colocated vLLM deployments—no FastAFD runs needed—enabling operators to decide before deploying.
- **Speedup decomposition**: AFD's speedup factors into a **batch gain** (larger batches from freed HBM), an **overlap gain** (MoE hidden behind attention), and a **disaggregation cost** (FFN GPUs serving no requests): $S_{\text{all}} = \text{batch gain} \times \text{overlap gain} \times \text{disaggregation cost}$.
- **Forward-looking insight**: Heterogeneous FFN placement on SRAM-based accelerators (e.g., Groq LPX paired with Vera Rubin NVL72) could raise AFD's speedup to an ideal **2.48×** with speculative decoding.

---

## Introduction and Theoretical Foundation

### Background

Mixture-of-experts (MoE) LLM decoding couples **stateful attention** (which reads growing KV caches from fixed per-GPU HBM) with **stateless expert FFNs** (which stream shared expert weights across requests). Two competing designs exist for serving these models on rack-scale systems:

1. **Colocated wide EP**: Every GPU stores a full copy of attention weights alongside a shard of routed experts, with experts sharded across 8–64 GPUs (EP8–EP64) over NVLink. This frees HBM for KV caches while each step executes attention, NVLink all-to-all routing, and expert FFN serially on the same GPU.

2. **Attention–FFN Disaggregation (AFD)**: Separates MoE inference into an attention pool (holding KV caches) and a dedicated FFN pool at a chosen attention-to-FFN (A:F) node ratio. Attention GPUs hold no expert weights, maximizing per-GPU decode batch.

### Motivation

The paper addresses a critical gap: prior AFD results were obtained on **older hardware** against **narrow colocated baselines** (TP/EP4), or via **analytical models** that disagree with each other. The GB200 NVL72 rack (72 GPUs, 1.8 TB/s NVLink per GPU) strengthens both designs simultaneously:

- NVLink enables rack-scale AFD and heterogeneous racks built around it.
- The same fabric lets colocated serving shard experts across the rack (EP8–EP64), reclaiming most of the HBM that AFD frees.

Additionally, **speculative decoding** (standard in production) changes the question: verifying $q$ draft tokens per step reuses KV-cache reads but multiplies routed expert work by $q$, which lands on the critical path under wide EP but on a partially idle FFN pool under AFD.

### Theoretical Foundation

The core theoretical contribution is the speedup decomposition:

$$
S_{\text{all}} = \frac{b_A}{b_V} \times \frac{T_V}{T_A} \times \frac{r}{r+1}\tag{2}
$$

Where:
- $r = N_A/N_F$: number of attention nodes per FFN node
- $b_A$, $b_V$: per-GPU decode batches of FastAFD and colocated wide EP
- $T_A$, $T_V$: decode step times of FastAFD and colocated wide EP
- **Batch gain** $b_A/b_V$: from removing expert weights from attention GPUs
- **Overlap gain** $T_V/T_A$: from taking MoE off the critical path
- **Disaggregation cost** $r/(r+1)$: from FFN GPUs holding no KV cache and serving no requests

---

## Methodology

### System Design (FastAFD)

Five key design choices differentiate FastAFD:

1. **MoE-boundary split with matched kernels**: Built on Mini-SGLang, keeping norms, QKV/O projections, and top-K routing on attention GPUs, using identical TensorRT-LLM FMHA attention kernels and block-FP8 expert precision as the vLLM baseline.

2. **GPU-initiated, one-sided communication**: Attention GPUs publish FP8 tokens and routing metadata into pre-allocated symmetric NVLink buffers (costing 24 SMs per attention GPU for the fused send/receive kernel), replacing host posts/polls.

3. **Persistent fused MegaMoE kernel**: FFN GPUs run a single kernel per layer fusing NVLink TMA pull (N2M), block-FP8 grouped GEMMs, activation, and push-back (M2N).

4. **Two ping-pong microbatches** ($k=2$): Splitting each attention GPU's batch into two microbatches hides remote MegaMoE execution behind local attention. NVL72's bandwidth covers the round trip with $k=2$, whereas slower interconnects require $k \geq 3$.

5. **Whole-step CUDA graph capture** with a one-step-ahead CPU coordinator: The host prepares step $t+1$ (batch composition, padding, buffer slots, KV-cache frees) while step $t$ runs.

### Per-Layer Overlap Model

The predicted step time is:

$$
\widehat{T}_A(r) = T_{\text{draft}} + 2\sum_{\ell=1}^{L} \max\{T_{a,\ell}, T_{e,\ell}(r)\}\tag{3}
$$

Where:
- $T_{\text{draft}}$: draft-and-accept time outside the layer loop (zero in standard decoding)
- $T_{a,\ell}$: attention stage time in layer $\ell$ (KV-cache reads, norms, QKV/O projections, top-K routing)
- $T_{e,\ell}(r)$: FFN stage time in layer $\ell$ (NVLink dispatch, expert-weight reads, grouped-GEMM compute over routed tokens from $r$ attention nodes)

### Experimental Setup

- **Hardware**: Single GB200 NVL72 rack (18 four-GPU nodes, 72-GPU NVLink domain)
- **Models** (chosen to span three architectural axes):
  - **MiniMax-M2.5**: 10B active of 229B, 256 experts (light-expert, full attention)
  - **Qwen3-235B**: 22B active of 235B, 128 experts (heavy-expert, full attention)
  - **Qwen3.5-397B**: 17B active of 397B, 512 experts (ultra-sparse, hybrid linear-plus-full attention)
- **Workloads**: Fixed prompts of 4K, 8K, or 16K tokens; steady-state closed-loop decode throughput; prefill excluded
- **Baselines**: vLLM wide EP tuned per workload (EP4 through EP64); FastAFD with A:F ratios from 1:1 to 17:1
- **Metrics**: All-GPU speedup $S_{\text{all}}$ (charging all GPUs), attention-GPU speedup $S_{\text{att}} = \frac{r+1}{r}S_{\text{all}}$, and $S_{\text{EP4}}$ against single-node EP4

---

## Empirical Validation / Results

### Standard Decoding Results

**Table 1. Standard decoding (d = 0): context-tuned wide EP erases most of FastAFD's gain over EP4.**

| Model | Ctx | Wide EP | A:F | $S_{EP4}$ | $S_{all}$ | $S_{att}$ |
|-------|-----|---------|-----|----------|-----------|-----------|
| MiniMax-M2.5 | 4K | EP32 | 9:1 | 1.40 | 1.00 | 1.12 |
| MiniMax-M2.5 | 8K | EP32 | 15:1 | 1.52 | 1.02 | 1.09 |
| MiniMax-M2.5 | 16K | EP32 | 17:1† | 1.56 | 1.02 | 1.08▼ |
| Qwen3 | 4K | EP16 | 5:1 | 1.30▼ | 1.06 | 1.28 |
| Qwen3 | 8K | EP16 | 7:1 | 1.41 | 1.10 | 1.26 |
| Qwen3 | 16K | EP32 | 11:1 | 1.42 | 1.17▲ | 1.28 |
| Qwen3.5 | 4K | EP32 | 5:1 | 2.25 | 1.16 | 1.39▲ |
| Qwen3.5 | 8K | EP32 | 5:1 | 2.36▲ | 1.10 | 1.33 |
| Qwen3.5 | 16K | EP32 | 5:1 | 2.11 | 0.99▼ | 1.19 |

*▲/▼: largest/smallest in each column; †: still rising at the 18-node rack limit.*

Key findings:
- FastAFD achieves **1.30–2.36×** over single-node EP4, but context-tuned wide EP compresses this to **0.99–1.17×** all-GPU speedup.
- Attention-GPU speedup $S_{\text{att}}$ reaches **1.08–1.39×** across all workloads.
- Wide EP peaks at interior widths (EP16–EP32), not EP64.

### Speculative Decoding Results

**Table 2. Speculative decoding (d = 3) at least halves FastAFD's best A:F ratio and raises $S_{att}$ on every workload.**

| Model | Ctx | Wide EP | A:F (d=0 → d=3) | $S_{all}$ | $S_{att}$ |
|-------|-----|---------|------------------|-----------|-----------|
| MiniMax-M2.5 | 4K | EP16 | 9:1 → 4:1 | 1.08 | 1.34 |
| MiniMax-M2.5 | 8K | EP16 | 15:1 → 5:1 | 1.02▼ | 1.22 |
| MiniMax-M2.5 | 16K | EP16 | 17:1 → 8:1 | 1.08 | 1.22▼ |
| Qwen3 | 4K | EP8 | 5:1 → 2:1 | 1.13▲ | 1.70▲ |
| Qwen3 | 8K | EP16 | 7:1 → 3:1 | 1.10 | 1.46 |
| Qwen3 | 16K | EP8 | 11:1 → 5:1 | 1.12 | 1.34 |

Key findings:
- Speculative decoding ($q=4$ verified tokens per step) **contracts optimal A:F ratios by more than half**.
- $S_{\text{all}}$ on MiniMax-M2.5 rises from parity to **1.08×**; Qwen3 holds at **1.10–1.13×**.
- $S_{\text{att}}$ rises to **1.22–1.70×** across all workloads.

### Model Validation

- Equation (3) validated against **all 88 measured FastAFD deployments** (standard + speculative decoding).
- **Mean error of 2.9%** across 65 deployments up to each workload's optimal ratio (3.0% standard, 2.6% speculative); never exceeds 7.4%.
- Model-selected ratio retains **95–100%** of measured peak throughput in standard decoding and **97–100%** in speculative decoding.

### Ablation Results

- **Fused MegaMoE** vs. modular DeepEP+DeepGEMM: cuts step latency by **50%** (Qwen3) and **44%** (MiniMax-M2.5) at 7:1 ratio; avoids 12+ GiB extra buffer memory per attention GPU.
- **Microbatch count**: $k=2$ achieves lowest step latency at both 3:1 and 7:1.
- **One-step-ahead coordinator**: disabling it adds 14 ms of synchronous host overhead per step, reducing throughput by **32%**.
- **MoE-boundary vs. CAD**: FastAFD achieves **1.28–1.48×** advantage over Core Attention Disaggregation, which saturates at 2:1–4:1 ratios (vs. FastAFD's 9:1–17:1).

---

## Theoretical and Practical Implications

### Win Condition

The coverage boundary and win condition are given by:

$$
\frac{b_A}{b_V} \cdot \frac{T_V}{T_A(r)} > 1 + \frac{1}{r}, \quad \text{or} \quad \frac{T_V}{T_A(r)} > 1 + \frac{1}{r} \text{ for } b_A \approx b_V\tag{4}
$$

- The FFN stage in layer $\ell$ stays hidden whenever $T_{e,\ell}(r) \leq T_{a,\ell}$ (coverage boundary).
- **General form**: batch gain × overlap gain must outweigh the disaggregation cost.
- **Matched-batch form**: when wide EP already reclaims most KV-cache memory, the overlap gain alone must exceed the disaggregation cost.

### Model-Specific Insights

- **MiniMax-M2.5** (light experts): sits at parity because neither gain repays even a small cost.
- **Qwen3** (heavy experts): batch gain similar to MiniMax-M2.5 but with a real overlap gain, enough to win.
- **Qwen3.5** (hybrid attention): leads on batch alone (1.4–1.6×) despite a slower step, until advantage shrinks to parity at 16K.

### Provisioning Trajectories

- **Full-attention MoEs**: Optimal A:F ratio $r^*$ fans out with context (9:1 → 17:1 for MiniMax-M2.5; 5:1 → 11:1 for Qwen3), consuming only 6–13% of the GPU fleet.
- **Hybrid Qwen3.5**: Maintains flat 5:1 ratio across 4K–16K due to linear-attention layers with fixed-size recurrent state updates.
- **Speculative decoding**: Shifts $r^*$ to 2:1–8:1, requiring 11–33% of the rack for FFN nodes.

### Heterogeneous FFN Placement

Placing the FFN pool on dedicated SRAM accelerators (e.g., Groq LPX paired with Vera Rubin NVL72):

- **Removes the disaggregation cost**: Per-GPU speedups rise from 0.99–1.17× to **1.08–1.39×** (standard) and from 1.02–1.13× to **1.22–1.70×** (speculative).
- **Zero-FFN-cost ceiling** (with SRAM-resident expert weights): speedups reach **1.20–1.76×** (standard) and **1.38–2.48×** (speculative), eliminating both the second pass of dense projections and uncovered FFN tail latency.

---

## Conclusion

The paper answers "when does AFD pay on rack-scale NVLink?" through measurement and modeling:

1. **AFD is a provisioning decision, not an architectural win**: On GB200 NVL72, AFD ranges from parity to 1.17× over per-workload-tuned wide EP when every FFN GPU is counted.

2. **The speedup decomposition** (batch gain × overlap gain × disaggregation cost) explains which models benefit: heavy-expert models (Qwen3) win via overlap gain; hybrid models (Qwen3.5) win via batch gain at shorter contexts; light-expert models (MiniMax-M2.5) sit at parity.

3. **The analytical model** predicts AFD throughput within 3% using only colocated profiles, enabling deployment decisions without running AFD.

4. **Two levers tilt the balance toward AFD**:
   - **Speculative decoding**: Multiplies MoE work, enlarging the overlap gain (adds up to 8% to AFD's advantage on light-expert models).
   - **Off-GPU expert accelerators**: Heterogeneous placement removes the disaggregation cost, raising ideal speedup to 2.48× with speculative decoding.

FastAFD is open source at https://github.com/hao-ai-lab/FastAFD.

---

_Markdown view of https://picx.dev/p/rwQIIl, served by PicX — AI-generated visual whiteboard summaries of research papers._