Summary (Overview)
- NVLinkOneSided is a one-sided expert-parallel (EP) all-to-all communication method for Mixture-of-Experts (MoE) inference, designed for GPUs within a single NVLink domain.
- It eliminates metadata exchange by having each source rank push tokens directly into rank-major buffers on target GPUs, which grouped GEMMs read in place via slot indices.
- Two synchronization protocols are provided: FENCE (high-throughput, for large batches) and SENTINEL (low-latency, for small batches/decoding).
- On Blackwell GPUs, NVLinkOneSided completes dispatch and combine 1.2–3.0× faster than DeepEP, with the largest speedups at small batch sizes; the resulting MoE layer is up to 2.4× faster than DeepEP+DeepGEMM and outperforms the fused DeepGEMM MegaMoE at up to 128 tokens per GPU.
- The implementation is modular, quantization-agnostic, and integrated into FlashInfer, vLLM, SGLang, and TensorRT-LLM.
Introduction and Theoretical Foundation
Mixture-of-Experts (MoE) models activate only a few of their many experts per token (Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al., 2022). They are deployed with data-parallel (DP) attention and expert-parallel (EP) MoE layers, a strategy popularized by DeepSeek-V3 (DeepSeek-AI, 2024). Each MoE layer requires two all-to-all communications:
- Dispatch: sends each token to ranks that own its experts.
- Combine: returns expert outputs to the source rank.
Key challenge: Inference processes few tokens per GPU at a time. Decoding is bounded to a few hundred tokens per rank to meet service-level objectives. At small batches, fixed costs dominate latency (synchronization between producers/consumers on different GPUs), not data movement. Measured round trips of dispatch and combine take tens of microseconds even when bytes would transfer in under a microsecond at theoretical NVLink bandwidth.
NVLink domain advantages: Within one NVLink domain, symmetric memory allows a rank to load from and store to a peer's memory directly. Blackwell GPUs have 18 fifth-generation NVLink links (100 GB/s each, 900 GB/s per direction). A GB200/GB300 NVL72 rack forms one 72-GPU NVLink domain.
Memory ordering guarantees: GPU memory is weakly ordered. A consumer that sees a producer's write is guaranteed to see earlier writes only if the producer ordered them with a system-scope release and the consumer reads with acquire semantics. Each aligned 32-bit word stored to peer memory is observed whole or not at all.
Methodology
1. One-Sided Communication with Rank-Major Layout
Each source rank routes its own tokens, so it knows locally which target ranks each token must reach. The source rank pushes each token directly into the dispatch workspace of every target rank (in symmetric memory), recording the target rank and slot of each selection. No metadata (token counts or expert ids) travels ahead of the data.
2. Rank-Major Layout
Each dispatch workspace has one region per source rank, which only that source rank writes. A region has slots (the largest number of tokens across ranks). This layout offers:
- De-duplication: A token is sent at most times instead of times as in per-expert transfers.
- Smaller workspaces: For DeepSeek-V4.1-Flash on eight GPUs with up to 4096 tokens/rank, the dispatch workspace takes 160 MiB (MXFP8) and combine workspace 320 MiB (BF16), versus 1020 MiB for DeepEP V2.5's expert-major GEMM input.
3. MoE Compute on Rank-Major Layout
The MoE layer runs: dispatch → indexing kernel → grouped GEMMs (FC1, FC2) → finalize → combine. Tokens stay at their slots; grouping by expert is expressed as slot indices. FC1 uses TMA gather4 instructions to load 128-byte slices of non-contiguous slots.
4. Synchronization Protocols
FENCE Protocol (Algorithm 1):
- Each thread block handles one token; blocks atomically increment slot counters per target rank.
- The last block to finish executes a system-scope release fence, publishes slot counts, and writes flags to all peers.
- Consumers spin until all peer flags arrive, then execute acquire fences.
SENTINEL Protocol (Algorithm 2):
- Source ranks push tokens into landing slots initialized with a sentinel value (e.g., 0xFFFFFFFF for BF16, which the producer avoids by storing 0x7FFF NaNs).
- Target ranks poll slots; a slot whose words all differ from the sentinel is complete—no fence needed.
- Token of source rank always lands in slot of region ; every source rank writes expert ids (invalid −1 where no token goes) to every rank.
5. Implementation Details
- Kernels use 16-byte vectorized loads/stores (data-level parallelism) and unrolled loops for remote stores to all target ranks (instruction-level parallelism).
- The whole MoE layer is captured in a CUDA graph with Programmatic Dependent Launch (PDL) for kernel overlap.
- Dispatch supports all quantization formats (BF16, FP8, MXFP8, NVFP4); FENCE can quantize partial results to FP8 before combine.
Empirical Validation / Results
Setup
- Tests on eight NVLink-connected B300 GPUs and a GB200 NVL72 rack.
- Models: DeepSeek-V4.1-Flash (hidden size 5120, 384 experts, top-6) and Kimi-K3 (hidden size 3584, 896 experts, top-16).
- Baselines: DeepEP V2.5, DeepGEMM MegaMoE, DeepEP + DeepGEMM.
All-to-All Performance
| Metric | NVLinkOneSided | DeepEP |
|---|---|---|
| 1 token/rank (DeepSeek-V4.1-Flash) | 17.4 µs (SENTINEL), 27.6 µs (FENCE) | 47.9 µs |
| 1–128 tokens/rank speedup | 1.75–3.00× | — |
| 256+ tokens/rank speedup | 1.17–1.63× | — |
| 4096 tokens/rank bandwidth | 68–71% of 900 GB/s | — |
Protocol crossover: SENTINEL is faster at decoding batches; FENCE at large ones, with crossover at 256 tokens/rank. At 1 token/rank, SENTINEL's dispatch skips FENCE's serial chain and takes half as long.
Payload precision: At 4096 tokens/rank, NVFP4 makes dispatch 3.0× faster than BF16 (byte ratio 3.5×); MXFP8 is 1.8× faster (byte ratio 1.9×). FP8 combine halves bytes but is only 1.4× faster.
MoE Layer Performance (Table 4, DeepSeek-V4.1-Flash)
| Tokens/rank | NVLinkOneSided (best) | DE+DG | MegaMoE |
|---|---|---|---|
| 1 | 36.7 µs (SENTINEL) | 74.7 µs | 49.8 µs |
| 128 | 183.3 µs (SENTINEL) | 235.9 µs | 188.7 µs |
| 256 | 206.5 µs | 251.2 µs | 194.1 µs |
| 4096 | 1185.2 µs | 1290.2 µs | 766.1 µs |
NVLinkOneSided outperforms fused MegaMoE up to 128 tokens/rank (by up to 30%) and beats DE+DG at every batch (up to 2.4×).
Skewed Routing (Table 5)
Under Zipf routing with α = 1.25 at 1024 tokens/rank, the busiest rank computes up to 1.9× more token–expert pairs than under perfect routing. NVLinkOneSided's communication grows less (a token reaches a target rank once regardless of how many local experts it selects): dispatch/combine take 160/135 µs vs. 267/400 µs for DE+DG.
Large EP Scaling (Table 6)
On 64 GPUs at 128 tokens/rank, the layer takes 103 µs vs. 182 µs on 8 GPUs. SENTINEL's round trip stays at 16.6–17.8 µs from 8 to 64 GPUs at 1 token/rank (no per-peer handshake), while FENCE's grows from 43.0 to 47.6 µs.
Theoretical and Practical Implications
Advantages:
- Faster than DeepEP at every batch size, with the largest gains where fixed costs dominate (decoding).
- Adaptable: quantization-agnostic, decoupled from MoE compute backend, modular interface.
- Memory-efficient: rank-major layout deduplicates tokens and reduces workspace sizes.
- Fault tolerance: every wait has a timeout and traps with an error message.
Limitations:
- NVLink domain constraint: Both protocols rely on direct loads/stores to peer memory and 32-bit words that arrive whole—RDMA does not offer these guarantees.
- Memory growth with EP size: Dispatch workspace holds one region per rank.
Comparison with MegaMoE: MegaMoE's strength is overlap (communication hidden behind Tensor Core work in a fused kernel), but its weaknesses are the metadata exchange and coupling of layout/transfer granularity to GEMM tiles. NVLinkOneSided's separate-kernel design wins at small batches where there is little computation to hide communication behind.
Conclusion
NVLinkOneSided implements dispatch and combine as one-sided communication within an NVLink domain:
- Source ranks push tokens into rank-major workspaces using only local routing information.
- MoE compute reads them through slot indices.
- Two protocols (SENTINEL for small batches, FENCE for large ones) synchronize the exchange.
Key results: Faster than DeepEP at every batch; the resulting MoE layer outperforms DeepGEMM MegaMoE at decoding batches. Available in FlashInfer and integrated into vLLM, SGLang, and TensorRT-LLM.
Future work: Carry the one-sided design into a fused operator (like MegaMoE) that pushes tokens without metadata into a rank-major layout and detects arrival from the data, enabling communication/computation overlap at large batches.
Related papers
- Late Entry in Looped Language Models
Late entry into deep context caches outperforms early exit in looped language models, yielding up to 8.4x exact and 12.5x lossy decoding speedups.
- Pro$^2$ Agent: Scientific Problem Probing for Autonomous AI R&D
Pro²Agent's explicit Problem State—tracking diagnosed bottlenecks, candidate solutions, and evidence—achieves a record 76% medal rate on MLE-Bench, outperforming code-centric agents.
- RIPPLE: Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit for Long-Horizon Agentic Coding
RIPPLE internalizes prefix-only recovery guidance into a single policy via stage-specific credit, improving coding agents by 4.8 points across four benchmarks without deployment-time critics.