Back to News

vLLM reports PD serving results for Qwen3.8-2.4T on GB300 NVL72

#vllm#pd-serving#qwen#performance

vLLM's blog reports PD serving performance for the Qwen3.8-2.4T model on a GB300 NVL72 cluster with an 8K/1K workload. In high-throughput scenarios, vLLM achieved 5000 total token throughput per GPU, and 180 generated tokens per user in low-latency scenarios, both on the Pareto frontier. The post provides srt-slurm recipes and details the decision-making process for reproduction and optimization.

Coverage timeline

  1. vLLM BlogvLLM Team
    PD Serving of Qwen3.8-2.4T

    ## TL;DR In this blog post we present our latest performance results on PD serving of the Qwen3.8-2.4T model using vLLM on a GB300 NVL72 cluster on 8K/1K workload. In high throughput scenario vLLM achieved **5000** total token throughput per GPU, and **180** generated tokens per user in the low latency scenario — both presented on pareto frontier below. In this blog post we explain in detail how we achieved these results, and give precise srt-slurm recipes to allow anyone to verify and reproduce them using local serving. But what is more important — we describe our decision making process we used to create these recipes. This is even more important than the actual performance results, since it allows you to squeeze performance out of PD serving on vLLM for any model you desire. ## Introduction Recently, we presented our Qwen3.5 PD serving results and showed that vLLM could reach 25K Total TPS/GPU. Straight after that, a new frontier Qwen-family model was released: Qwen3.8-2.4T. In this

Related stories

  • vLLM Uses AI Agents to Optimize Kimi K3 Decode on NVIDIA GB300, Hitting 840+ Tokens/s

    The vLLM team reports using agent-assisted development to build a unified CuTe DSL decode path for Kimi K3 on NVIDIA GB300, covering attention, MoE, and communication across 8 or 16 GPUs. The optimizations achieve over 840 tokens/s at batch 1 on 16 GPUs with DSpark (synthetic AL=6), 1.5–2× higher decode throughput in several configurations, and up to 1.8× higher P90 interactivity on the AgentX workload. The team notes that agent-optimized inference code is outpacing their manual review and maintenance workflow.

    1 reports · 1 sources

  • vLLM Engineers Detail Scaling Kimi K3 Inference on AMD MI355X to Match NVIDIA B300

    vLLM engineers published a blog post describing how they scaled inference for the Kimi K3 model on AMD Instinct MI355X GPUs, achieving per-GPU throughput matching NVIDIA HGX B300 on the AgentX benchmark. The post details the bottleneck analysis and key optimization decisions that enabled this performance parity.

    1 reports · 1 sources

  • vLLM Cuts TTFT Up to 3.3x with Bi-Directional KV Transfer; Push-Based NIXL Connector Improves Disaggregated Inference

    vLLM introduced bi-directional KV transfer that lets prefill nodes reuse previously computed KV from decode nodes on later conversation turns, cutting redundant prefill recompute and reducing TTFT by up to 3.3x on long multi-turn prompts. Separately, a push-based NIXL connector for disaggregated inference transfers KV over RDMA directly from prefill to decode memory, removing router relay, decode-side allocation, and transfer setup from the TTFT critical path.

    2 reports · 1 sources

  • vLLM Adds Day-0 Support for NVIDIA Vera Rubin NVL72, Claims 7.8x Throughput over GB200

    vLLM announced support for NVIDIA Vera Rubin NVL72, claiming 7.8x throughput over GB200 NVL72. The platform offers daily container builds and day-0 model support for DeepSeek, Moonshot AI, Z.ai, and MiniMax, leveraging Blackwell kernel compatibility and Rubin-tuned kernels via FlashInfer 0.7.0.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe