vLLM Uses AI Agents to Optimize Kimi K3 Decode on NVIDIA GB300, Hitting 840+ Tokens/s
The vLLM team reports using agent-assisted development to build a unified CuTe DSL decode path for Kimi K3 on NVIDIA GB300, covering attention, MoE, and communication across 8 or 16 GPUs. The optimizations achieve over 840 tokens/s at batch 1 on 16 GPUs with DSpark (synthetic AL=6), 1.5–2× higher decode throughput in several configurations, and up to 1.8× higher P90 interactivity on the AgentX workload. The team notes that agent-optimized inference code is outpacing their manual review and maintenance workflow.
Coverage timeline
vLLM BlogBenjamin Chislett (NVIDIA)
Kimi K3 decode performance on NVIDIA GB300. Coding agents can now help with some of the hardest parts of model optimization: writing complex GPU kernels, performing cross-operator fusions, overlapping computation with communication, and tuning execution across different batch sizes and GPU configurations. For Kimi K3, we used agent-assisted development to build a unified CuTe DSL decode path on the NVIDIA GB300, covering attention, MoE, and communication across 8 or 16 GPUs. The optimizations deliver over **840 tokens/s** at batch 1 on 16 NVIDIA GB300 GPUs with DSpark (synthetic AL=6), **1.5–2×** higher decode throughput in several tested configurations, and up to **1.8×** higher P90 interactivity on the AgentX workload. Agent-optimized inference code is outpacing our review and maintenance workflow. The manual review process of understanding every line of every kernel, mapping out potential issues, and testing every supported configuration, is very time-consuming work. On the other ha
Related stories
vLLM Engineers Detail Scaling Kimi K3 Inference on AMD MI355X to Match NVIDIA B300
vLLM engineers published a blog post describing how they scaled inference for the Kimi K3 model on AMD Instinct MI355X GPUs, achieving per-GPU throughput matching NVIDIA HGX B300 on the AgentX benchmark. The post details the bottleneck analysis and key optimization decisions that enabled this performance parity.
vLLM reports PD serving results for Qwen3.8-2.4T on GB300 NVL72
vLLM's blog reports PD serving performance for the Qwen3.8-2.4T model on a GB300 NVL72 cluster with an 8K/1K workload. In high-throughput scenarios, vLLM achieved 5000 total token throughput per GPU, and 180 generated tokens per user in low-latency scenarios, both on the Pareto frontier. The post provides srt-slurm recipes and details the decision-making process for reproduction and optimization.
vLLM Adds Day-0 Support for NVIDIA Vera Rubin NVL72, Claims 7.8x Throughput over GB200
vLLM announced support for NVIDIA Vera Rubin NVL72, claiming 7.8x throughput over GB200 NVL72. The platform offers daily container builds and day-0 model support for DeepSeek, Moonshot AI, Z.ai, and MiniMax, leveraging Blackwell kernel compatibility and Rubin-tuned kernels via FlashInfer 0.7.0.
Inferact runs Kimi K3 on 16 Google TPU v7 at 709 tokens/s, 57% faster than GB200
Inferact, a startup founded by the original vLLM team, reported that 16 Google TPU v7 chips ran Kimi K3 at 709 tokens per second, 57% faster than 16 Nvidia GB200 GPUs under identical conditions. The speedup comes from a custom megakernel inference kernel and DeepSeek's DSpark speculative decoding framework; the code has been open-sourced.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe