Back to News

vLLM Engineers Detail Scaling Kimi K3 Inference on AMD MI355X to Match NVIDIA B300

#vllm#amd#inference#kimi-k3

vLLM engineers published a blog post describing how they scaled inference for the Kimi K3 model on AMD Instinct MI355X GPUs, achieving per-GPU throughput matching NVIDIA HGX B300 on the AgentX benchmark. The post details the bottleneck analysis and key optimization decisions that enabled this performance parity.

Coverage timeline

  1. vLLM BlogAMD and Embedded LLM
    Scaling Kimi K3 on AMD Instinct MI355X: Matching NVIDIA HGX B300 Throughput per GPU on AgentX

    Inside the Kimi K3 optimization effort: understanding the bottlenecks and the decisions that made inference faster

Related stories

  • vLLM Adds Day-0 Support for NVIDIA Vera Rubin NVL72, Claims 7.8x Throughput over GB200

    vLLM announced support for NVIDIA Vera Rubin NVL72, claiming 7.8x throughput over GB200 NVL72. The platform offers daily container builds and day-0 model support for DeepSeek, Moonshot AI, Z.ai, and MiniMax, leveraging Blackwell kernel compatibility and Rubin-tuned kernels via FlashInfer 0.7.0.

    1 reports · 1 sources

  • NVIDIA launches Nemotron 3.5 Super VL, 120B open model; vLLM adds Day 0 support

    NVIDIA has launched Nemotron 3.5 Super VL, a 120B-parameter open model for enterprise agents, claiming it outperforms frontier models on specialized benchmarks. vLLM announced Day 0 support for the model, enabling multimodal agent workflows with image, document, and video understanding, OpenAI-compatible serving, and multi-token prediction.

    2 reports · 2 sources

  • Inferact runs Kimi K3 on 16 Google TPU v7 at 709 tokens/s, 57% faster than GB200

    Inferact, a startup founded by the original vLLM team, reported that 16 Google TPU v7 chips ran Kimi K3 at 709 tokens per second, 57% faster than 16 Nvidia GB200 GPUs under identical conditions. The speedup comes from a custom megakernel inference kernel and DeepSeek's DSpark speculative decoding framework; the code has been open-sourced.

    1 reports · 1 sources

  • vLLM reports PD serving results for Qwen3.8-2.4T on GB300 NVL72

    vLLM's blog reports PD serving performance for the Qwen3.8-2.4T model on a GB300 NVL72 cluster with an 8K/1K workload. In high-throughput scenarios, vLLM achieved 5000 total token throughput per GPU, and 180 generated tokens per user in low-latency scenarios, both on the Pareto frontier. The post provides srt-slurm recipes and details the decision-making process for reproduction and optimization.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe