vLLM Blog Details Optimizations Behind 2.8x Throughput for Kimi K3 Serving
A vLLM blog post describes optimizations that achieve 2.8x throughput for serving Kimi K3, covering scheduling, KDA prefix caching, ReplaySSM state recovery, PD disaggregation and state offload, parallelism, MoE, and GPU kernels. The post details the technical approaches behind the performance gains.
Coverage timeline
vLLM BlogWentao Ye, Canlin Guo, Yongye Zhu, Jiangyun Zhu, Ziming Huang, Wei Zhao, Michael Goin, Jie Li
Kimi K3 serving optimizations across scheduling, KDA prefix caching, ReplaySSM state recovery, PD disaggregation and state offload, parallelism, MoE, and GPU kernels.
Related stories
vLLM Boosts DeepSeek-V4.1-Flash: 1.9x Low-Concurrency Speed, 5x Agentic Throughput
According to a vLLM blog post dated October 7, 2026, within three weeks of DeepSeek-V4.1-Flash's release, vLLM optimizations made the model 1.9x faster at low concurrency and increased its throughput 5x on the SemiAnalysis AgentX benchmark. The improvements are attributed to SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions.
vLLM Publishes Practical Guide to Disaggregated Serving with GPU-less Frontend
The vLLM team released a practical guide to disaggregated serving, detailing prefill/decode separation and the new GPU-less frontend. The guide explains the benefits, how to run it end to end in vLLM today, and outlines remaining work. It targets AI/LLM engineers seeking to optimize inference infrastructure.
Xiaomi MiMo Team Details Tool-Call Repetition Fix in MiMo-V2.6
Xiaomi's MiMo team published a technical blog post on September 27, 2026, diagnosing and mitigating tool-call repetition in their MiMo-V2.6 model. The post shares lessons from scaling reinforcement learning, highlighting a 'reward blind spot' when optimizing for correctness.
1 reports · 1 sources
Inferact runs Kimi K3 on 16 Google TPU v7 at 709 tokens/s, 57% faster than GB200
Inferact, a startup founded by the original vLLM team, reported that 16 Google TPU v7 chips ran Kimi K3 at 709 tokens per second, 57% faster than 16 Nvidia GB200 GPUs under identical conditions. The speedup comes from a custom megakernel inference kernel and DeepSeek's DSpark speculative decoding framework; the code has been open-sourced.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe