Inferact runs Kimi K3 on 16 Google TPU v7 at 709 tokens/s, 57% faster than GB200
Inferact, a startup founded by the original vLLM team, reported that 16 Google TPU v7 chips ran Kimi K3 at 709 tokens per second, 57% faster than 16 Nvidia GB200 GPUs under identical conditions. The speedup comes from a custom megakernel inference kernel and DeepSeek's DSpark speculative decoding framework; the code has been open-sourced.
Coverage timeline
量子位量子位
谷歌TPU跑Kimi比英伟达GPU快57%！用的还是DeepSeek推理框架 听雨 2026-09-26 15:12:05 来源： 量子位 vLLM人马创业公司团队出品 克雷西 发自 凹非寺 量子位 | 公众号QbitAI 16块谷歌TPU v7跑Kimi K3，每秒跑出了709个Token，比老黄的GB200快了57%。 做出这个成绩的，既不是造Kimi的月之暗面，也不是造TPU的谷歌，是一家叫Inferact的推理创业公司。 Inferact创始团队就是vLLM的原班人马，今年拿了a16z领投的1.5亿美元种子轮，估值8亿美元。 他们给TPU写了一种叫megakernel的推理内核，把模型推理时原本要调度的几百个小程序焊成一个大程序。 配合TPU独特的片上内存架构，megakernel把内存带宽利用率逼到了硬件理论峰值附近。 配上DeepSeek提出的DSpark加速框架，K3在TPU上跑出了前面提到的每秒709 token的成绩。 这套代码，现在已经开源。 同样16块芯片，TPU快57% Inferact把TPU和英伟达GPU放在完全相同的条件下跑了一轮benchmark，TPU赢了。 测试是用16块TPU v7 Ironwood对阵16块英伟达GB200，两边都跑Kimi K3，都用vLLM推理引擎，唯一的变量是芯片和底层的kernel实现。 最终，TPU跑出的成绩是每秒709个token，GB200跑出每秒452个，TPU的速度快了57%。 这个速度里有DSpark推测解码的功劳。 DSpark是由DeepSeek提出的推理加速框架，其原理是让一个小模型先快速猜出一串候选token，然后大模型再对这串候选token做批量验证，猜对的直接跳过，猜错的回退重来。 Inferact在TPU上跑通了这套流程，acceptance length达到了6，也就是每轮猜出的token里，平均有6个能直接通过大模型的验证，单步decode的耗时大约在8.5毫秒。 关掉推测解码看裸速，差距同样拉得开。 在batch size为1的情况下，TPU每秒能跑249个token，GB200只有127个，差距接近一倍。 把batch size放大到8，TPU达到865个token每秒，GB200只有636个。 另外在Qwen上，两种芯片的差距更大。 Inferact用4块TPU v7
Related stories
DeepSeek open-sources DeepInfer inference stack, reports 82% gross margin
DeepSeek released its internal inference framework DeepInfer under the Apache 2.0 license, along with the token codec component deepseek-recipe. The framework, which supports DeepSeek-V4.1-Flash and targets Nvidia SM100 GPUs and Huawei Ascend 950 NPUs, reportedly achieves a throughput of 155 trillion tokens per day per cluster, with API revenue of $38 per GPU-hour at off-peak pricing, implying a gross margin of 82% after GPU rental costs. The release sparked discussions about the profitability of AI inference services.
3 reports · 3 sources2335 points · 890 comments
vLLM Uses AI Agents to Optimize Kimi K3 Decode on NVIDIA GB300, Hitting 840+ Tokens/s
The vLLM team reports using agent-assisted development to build a unified CuTe DSL decode path for Kimi K3 on NVIDIA GB300, covering attention, MoE, and communication across 8 or 16 GPUs. The optimizations achieve over 840 tokens/s at batch 1 on 16 GPUs with DSpark (synthetic AL=6), 1.5–2× higher decode throughput in several configurations, and up to 1.8× higher P90 interactivity on the AgentX workload. The team notes that agent-optimized inference code is outpacing their manual review and maintenance workflow.
1 reports · 1 sources
vLLM Engineers Detail Scaling Kimi K3 Inference on AMD MI355X to Match NVIDIA B300
vLLM engineers published a blog post describing how they scaled inference for the Kimi K3 model on AMD Instinct MI355X GPUs, achieving per-GPU throughput matching NVIDIA HGX B300 on the AgentX benchmark. The post details the bottleneck analysis and key optimization decisions that enabled this performance parity.
Colibrì: C-based inference framework runs 744B-param GLM-5.2 on laptops via SSD offloading, no GPU needed
Colibrì, an open-source C-based hierarchical inference framework, has gained 32k GitHub stars for enabling large MoE models to run on laptops with limited RAM by offloading inactive experts to SSD, without requiring a GPU. It supports models from GLM-5.2/5.3, DeepSeek V4 Flash, Qwen, to 975B Inkling and 2.8T Kimi K3, with RAM requirements as low as 16GB for int4-processed GLM-5.2.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe