vLLM Blog: FlyDSL MonoKernels Boost Agent Decode on AMD Instinct GPUs
The vLLM blog describes how FlyDSL MonoKernels enable high-interactivity decode for agents on AMD Instinct GPUs. The approach uses model-specific fusion, data-ready handoffs, and specialized-decode integration to keep agents responsive.
Coverage timeline
vLLM BlogFelix Li, Lingpeng Jin, Hattie Wu, Carlus Huang, Fangzhou Ai, Zaidy Ali, Amanzhol Salykov, Hyukjoon Lee, Ziming Huang, Jun Chen, Yifeng Huang, Bin Ding, Xiaobing Zhang, Yi Gan, Vinayak Gokhale, Hongxia Yang, Doug Lehr, Richard Li, Andy Luo, Peng Sun, and Emad Barsoum
How FlyDSL MonoKernels keep agents responsive with high-interactivity decode in vLLM on AMD Instinct GPUs, using model-specific fusion, data-ready handoffs, and specialized-decode integration.
Related stories
vLLM Adds Day-0 Support for Flint Sparse Model Checkpoints
vLLM announced day-0 support for Flint checkpoints, a sparse model with 40 billion total parameters and 2.5 billion active parameters. On a single GPU, vLLM decodes a single request at approximately 500 tokens per second using these checkpoints.
1 reports · 1 sources
GLM-5.3 served at 607.8 tok/s across two Nvidia DGX stations with vLLM
A hands-on engineering write-up reports serving GLM-5.3 at 607.8 tokens per second across two Nvidia DGX stations using the vLLM inference engine. The post details the setup and performance findings, offering practical insights for high-throughput LLM deployment.
1 reports · 1 sources149 points · 71 comments
FlashStates Hybrid Cache Management Boosts Prefill Throughput Up to 3.14x vs vLLM
Researchers from Together AI, UIUC, and UT Austin propose FlashStates, a hybrid state cache management system for mixed-architecture LLMs that combine full attention with linear layers. On NVIDIA B200 GPUs, replaying real Claude Code traces, FlashStates achieves up to 3.14x prefill throughput over vLLM and reduces p90 time-to-first-token (TTFT) by up to 88.4%, with benefits persisting when integrated with a 512GiB Mooncake CPU cache.
1 reports · 1 sources

vLLM Serves DeepSeek-V4.1-Flash at 954K Tokens/s per GPU on GB300 NVL72
vLLM's blog reports that using P/D disaggregation and Wide-EP on a GB300 NVL72 rack, it serves DeepSeek-V4.1-Flash on SemiAnalysis AgentX at 954K tokens/s per GPU, matching DeepInfer's performance. This result is 7.9x the aggregated single-node result, demonstrating significant scaling.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe