Back to News

vLLM Introduces Tiered KV Cache Offloading Framework

#vllm#kv-cache#offloading#serving

vLLM's blog details a host-centric framework for scaling KV cache across host memory, filesystems, object stores, and remote peers. This tiered offloading approach aims to reduce recomputation and increase serving capacity.

Coverage timeline

  1. vLLM BlogOr Ozeri, Danny Harnik, Ronen Schaffer, Itay Etelis, Varun Sundar Rabindranath
    Tiered KV Cache Offloading in vLLM

    A host-centric framework for scaling KV cache across host memory, filesystems, object stores, and remote peers — reducing recomputation and increasing serving capacity.

Related stories

  • vLLM Publishes Practical Guide to Disaggregated Serving with GPU-less Frontend

    The vLLM team released a practical guide to disaggregated serving, detailing prefill/decode separation and the new GPU-less frontend. The guide explains the benefits, how to run it end to end in vLLM today, and outlines remaining work. It targets AI/LLM engineers seeking to optimize inference infrastructure.

    1 reports · 1 sources

  • Random KV Cache Eviction Matches Strong Baselines, Boosts Throughput by 43%

    Salesforce AI Research and UIUC propose Random Attention, which randomly evicts KV cache entries in reasoning traces instead of using content-based importance scores. It matches strong baselines across four models and six reasoning tasks, and improves inference throughput by 32%–43% over the strongest baseline TriAttention in vLLM serving tests.

    1 reports · 1 sources

  • vLLM Blog Details Optimizations Behind 2.8x Throughput for Kimi K3 Serving

    A vLLM blog post describes optimizations that achieve 2.8x throughput for serving Kimi K3, covering scheduling, KDA prefix caching, ReplaySSM state recovery, PD disaggregation and state offload, parallelism, MoE, and GPU kernels. The post details the technical approaches behind the performance gains.

    1 reports · 1 sources

  • vLLM integrates HiSparse memory tier for GLM 5.3 KV cache offloading

    The vLLM blog describes integrating HiSparse as a pressure-driven memory tier that works with the Hybrid Memory Allocator and KV offloading. This allows GLM 5.3 requests to continue decoding when their KV cache no longer fits in GPU memory, maintaining high concurrency.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe