Back to News

vLLM Cuts TTFT Up to 3.3x with Bi-Directional KV Transfer; Push-Based NIXL Connector Improves Disaggregated Inference

#vllm#kv-transfer#ttft#disaggregated-inference

vLLM introduced bi-directional KV transfer that lets prefill nodes reuse previously computed KV from decode nodes on later conversation turns, cutting redundant prefill recompute and reducing TTFT by up to 3.3x on long multi-turn prompts. Separately, a push-based NIXL connector for disaggregated inference transfers KV over RDMA directly from prefill to decode memory, removing router relay, decode-side allocation, and transfer setup from the TTFT critical path.

Coverage timeline

  1. vLLM BlogSunita Nadampalli (AWS)
    Improve TTFT for Multi-Turn Agentic Workloads with Bi-Directional KV Transfer in vLLM

    Bi-directional KV transfer lets vLLM prefill nodes reuse previously computed KV from decode nodes on later conversation turns, cutting redundant prefill recompute and reducing TTFT by up to 3.3x on long multi-turn prompts.

  2. vLLM BlogSunita Nadampalli (AWS), Nicolò Lucchesi (Mistral)
    Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM

    A push-based NIXL connector for vLLM disaggregated serving: the prefill instance writes KV directly into decode memory over RDMA, taking the router relay, decode-side allocation, and transfer setup off the TTFT critical path.