Back to News

vLLM Achieves Near-Zero-Downtime Scaling with Asynchronous Preparation and NIXL EP

#vllm#elastic-expert-parallelism#nixl#scaling

The vLLM team published a blog post describing how asynchronous preparation and NIXL EP enable near-zero-downtime scaling for Elastic Expert Parallelism (EP). The approach reduces downtime during dynamic scaling of expert-parallel models, improving operational efficiency for large-scale inference.

Coverage timeline

  1. vLLM BlogItay Alroy (NVIDIA)
    Achieving near-zero-downtime scaling in Elastic Expert Parallelism

    How asynchronous preparation and NIXL EP enabled near-zero-downtime Elastic EP scaling in vLLM.

Related stories

  • vLLM Introduces Online-Fitted Acceptance Estimator for Adaptive Verification

    The vLLM blog describes an online-fitted acceptance estimator that enables Adaptive Verification for draft-model speculators without requiring a confidence head. The estimator predicts the target's acceptance probability from the drafter's own logits and refits continuously during serving, allowing every draft-model speculator to use the technique.

    1 reports · 1 sources

  • vLLM Cuts TTFT Up to 3.3x with Bi-Directional KV Transfer; Push-Based NIXL Connector Improves Disaggregated Inference

    vLLM introduced bi-directional KV transfer that lets prefill nodes reuse previously computed KV from decode nodes on later conversation turns, cutting redundant prefill recompute and reducing TTFT by up to 3.3x on long multi-turn prompts. Separately, a push-based NIXL connector for disaggregated inference transfers KV over RDMA directly from prefill to decode memory, removing router relay, decode-side allocation, and transfer setup from the TTFT critical path.

    2 reports · 1 sources

  • RAVEL: Training-Free Method Improves Vision Encoder Evaluation for Multimodal LLMs

    Researchers proposed RAVEL, a training-free evaluation method with two improvements—stabilized PCA whitening and patch-level similarity scoring—to predict how well vision encoders perform in multimodal LLMs. Experiments on 210 multimodal LLMs showed existing cross-modal methods degrade as candidate pools grow, while RAVEL achieved an average Spearman correlation of 0.85, outperforming prior methods by up to 101.1%.

    1 reports · 1 sources

  • vLLM Publishes Practical Guide to Disaggregated Serving with GPU-less Frontend

    The vLLM team released a practical guide to disaggregated serving, detailing prefill/decode separation and the new GPU-less frontend. The guide explains the benefits, how to run it end to end in vLLM today, and outlines remaining work. It targets AI/LLM engineers seeking to optimize inference infrastructure.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe