vLLM Migrates CUDA Build to LibTorch Stable ABI for PyTorch Upgrades
The vLLM team detailed its migration of the CUDA build and compiled dependencies to the LibTorch Stable ABI. This change allows compatible PyTorch upgrades without requiring a rebuild of vLLM, simplifying deployment and maintenance.
Coverage timeline
vLLM BlogvLLM Team, Red Hat PyTorch Team, and Meta PyTorch Team
How vLLM migrated its CUDA build and compiled dependencies to the LibTorch Stable ABI, enabling compatible PyTorch upgrades without rebuilding vLLM.
Related stories
Google launches TorchTPU, vLLM adds TPU backend
Google announced TorchTPU, an open-source version of PyTorch that lets AI applications run on TPUs and Google Cloud via simple configuration changes without rewriting code. Concurrently, vLLM introduced vllm-torchtpu, a new backend for Google Cloud TPUs built on TorchTPU, featuring Pallas kernels, multi-host parallelism, KV cache offloading, and disaggregated serving. These moves, following Amazon's similar tool, challenge Nvidia's dominance in AI development.
3 reports · 3 sources
vLLM Introduces Online-Fitted Acceptance Estimator for Adaptive Verification
The vLLM blog describes an online-fitted acceptance estimator that enables Adaptive Verification for draft-model speculators without requiring a confidence head. The estimator predicts the target's acceptance probability from the drafter's own logits and refits continuously during serving, allowing every draft-model speculator to use the technique.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe