Back to News

vLLM Adds Tenstorrent Plugin for Mesh-Optimized LLM Serving

#vllm#tenstorrent#llm-serving#mesh-architecture

vLLM announced an out-of-tree platform plugin enabling LLM serving on Tenstorrent accelerators, with optimizations tailored to the mesh architecture. Key features include phase-based scheduling, single-process data parallelism on Galaxy systems, on-device sampling with host fallback, and asynchronous decode overlap.

Coverage timeline

  1. vLLM BlogTenstorrent Team
    Serving LLMs on Tenstorrent Hardware: Inside the vLLM TT Plugin

    Tenstorrent accelerators join vLLM as an out-of-tree platform plugin, driven by mesh-architecture choices: phase-based scheduling, single-process data parallelism on Galaxy, on-device sampling with host fallback, and async decode overlap.

Related stories

  • vLLM Adds Distribution-Preserving Gumbel-Max Text Watermarking

    The vLLM project has documented its implementation of distribution-preserving Gumbel-max text watermarking. The technique uses efficient GPU kernels for watermark insertion, statistical detection for verification, and supports speculative decoding with safeguards against repeated contexts.

    1 reports · 1 sources

  • vLLM Announces vllm-metal for Apple Silicon with Concurrent Serving

    vLLM announced vllm-metal, bringing its paged, continuously batched serving stack to Apple Silicon. The new offering features flatter time-to-first-token under concurrent agent load, batched multi-token prediction, and automatic M5 prefill acceleration.

    1 reports · 1 sources

  • Researchers Break 1.58-Bit Barrier for Ternary LLMs

    A technical report describes a method that surpasses the previously considered 1.58-bit-per-weight limit for ternary (weights in {-1, 0, 1}) large language models. The approach reportedly achieves lower perplexity and better task performance than standard ternary models at the same model size. This could enable more efficient deployment of LLMs on resource-constrained devices.

    1 reports · 1 sources242 points · 41 comments

  • vLLM Introduces Tiered KV Cache Offloading Framework

    vLLM's blog details a host-centric framework for scaling KV cache across host memory, filesystems, object stores, and remote peers. This tiered offloading approach aims to reduce recomputation and increase serving capacity.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe