Back to News

Ant Group and vLLM Community Open-Source Humming Kernel Library for Quantized Inference

#quantization#kernel#inference#open-source

Ant Group and the vLLM community have open-sourced Humming, a high-performance kernel library for quantized inference. It supports NVIDIA GPUs from Turing onward and CUDA-compatible devices like T-Head's Zhenwu 810E, with a unified architecture for diverse weight and activation formats. Humming reportedly outperforms existing implementations from vLLM Marlin, FlashInfer, and DeepGEMM on many workloads.

Coverage timeline

  1. vLLM BlogAnt Group and the vLLM Community
    Announcing Humming: Fast, Flexible Kernels for Quantized Inference

    Ant Group and the vLLM community are open-sourcing Humming, a general-purpose, high-performance kernel library for quantized inference. Humming offers the following features: * **Broad device support**: Supports NVIDIA GPU architectures from Turing onward, along with CUDA-compatible devices such as T-Head's Zhenwu 810E. * **Extensive quantization support**: A unified architecture supports diverse combinations of weight and activation formats for dense and MoE workloads, covering a wide range of integer and floating-point types, bit widths, and scaling schemes. * **Strong execution performance**: Significantly outperforms existing implementations from vLLM Marlin, FlashInfer, and DeepGEMM across many representative workloads. Repository: github.com/vllm-project/humming ## Background As models and inference workloads grow, memory capacity, memory bandwidth, and compute costs increasingly constrain deployment. This is especially apparent in MoE models: each token activates only a subset o

Related stories

  • Prime Intellect launches Prime Inference serving platform for open models

    Prime Intellect announced the launch of Prime Inference, a serving platform for frontier open-source models offering resilient serverless and reserved serving across multiple datacenters. The platform's production architecture includes prefill/decode optimization, NVFP4 KV compression, and NIXL transfer, as demonstrated in its GLM-5.3 deployment on GB200 NVL72.

    2 reports · 1 sources

  • vLLM Cuts TTFT Up to 3.3x with Bi-Directional KV Transfer; Push-Based NIXL Connector Improves Disaggregated Inference

    vLLM introduced bi-directional KV transfer that lets prefill nodes reuse previously computed KV from decode nodes on later conversation turns, cutting redundant prefill recompute and reducing TTFT by up to 3.3x on long multi-turn prompts. Separately, a push-based NIXL connector for disaggregated inference transfers KV over RDMA directly from prefill to decode memory, removing router relay, decode-side allocation, and transfer setup from the TTFT critical path.

    2 reports · 1 sources

  • Ant Group open-sources Ling-3.1-flash, a 560B-parameter MoE model with 1M context

    Ant Group's Ling-3.1-flash, a 560B-parameter mixture-of-experts model with 25B active parameters and a 1M-token context window, was released open-source on October 15. The model targets long-horizon complex task execution, using a hybrid linear attention design with KDA layers and Gated MLA to manage computational costs on long sequences.

    1 reports · 1 sources

  • LoopVL open-source VLM shows data efficiency with 260x less training data

    Researchers from Renmin University, Baidu, and others released LoopVL, an open-source vision-language model based on the Loop Transformer. Despite using only 0.14T training tokens versus about 36T for Qwen3.5-2B and InternVL3.5-2B, LoopVL matches or exceeds them on benchmarks like LogicVista, MMMU-Pro, and MathVision, demonstrating high data efficiency. The team also observed cross-loop visual attention reorganization, termed 'Visual Aha Moment'.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe