Back to News

vLLM and Novita AI Announce Chord, an INT4 MoE Backend for Kimi K2.x

#vllm#int4#moe#kimi-k2

vLLM and Novita AI announced Chord, a faster INT4 MoE backend for Kimi K2.x models, claiming up to 1.3x speedup on H200 and 2.15x on untuned B300. The backend supports uint4, group-32, BF16 scales and the compressed-tensors pack-quantized INT4 group-32 checkpoint format; unsupported quantization schemes fail at load. The distribution provides both chord and humming module roots, with grouped integration with vLLM's Humming backend still in progress.

Coverage timeline

  1. vLLM BlogNovita AI and the vLLM Team
    vLLM x Novita AI: Chord, Faster INT4 MoE for Kimi K2.x. Up to 1.3x on H200, 2.15x on Untuned B300

    # Do not co-install inclusionAI/humming: Chord intentionally owns that import name (for indexed path, grouped integration is WIP). vllm serve <kimi-k2.x-int4-model> --quantization humming # or select moe_backend="humming" in the vLLM configuration ``` The distribution provides both `chord` and `humming` module roots. vLLM's lazy facade resolves `humming.{dtypes,config,layer,schema,utils.weight}`; the default indexed path can use this existing integration without a Chord-specific framework patch on branches with the WNA16 group-scale support noted below. The shipped schema supports uint4, group-32, BF16 scales and the compressed-tensors pack-quantized INT4 group-32 checkpoint format used by Kimi K2.x; unsupported quantization schemes fail at load instead of silently selecting a wrong kernel. **Grouped integration with vLLM's Humming backend is WIP.** The standalone grouped operator API is shown below. TP8 remains the indexed `h200_tp8` profile because one TP8 weight must serve both phas

Related stories

  • vLLM Adds Day-0 Support for NVIDIA Vera Rubin NVL72, Claims 7.8x Throughput over GB200

    vLLM announced support for NVIDIA Vera Rubin NVL72, claiming 7.8x throughput over GB200 NVL72. The platform offers daily container builds and day-0 model support for DeepSeek, Moonshot AI, Z.ai, and MiniMax, leveraging Blackwell kernel compatibility and Rubin-tuned kernels via FlashInfer 0.7.0.

    1 reports · 1 sources

  • vLLM-Omni and FastVideo Optimize MiniMax H3 Serving, Generate MP4 in 8.7s on Eight B300 GPUs

    A technical post from vLLM-Omni and FastVideo describes system-level optimizations for serving MiniMax H3, a multimodal model that generates video with synchronized audio. The optimizations span the full resident pipeline, including attention and communication, fused DiT operators, parallel VAE decoding, compact output transfer, and parallel MP4 construction. FastVideo's FastH3 reduces 49 DiT forward passes to 4, achieving real-time full MP4 generation in 8.678–8.710 seconds on eight B300 GPUs for a 10.125-second clip.

    2 reports · 2 sources

  • vLLM Blog Details Optimizations Behind 2.8x Throughput for Kimi K3 Serving

    A vLLM blog post describes optimizations that achieve 2.8x throughput for serving Kimi K3, covering scheduling, KDA prefix caching, ReplaySSM state recovery, PD disaggregation and state offload, parallelism, MoE, and GPU kernels. The post details the technical approaches behind the performance gains.

    1 reports · 1 sources

  • vLLM v0.28.0 adds Kimi-K3 optimizations, DeepSeek V4 sparse MLA support

    vLLM v0.28.0 release notes detail major performance optimizations for Kimi-K3 and DeepSeek V4 inference, including new kernels, speculative decoding, and memory savings. The release includes 584 commits from 270 contributors, with features such as fused FlashKDA kernels, adaptive speculative token budgets, and sparse MLA support for DeepSeek V4.

    1 reports · 1 sources107 points · 43 comments

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe