Cloudflare details KV cache quantization and weight compression for Kimi and GLM
Cloudflare's Workers AI team described three techniques to serve Moonshot's Kimi K-series and Z.ai's GLM models efficiently on GPUs: quantizing the KV cache, compressing model weights, and protecting the shared cache. These optimizations aim to fit large, long-context mixture-of-experts models into memory while maintaining speed and supporting more customers.
Coverage timeline
Hacker Newsascorbic
# Smaller, faster, safer: running Kimi and GLM at scale Alex Reneau,Kevin Flansburg,andChi McIsaac 6 minute read COPY URL Workers AI runs inference for some of the best open models in the world on GPUs in Cloudflare data centers close to your users. Two of the most capable, and most demanding, are Moonshot's Kimi K-series and Z.ai's GLM. They are large, long-context, mixture-of-experts models, and they are wonderful to use. They are also very hard to serve efficiently because of memory constraints. We've written before about how we serve large models on Workers AI and about separating the prefill and decode phases of inference to get more out of each GPU. This post looks at three techniques we layer on top of that to fit these models into memory and keep them fast: quantizing the KV cache, compressing the model weights, and, because both of those pack more requests onto shared hardware, protecting the cache those requests share. These optimizations enable us to support more customers a