Predictive Speculative KV Replication Proposed for Bursty LLM Inference
A new technique, predictive speculative KV replication, is proposed to handle bursty LLM inference workloads by pre-replicating key-value cache entries across clusters based on predicted request patterns. The approach targets the router and queue layers, where prefix KV cache reuse is critical, noting that Llama-3.3-70B requires about 320 KiB per token of KV cache. This aims to reduce latency and improve cache hit rates during traffic spikes.
Coverage timeline
Hacker Newsshreybirmiwal
## Background When an inference server receives an LLM request, it first hits a router such as SGLang Model Gateway or Dynamo, which decides which cluster to send its request to based on a routing policy. Then it hits a queue in that particular cluster, which may be one GPU or a tensor-parallel group of GPUs serving the same copy of one model. Router layer sending requests to GPU clusters You next need to understand KV cache management. In LLM generation, every request is a sequence of words. As long as two requests have the exact same prefix, they can reuse a lot of the computed math (the KV). It is important to note that the prefix must match exactly, so even a single token difference near the start of a request will break the KV cache. Two requests can share KV only when their prefix matches exactly This shared prefix is called the prefix KV cache. For Llama-3.3-70B, it is about 320 KiB per token, so even a 1,000-token cached prefix is roughly 320 MiB of KV. Typically, the system pr